Principal Site Reliability Engineer - Hybrid

Your opportunity

At Schwab, you are empowered to make an impact on your career. Here, innovative thought meets creative problem solving, helping us “challenge the status quo” and transform the finance industry together. 

As a Principal Site Reliability Engineer for Schwab's Technology Solutions organization, you will be responsible for building a purposeful, proactive, and sustainable approach to reliability on a foundation of SRE principles. You will partner with multiple support teams, architects, developers, and other stakeholders to develop common tools and guidance and drive adoption of key reliability engineering practices in support of large-scale and mission-critical services.  Through your deep SRE knowledge and history of implementation, you will have open, candid conversations with senior leaders and engineers and play a pivotal role in establishing a foundational SRE practice at Schwab. 

The SRE Enablement team is part of Schwab’s Strategy & Business Management (SBM) organization where you will work with your counterparts to help implement the SRE practice through evangelization, organizational engagement, resiliency programs, office hours, leadership forums and an engineering focused Community of Practice.

What you have

Required Qualification

• 5+ years in SRE role plus at least 3 years in an architect or technical leadership role.
• Minimum 3+ years of experience designing and implementing highly scalable and fault tolerant systems.
  • In-depth knowledge of resilience patterns (i.e. circuit breakers, timeouts, retries, etc.) and how to design and implement them.
  • In-depth knowledge of CICD processes and tools to ensure software is delivered safely using known deployment strategies (i.e. blue/green, canary deployments, feature toggles, etc.).
• 3+ years hands-on experience with monitoring and observability tools (e.g., Prometheus, Grafana, Datadog, Splunk), with a proven track record of setting up dashboards and alerts.
• Developed at least 5 scripts or tools that reduced repetitive operational toil.
• Authored technical postmortems (at least weekly) with root cause analyses and documented action items that resulted in measurable resiliency improvements.
• Authored and maintained comprehensive SRE documentation for at least 3 critical systems or workflows, including incident response guides, runbooks, operational playbooks, SLO implementation, and observability. 
• Contributed to the SLO strategy for at least 5 teams, ensuring alignment with business and client objectives.
• Presented findings or led training sessions at least twice annually to share SRE practices, enhancing team performance or adoption rates for reliability engineering methods.
• Managed or mentored at least 2 junior engineers or teams in SRE best practices, with improvements in incident resolution speed and reliability metrics.
• Led or participated in at least three cross-functional SRE-focused initiatives that included key stakeholders from both technical and business units.
  Participated in resilience or chaos engineering exercises at least yearly, with documentation showing a reduction in unplanned downtime.

Preferred Qualifications

• Evangelize SRE mindset and practices across the Schwab Technology Solutions organization.
• Partner with support, development, and business stakeholders to develop, measure, and leverage service level objectives.
• Design and develop solutions to eliminate toil and manual effort from day-to-day support responsibilities.
• Identify and implement improvements to logging, metrics, and tracing telemetry and triaging capabilities across a diverse technology stack.
• Lead complex triage and postmortem activities for critical issues and drive prioritization/resolution of remediation items.
• Perform chaos engineering experiments to improve application resilience to known and unknown failures.
• Document reliability guidance and best practices. Advocate for and drive adoption of said practices.
• Foster a culture of learning through coaching, mentoring, and knowledge sharing around reliability practices, processes, and tools.
• Develop tools, frameworks, and instrumentation to validate and increase release success for applications.