Originally published on Squadcast.

Traditionally, developing applications and running them in production was seen as completely separate worlds, usually being the focus and concern of different teams. This kind of separation gives birth to the proverbial wall that separates development and operations, where developers “throw” their code over the wall and expect operations teams to run and manage them in production. This results in teams having different and conflicting goals: development teams prioritize building and shipping new features while operations teams focus on system stability, where code changes are seen as potential threats.

With systems getting larger and larger, this type of situation does not scale, and a new way of doing things needed to emerge. And in 2009 Patrick Debois coined the term “DevOps”, a set of principles and practices with emphasis on people and culture and the goal of improving collaboration between development and operations teams.

A few years before, facing a similar situation, Google put Benjamin Treynor in charge of a team with the goal of ensuring that Google’s websites were available, reliable, and as serviceable as possible. Since Benjamin was a software engineer he approached an operations problem with a software engineering perspective, giving birth to what is today known as Site Reliability Engineering (SRE): “what happens when you ask a software engineer to design an operations team”. SRE rose to fame around 2016 after Google published the book Site Reliability Engineering: How Google Runs Production Systems, describing a lot of the practices they use to run their systems in production.

SRE is all about running reliable production systems. Big companies like Google, Facebook, or Amazon run large production systems and face many challenges most companies rarely do. Despite that, the way they run their systems can help us run ours. But even armed with all that information, how do we go about starting our SRE journey?

Firefighting

In the beginning, teams are reactive. There are a lot of moving parts, highly complex systems, failures appearing right, left, and center. The world is chaotic and it seems close to impossible to apply any sort of engineering practice to tackle the situation. At this stage, teams don’t have many options and need to keep struggling, responding to a myriad of crises, while trying to reserve some time, energy, and focus to try and improve the situation. Putting brakes on feature development can help stabilize a system, getting teams some relief. Adding people to teams that can focus on automation, can help reduce toil.

This is a difficult stage for teams. It requires a dual perspective since they will need to keep systems running while building a new approach to run operations. This is also a reflection on operations teams being brought late into the product cycle. They can be made aware of a new service or a new set of requirements just before a service needs to go live, or even worse, when something is already live and failing miserably.

Gatekeeping

Usually, one of the first measures that are put in place to deal with firefighting is gatekeeping. The goal is to make every change to production pass through and be approved by the SRE team. On a small scale, this approach can work for a while. But, as systems and teams grow, SRE teams start being a choke point, limiting change and slowing down other teams.

At this stage, SRE teams work to become more engaged with development teams by implementing processes. While this reflects positive engagement, it can derail into an us-versus-them struggle, leading teams to circumvent the processes and SRE teams all together.

Partnering

When there are frameworks in place (e.g. SLOs and Error Budgets) that can be used to measure reliability and make decisions on how to act, SRE teams can remove themselves from the critical paths to production. They can then partner with development teams and work together to meet the desired reliability criteria. At the same time, SRE teams will be continually involved in how to measure the impact on user experience. At this stage, SREs are involved in the process earlier in the cycle and work alongside development teams right from the start, on the reliability of the product.

Relationships between teams become a lot less antagonistic. SRE teams are regularly sought for cooperation since they enable much higher satisfaction and long-term value.

Enabling

When reliability is part of the process, SREs are involved in the full lifecycle of a service, from creation to decommissioning. SREs can act as consultants, bringing consciousness about business goals, reliability, and security. Processes are in place to allow teams to measure how reliable their services are and how to act when necessary. This allows scalability since teams can operate independently while being able to rely on SREs when necessary. Development teams have the ability to pull SREs and SREs don’t need to impose demands. A voluntary, data-supported engagement between teams becomes enjoyable and sustainable.

Enabling SRE teams to become highly functional is a long process. It takes time and some steps are “mandatory”. Starting in firefighting mode is almost inevitable for most organizations. It is then, when they realize that a new approach to manage operations needs to emerge. At this stage, some time has to be set aside so that teams can work on processes and tools to help ease the pain of constant firefighting. A lightweight gatekeeping stage can help bring teams together as well as identifying processes and automation that would pave the way to the partnering stage. Enabling stage is the end goal where teams work together, armed with tools that help measure reliability, and processes that help prioritize work and decide how to proceed when reliability is at risk.

While some short-circuiting can be done to accelerate the process, some stages build on others and use them as foundations. High-level management support can help accelerate the evolution by allowing teams to focus on prioritizing the work that will make them reach the enabling stage. Focus on success stories to help build momentum. A lot of SRE is cultural, a different way of doing things. Successes help build a narrative and motivate people to follow the same path.

Creating an SRE team can seem like a daunting task and there are several topologies that can be adopted. Each organization needs to find what better suits its needs. But there are some guiding principles to help navigate this journey:

  • Start small and iterate - in the beginning, keep it simple. Select a small group of people to implement SRE practices. Find what works and what doesn’t. Iterate and onboard new practices
  • Find the right people - SRE teams should have a mix of domain knowledge. Either through internal or external hiring, aim to build teams with a diverse set of skills
  • Scan for the right qualities - depending on the scope and topology of the team, the skill set might defer. Some good qualities to scan for are the ability to see the bi picture, the ability to troubleshoot, good communication skills, desire to remove toil, a knack to dig deeper, and do excellent teamwork
  • Training - a lot of SRE work is cultural. The ability to communicate well and provide training to teams is paramount
  • Guidelines and Governance - SRE teams help facilitate the adoption of a reliability mindset. They can help establish guidelines that make it easier for teams to onboard reliability practices

Either through a separate SRE team or any other topology, it’s important to start simple and iterate. Introduce SRE principles, assess, find out what works, and then incorporate. Navigating from Firefighting to Enabling is a journey and it will be slightly different for each organization.