The Staff Engineer's Path

the senior level as the “anchor” level for a career ladder. As Rogers says, “The levels below are for people to grow their autonomy; the levels above increase impact and responsibility.”

The Medium engineering team wrote a blog post that lays out three reasons titles are necessary: “Helping people understand that they are progressing, vesting authority in those people who might not automatically receive it, and communicating an expected competency level to the outside world.”

Individual teams tend to optimize for their own interests; an engineer on a single team is likely to be laser-focused on achieving that team’s goals. But often decisions that seem to belong to one team have consequences that extend far beyond that team’s boundaries. The local maximum, the best decision for a single group, might not be anything like the best decision when you take a broader view.

To avoid local maxima, teams need decision makers (or at least decision influencers) who can take an outsider view—who can consider the goals of multiple teams at once and choose a path that’s best for the whole organization or the whole business.

the murky and difficult parts of the project are not fascinating algorithmic research problems: they involve spelunking through legacy code, negotiating with busy teams that don’t want to change anything, and divining the intentions of engineers who left years ago.

One way to keep a project moving is to have someone who feels ownership for the whole thing, rather than any of its individual parts. Even before the project kicks off, that person can scope out the work and build a proposal. Once the project is underway, they’re likely to be the author or coauthor of the high-level system design and a main point of contact for it. They maintain a high engineering standard, using their experience to anticipate risks and ask hard questions. They also spend time informally mentoring or coaching—or just setting a good example for—the leads of individual parts of the project. When the project gets stuck, they have enough perspective to track down the causes and unblock it (more on that in Chapter 6). Outside the project, they’re telling the story of what’s happening and why, selling the vision to the rest of the company, and explaining what the work will make possible and how the new project affects everyone.

Whenever there’s a feeling of “someone should do something here,” there’s a reasonable chance that the someone is you.

At senior level, maybe your manager advised you on which problems were important to solve, and left it to you to figure out what to do about it. At staff + levels, your manager should be bringing you information and sharing context, but you should be telling them what’s important just as much as the other way around.

Reporting “high” in the org chart, such as to a director or VP, will give you a broad perspective. The information you get will be high-level and impactful, and so will the problems you’re asked to solve. If you’re reporting to a very competent senior person, watching them make decisions, run meetings, or navigate a crisis can be a unique and valuable learning experience. That said, you’ll probably get a lot less of your manager’s time than you would if you had a local manager. Your manager might have less visibility into your work and therefore might not be able to advocate for you or help you grow.

if you’re reporting to someone low in the org hierarchy, make sure to have skip-level meetings with your manager’s manager. 8 Find ways to stay connected to your organization’s goals.

A narrow scope can mean that there’s not enough work to keep you busy, and that you may overshadow less experienced people and take learning opportunities away from them. If you always have time to answer all of the questions and take on all of the tricky problems, nobody else gets experience in doing that.

An engineer who’s not busy can be inclined to make work for themselves. When you see a vastly overengineered solution to a straightforward problem, that’s often the work of a staff engineer who should have been assigned to a harder problem.

Yonatan Zunger, distinguished engineer at Twitter, describes the four disciplines that are needed in any job in the world: Core technical skills Coding, litigation, producing content, cooking—whatever a typical practitioner of the role works on Product management Figuring out what needs to be done and why, and maintaining a narrative about that work Project management The practicalities of achieving the goal, removing chaos, tracking the tasks, noticing what’s blocked, and making sure it gets unblocked People management Turning a group of people into a team, building their skills and careers, mentoring, and dealing with their problems Zunger notes that the higher your level, the less your mix of these skills corresponds with your job title: “The more senior you get, the more this becomes true, the more and more there is an expectation that you can shift across each of these four kinds of jobs easily and fluidly, and function in all rooms.”

Be wary of choosing a project that already has a lot of senior people on it. Scope out who else is working on the problem and whether they seem likely to succeed at solving it. Some projects may even be slowed by an extra leader joining. 11 In general, if there are more people being the wise voice of reason than there are people actually typing code (or whatever your project’s equivalent is), don’t butt in. Try to choose a problem that actually needs you and that will benefit from your attention.

Uncovering this map means taking a long view and evaluating the purpose of your work. Is each project a goal in itself, or is it just a milestone along the path to the actual goal? Sometimes you’ll discover that there isn’t a destination at all or that there are several incompatible ones. When nobody has declared what the treasure is, or everyone disagrees on how to get to it, a staff engineer can have a huge impact by creating a vision or strategy, making decisions, or otherwise drawing a brand-new treasure map for the organization.

When you join a new company, most of the big picture is completely unknown to you. A big part of starting a new job is building context, learning how your new organization works, and uncovering everyone’s goals. Think of it like the fog of war in a video game, where you can’t see what awaits you in the parts of the map you haven’t explored yet. As you scout around, you clear the fog and get a better picture of the terrain, learning what’s surrounding you and whether there are wolves coming to bother your villagers. You can set out to uncover the obscured parts in all three of the maps and find ways to make that information easy for other people to understand. For instance: Your locator map can help you make sure the teams you work with really understand their purpose in the organization, who their customers are, and how their work affects other people. Your topographical map can help highlight the friction and gaps between teams and open up the paths of communication. Your treasure map can help you make sure everyone knows exactly what they’re trying to achieve and why. You’ll be able to clear some parts of the map through everyday learning, but you’ll need to deliberately set out to clear other parts. A core theme of this chapter is how important it is to know things: to have continual context and a sense of what’s going on. Knowing things takes both skill and opportunity, and you might need to work at it for a while before you start seeing what you’re not seeing.

the local maximum starts to feel really important. The more time you spend staring at your own group’s problems, the more they seem special and unique and worthy of special, unique solutions. And sometimes they are! But it’s unusual to find a problem that is genuinely brand new. If you check for prior art and preexisting solutions, you’ll spend less time reinventing wheels.

“Nines don’t matter when users aren’t happy.”

Amazon’s tenet of “respect what came before” includes a reminder that “many problems are not essentially new.” You’ll come up with better solutions if you study what other people have already done before creating some new thing. Remember that your goal is to solve the problem, not necessarily to write code to solve it. Take the time to understand what already exists—inside and outside your organization—before building something new.

For online conversations, I like Rands Leadership Slack: the #architecture and #staff-principal-engineering channels are gold. The LeadDev Slack’s #staffplus channel is very active during events, too. I subscribe to the monthly InfoQ Software Architects’ Newsletter, as well as the VOID report, and SRE Weekly. I read the Raw Signal newsletter for a weekly dose of a manager point of view. I also eagerly await the quarterly Thoughtworks Radar.

Team tectonics have similar properties. As domains of responsibility smash against each other, they form an organizational terrain, complete with overlaps and conflict, ridges and chasms. Figure 2-6. Simplified map of Earth’s principal tectonic plates. (adapted from: Scott Nash, public domain, https:// oreil.ly/ UdeNz). Reorganizations can disrupt communication between groups that need to work closely together. Teams that are under a heavy load can entrench and put up barriers. A new senior leader can cause an earthquake that reshapes the landscape overnight.

Depending on where you are, you’ll need to frame your initiatives differently. If you’re somewhere that moves like lightning, you’ll want an incremental path that shows value immediately. In a more deliberate environment, you’ll need to show that you’ve thought through the whole plan. This is tightly connected to oral and written culture, too.

Westrum classified organizations and their influence on information flows using three categories (see Table 2-1): Pathological A low-cooperation culture where power and status are the goal and people hoard information; in Westrum’s words, “a preoccupation with personal power, needs, and glory” Bureaucratic A rule-oriented culture where information moves through standard channels and change is difficult; the preoccupation here, Westrum writes, is with “rules, positions, and departmental turf” Generative A mission-oriented, high-trust, high-cooperation culture where information flows freely; Westrum calls this “concentration on the mission itself”

The shadow org chart helps you understand who the influencers of the group are, and it’s probably not the same as the actual org chart. These influencers are the people you need to convince before a change can happen. The authors identify “connectors” who know people all across the org, and “old-timers” who, regardless of rank or title, wield influence just from being around a long time. These folks are likely to have a good pulse on what can and can’t work, and the people who do have rank and title will likely trust them and rely on their good judgment when making decisions. If you can get their buy-in, you’re making good progress.

The Lean manufacturing folks talk about gemba, the idea of walking the manufacturing floor and seeing how things actually operate. Find some avenues to stay attached to the work that teams around you are doing.

I asked on Rands Leadership Slack about how everyone approaches knowing things, and a common thread was paying attention to information that isn’t secret exactly, but isn’t necessarily for you. This included reading senior people’s calendars, skimming agendas or notes for meetings you’re not in, and—something that had never occurred to me—looking at the full list of Slack channels sorted by most recently created so you can see what new projects are happening.

A technical vision is sometimes called a “north star” or “shining city on the hill.” It doesn’t set out to make all of the decisions, but it should remove sources of conflict or ambiguity and empower everyone to choose their own path while being confident that they’ll end up at the right place. Resources for Writing a Technical Vision If you’re setting out to write a technology vision document, here are some resources I recommend: Fundamentals of Software Architecture by Mark Richards and Neal Ford (O’Reilly) Chapter 4 of Making Things Happen by Scott Berkun (O’Reilly) “How to Set the Technical Direction for Your Team”, by James Hood of Amazon “Writing Our 3-Year Technical Vision”, by Daniel Micol of Eventbrite There’s no particular standard for what a vision looks like. It could be a pithy inspirational “vision statement” sentence, a 20-page essay, or a slide deck. It might include: A description of high-level values and goals A set of principles to use for making decisions A summary of decisions that have already been made An architectural diagram It could be very detailed and go into technology choices, or it could stay high-level and leave all of the details to whoever is implementing it. Whatever you create, it should be clear and opinionated, it should describe a realistic better future, and it should meet your organization’s needs. If you could wave a magic wand and be done, what would your architecture, processes, teams, culture, or capabilities be? That’s your vision.

Just like a technical vision, a technical strategy should bring clarity—not about the destination, but about the path there. It should address specific challenges in realistic ways, provide strong direction, and define actions that the group should prioritize along the way. A strategy won’t make all of the decisions, but it should have enough information to overcome whatever difficulties are stopping the group from getting to where it needs to go. Resources for Writing a Technical Strategy The canonical book on strategy is Good Strategy/ Bad Strategy by Richard Rumelt (Currency). I recommend taking the time to read it if you can. Other great resources include: “Technical Strategy Power Chords” by Patrick Shields “Getting to Commitment: Tackling Broad Technical Problems in Large Organizations” by Mattie Toia “A Survey of Engineering Strategies” by Will Larson Technology Strategy Patterns by Eben Hewitt (O’Reilly) Rands Leadership Slack, specifically the channels #technical-strategy and #books-good-strategy-bad-strategy

In Good Strategy/ Bad Strategy, Rumelt describes “the kernel of a strategy”: a diagnosis of the problems, a guiding policy, and actions that will bypass the challenges. Let’s look at each one. The diagnosis “What’s going on here?” The diagnosis of your situation needs to be simpler than the messy reality, perhaps by finding patterns in the noise or using metaphors or mental models to make the problem easy to understand. You’re trying to distill the situation you’re in down to its most essential characteristics so that it’s possible to really comprehend it. This is difficult. It will take time. Guiding policy The guiding policy is your approach to bypassing the obstacles described in the diagnosis. It should give you a clear direction and make the decisions that follow easier. Rumelt says it should be short and clear, “a signpost, marking the direction forward.” Coherent actions Once you’ve got a diagnosis and guiding policy, you can get specific about the actions you’re going to take—and the ones you won’t. Your actions will almost certainly involve more than technology: you might have organizational changes, new processes or teams, or changes to projects. I really can’t stress this enough: you’ll commit time and people to these actions rather than to the long list of other ideas that were on the table at the start. This kind of focus will likely mean that you and others don’t get to do some things you’ve been excited about. It is what it is.

getting people to agree isn’t a chore that stands in between you and the real work of solving the problem: the agreement is the work.

be wary of fighting for a marginally better path at the cost of not making a decision at all. As Will Larson writes, “Give your support quickly to other leaders who are working to make improvements. Even if you disagree with their initial approach, someone trustworthy leading a project will almost always get to a good outcome.”

Early on, you might ask broad, open-ended questions in your interviews. “We’re creating a plan for X. What’s important to include?” What’s it like working with…?” If you could wave a magic wand, what would be different about…?”

Take a tactic from the Internet Engineering Task Force (IETF), whose principles of decision making famously reject “kings, presidents, and voting” in favor of what they call “rough consensus,” positing that “lack of disagreement is more important than agreement.” In other words, take the sense of the group, but don’t insist that everyone must perfectly agree. Rather than asking, “Is everyone OK with choice A?” ask, “Can anyone not live with choice A?” When IETF working groups make decisions, they’re looking for a large majority of the group to agree and for the major dissenting points to have been addressed and debated, even if not to everyone’s satisfaction. There may not be an outcome that makes everyone happy, and they’re OK with that.

There’s a political science concept called the Overton window (Figure 3-3): the range of ideas that are tolerated in public discourse without seeming too extreme, foolish, or risky. If your ideas are too futuristic for the people you need to believe in them, your colleagues will dismiss your document and you’ll lose credibility

the process of nemawashi, one of the pillars of the Toyota Production System. It means sharing information and laying the foundations so that by the time a decision is made, there’s already a consensus of opinion. 11 If there’s someone who’ll need to give a thumbs-up to your plan, you’ll want those people to show up to any decision-making meeting already convinced that the change is the right thing to do. I’ve always framed this as “Don’t call for a vote until you know you have the votes,”

Here’s how I start, no matter the size of the project: I create a document, just for me, that’s going to act as an external part of my brain for the duration of the project. It’s going to be full of uncertainty and rumors, leads to follow, reminders, bullet points, to-dos, and lists. When I’m not sure what to do next, I’ll return to that document and look at what Past Me thought was important. Putting absolutely everything in one place at least removes the “Where did I write that down?” problem.

Here are some topics you might cover at your kickoff: Who everyone is What the goals of the project are What’s happened so far What structures you’ve all set up What’s happening next What you want people to do9 How people can ask questions and find out more

oral versus written company cultures. The bigger the company, the more likely that your culture has shifted toward the latter, and that there is some expectation that you will write and review design documentation. That’s because it’s very difficult to have many people achieve something together without shared understanding, and it’s hard to be sure you have that shared understanding without a written plan. Whether you’re creating features, product plans, APIs, architecture, processes, configuration, or really anything else where multiple people need to have the same understanding, you won’t truly know if people understand and agree until you write it down. Writing it down doesn’t mean you need a 20-page technology deep dive for every tiny change. A short, snappy, easy read can be perfect for getting a group on (literally) the same page. But you should at least include the important aspects of the plan, and let other people get in touch with you if they see hazards in your path. Asking for review on a design doesn’t just mean asking about the feasibility of an architecture or a series of steps; it includes agreeing on whether you’re solving the right problem at all and whether your assumptions about other teams and existing systems are correct. An idea that seems like an obvious path for one team may cause work for or break the workflows of another org. As my friend Cian Synnott says, a written design is a very cheap iteration.

What goes in an RFC? Your company may already have its own RFC template, and you should follow that if it exists. However, here are the headings that I put in every RFC, and that I think should be included at an absolute minimum.

Depending on what you’re trying to do, the design section could include: APIs Pseudocode or code snippets Architectural diagrams Data models Wireframes or screenshots Steps in a process Mental models of how components fit together Organizational charts Vendor costs Dependencies on other systems What matters is that at the end, your readers should understand what you intend to do and should be able to tell you whether they think it will work.

Has anyone else at your company ever tried something similar, and why isn’t their solution a good fit? I have a policy that if a plausible-seeming option already exists inside the company and we’re not going to use it, the RFC author has to send the new design to the people who own that system and give them an opportunity to respond.

If it seems trivial, it’s because you don’t understand it.

Discussing the smallest decisions the most Who doesn’t love a good bikeshed discussion! The expression “bikeshedding” came from C. Northcote Parkinson’s 1957 “Law of Triviality”, which holds that since it’s much easier to discuss a trivial issue than a difficult one, that’s where teams tend to spend their time. 15 Parkinson’s example was a fictional committee evaluating the plans for a nuclear power plant but spending the majority of its time on the easiest topics to grasp, like what materials to use for the staff bicycle shed. 16 Tech people are usually aware of the concept of bikeshedding, but even senior people drift into writing long paragraphs about the most trivial, reversible decisions, while not engaging at all with the ones that are harder to grasp or to find consensus on.

As Joy Ebertz points out, “Writing code is rarely the highest leverage thing you can spend your time on. Most of the code I write today could be written by someone much more junior.” Ebertz notes, however, that coding gives you a depth of understanding that’s hard to gain otherwise and helps you spot problems. What’s more, “If spending a day a week coding keeps you engaged and excited to come to work, you will likely do better in the rest of your job.” Finally, staying involved in the implementation ensures that you feel the cost of your own architectural decisions as much as your team does.

a “watermelon” project? They’re all green on the outside, but the inside is red.

Write things down. Be clear and opinionated. Wrong gets corrected, vague sticks around.

the “Three Bullets and a Call to Action” method that Brian Fitzpatrick and Ben Collins-Sussman outline in their book, Debugging Teams (O’Reilly). As they write: “A good Three Bullets and a Call to Action email contains (at most) three bullet points detailing the issue at hand, and one—and only one—call to action. That’s it, nothing more—you need to write an email that can be easily forwarded along. If you ramble or put four completely different things in the email, you can be certain that they’ll pick only one thing to respond to, and it will be the item that you care least about. Or worse, the mental overhead is high enough that your mail will get dropped entirely.”

The American Society of Civil Engineers publishes its engineering grades, and its requirements include number of years of experience: Grade V (Typical titles: Senior engineer, program manager): 8 + years Grade VI (Typical: Principal engineer, district engineer, engineering manager): 10 + years Grade VII (Typical: Director, city engineer, division engineer): 15 + years Grade VIII (Typical: Bureau engineer, director of public works): 20 + years We’re less rigorous in software engineering.

I love asking for an “ELI5,” a term that comes from Reddit and means “Explain it like I’m five years old.” It’s a helpful shortcut to mean “Look, rather than guessing my level of understanding, just spell it out for me. I promise not to be offended if you tell me things I already know.” (The social contract here is that you can’t get offended if they start with the very basics of the topic.)

Your disaster plans will invariably have something go wrong, so simulate disaster with chaos engineering tooling or controlled outages. Drills, game days, or tabletop exercises can let you uncover which parts of your response won’t work. And of course, if you haven’t tested restoring your backups, assume you don’t have any backups.

Successful teaching includes hands-on learning and activating knowledge: the student should be doing as well as listening. Find opportunities to let them lead, whether that means using the tool themselves, typing commands, or opening tabs on their laptop instead of yours. It’s even better if they end up with an “artifact” to refer back to, like a diagram or code snippet.

engineering director Cate Huston offers five metrics for evaluating your job health: Whether you’re learning Whether you’re investing in transferable skills or navigating dysfunction How you feel about recruiting other people to your team How confident you feel How stressed you feel A great job situation keeps you growing toward your goals, and your self-confidence and abilities stay high.

As Huston says, “Sometimes five years of experience is just… the same year of experience, five times over.” Oof.