#49: Survivor Bias
Hello! Today, let’s explore the latest cognitive bias of the week: survivor bias.
It’s World War 2. You are the engineer responsible for strengthening the planes that go into battle. As it’s war, some planes return from fights, and unfortunately, some do not return due to crashes. As a data enthusiast, you ask for a report showing where the returning planes were hit. You receive this:
Based on this data, you decide to reinforce the five areas where returning planes were hit the most:
The new model is deployed, and surprise—there's no improvement in the rate of planes returning from combat. Can you guess what went wrong?
The main issue in this analysis is that the report only accounts for planes that returned, not those that crashed. In other words, instead of strengthening the blue areas, we should have focused on all the other parts of the plane, as these were likely the cause of the crashes.
This example illustrates survivor bias: the tendency to focus on entities that have successfully passed a selection process while neglecting those that didn’t. Said differently: when we focus only on the successes and overlook the failures.
Some examples of when survivor bias arises:
Project success stories: Even when we manage to ship an important project as a team, it’s crucial to reflect on what could be improved, not just celebrate our wins.
FAANG idolization: It can be easy to idolize big players like Amazon, Meta, and Google. For instance, many people migrate to a monorepo setup because it works for Google, ignoring all the companies for which it didn’t work (mainly because of the substantial tooling required for it to succeed1).
Interview process: Focusing solely on a candidate's successes can be misleading. We can learn at least as much from someone’s failures as it can provide some insights into their decision-making, for instance.
Tech stacks and patterns: Assuming a specific tech stack (e.g., Kubernetes) or pattern (e.g., microservices) is the best choice just because some successful companies use it, without considering failed projects that used the same stacks/patterns.
Pair programming: Assuming pair programming is effective for all teams based on successful examples while overlooking teams that found it disruptive or unnecessary.
Career path: Believing that the career trajectories of some successful engineers should be the way to go. For example, many think contributing to open-source will significantly advance their careers, ignoring those who spent time on low-impact contributions2.
Survivor bias can lead us to make decisions that don’t accurately reflect the bigger picture. Let’s be aware of this bias to make more informed decisions in our projects, teams, and career paths.
Tomorrow, you will receive your weekly recap.
I’ll discuss monorepos in a future issue: why it works for us at Google and why it may not work outside Google.
Let’s be clear: I’m not suggesting you should avoid open-source contributions. What I mean is that not everyone gains the same benefits from these activities. The same applies to various activities such as public speaking, hackathons, and blogging. The most important part is to find what works best for you.