Confirmation Bias
Hey there! Let’s keep discussing cognitive biases, and today, we will focus on confirmation bias.
Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that confirms or supports a prior assumption or belief. It affects all of us, often without even realizing it.
What are the impacts? When we make an assumption:
We might only select information that supports this assumption.
We might mentally ignore or dismiss information that would discard this assumption.
We might interpret ambiguous information in a way that confirms this assumption.
Think about someone who believes in a conspiracy theory, like flat-Earth believers1:
They seek out content (videos, blogs, etc.) that support the theory.
They dismiss scientific evidence or expert opinion that debunks it.
They interpret ambiguous observations (e.g., observing the horizon from the ground is seemingly flat) as proof that the Earth is flat.
Where can we observe this bias in our work? Let’s discuss two concrete examples.
When troubleshooting an incident, we often start with an assumption; for example: “I think the app crashed due to a spike in user requests“. It’s natural to begin by looking at dashboards and metrics that might validate this theory; this isn’t the confirmation bias yet. However, here’s when the bias comes in:
If we focus solely on dashboards and metrics to confirm our assumption, ignoring those that could point to a different reason (e.g., a recent code change or a network issue).
If we interpret ambiguous information in ways that support our initial assumption rather than considering alternative explanations.
This happened to me about a year ago when I worked on an incident at Google. I found myself locked into a single assumption that ended up being a wrong diagnosis of an incident. I was very frustrated with myself, but that was a good lesson to learn to be more cautious the next time.
Another example I faced a few years ago in a previous company. A PM on my team was convinced that a new way to implement a feature would improve the efficiency of the system from a business standpoint. We set up A/B tests to validate the change, but the PM designed the experiment in a way that primarily collected metrics favorable to his idea, ignoring some that would invalidate his idea:
Without realizing it, we measured success based on metrics that didn’t prove anything. We went all-in on the new implementation, only to find out months later that efficiency was actually decreasing. We had to roll back the change, wasting valuable time and resources.
So, how can we avoid confirmation bias?
Bias awareness: As discussed yesterday, knowing that a bias exists is a critical first step, as it can help us be aware that our decisions are not always rational.
Actively seek contradictory information: When we make an assumption, we should make a conscious effort to look for pieces of evidence that can challenge them.
Assumption awareness: One technique I find useful when I am working on an incident or anything that starts with an assumption is writing down my assumption in my notebook. For example: “Assumption: The app crashed due to a spike in user requests“ with the word assumption underlined. Writing down my assumption helps me remember that it isn’t proven yet and encourages me to explore other possibilities.
Staying open: We should stay open to changing our minds by embracing the possibility of being wrong. There’s nothing more difficult in a team than working with someone sure to be right when there’s no factual evidence yet.
Devil’s advocate: Ask someone to play the devil’s advocate. By seeking a diverse perspective and listening to their reasoning, it can challenge our own biases.
Tomorrow, we will discuss the streetlight effect.
Please don’t tell me we have flat earther among the subscribers of The Coder Cafe :)