#48: Streetlight Effect
Hello! Today, let’s talk about an observational cognitive bias: the streetlight effect.
Let’s start this issue with a story. This story takes place on a dark street with a single streetlight:
A policeman notices a drunk man searching for something under a streetlight. Curious, he approaches and asks what the man is looking for.
“I lost my keys“, says the drunk. Naturally, the policeman starts helping him search under the streetlight.
After one minute or two of unsuccessful search, the policeman asks, “Are you sure you lost them here?“.
The drunk replies, “No, but this is where the light is best“.
This story illustrates the streetlight effect—a cognitive bias when people focus on what’s easily visible, even if it isn’t the most relevant information.
Five concrete examples in software engineering in which we can observe the streetlight effect:
Observability: Teams often rely on existing dashboards and assume everything is fine if those metrics are green. However, these dashboards might not reflect what’s happening from the user’s perspective or cover critical, hidden aspects of system health.
Tech debt: Developers may focus on addressing visible or minor technical debt because they are used to this part of the codebase while ignoring deeper and more impactful technical debt that is less obvious but more critical.
A/B testing: Teams may focus on easily measurable data to determine the success or failure of an A/B testing scenario. However, this data might not capture efficiently the overall user experience, leading to misleading results.
Code reviews: Developers may focus on easily visible issues, such as syntax or styling, but missing more important problems, such as potential edge cases, logic issues, or architectural flaws.
Optimization: Developers may focus on optimizing some parts of the code that are easy to monitor in terms of latency, ignoring potentially less visible parts that have a stronger impact on performance.
Why should we care? The streetlight effect can lead to serious consequences such as making wrong decisions, low efficiency, or a decrease in user satisfaction.
Avoiding the streetlight effect isn’t easy. First, it’s important to be aware of its existence1. Then, we can think about possible strategies such as:
Challenging the defaults: Regularly listing and challenging existing assumptions. Are we focusing on the right metrics? Is this the right data?
Data-driven decisions: Making decisions based on comprehensive and accurate data, not assumptions.
Postmortems: After incidents or failures, we should go beyond immediate causes and explore contributing factors that might not have been immediately visible.
Let’s be mindful of the streetlight effect for us and our teams to make better and more solid decisions.
Tomorrow, we will discuss the survivor bias.
Again.