Complacency is not bias: sorting out the automation vocabulary

I have been using the term automation complacency, as observed in aviation with autopilots, quite often lately. In proposals, in lectures, in reviews of other people's papers. Recently, I stopped and asked myself whether I could actually distinguish it from automation bias. I could not.

Naturally, I wanted to find out more. When I went back to the human factors literature, I found that, over roughly four decades, the field had separated these terms carefully. Therefore, we in software engineering should not muddle these concepts and be careful with the distinctions.

So this post is my attempt to sort them out, and then to ask which of them actually transfer to developers working with LLMs and coding agents.

The chain

The image at the top shows the structure of this argument. These terms are not interchangeable. They represent different stages in a causal chain, and each requires a different solution. The process moves from a design decision at the top to an accountability problem at the bottom. Complacency and automation bias are parallel failures of attention and judgment.

Lisa Bainbridge's "Ironies of Automation" from 1983 is a good starting point. Four pages. She argued that automating easy tasks leaves the operator with the difficult remains. The human must monitor a system that rarely fails and then intervene perfectly when it does. The convenience of the designer creates a problem for the operator.

The process starts with a design decision rather than human error. Parasuraman, Sheridan, and Wickens (2000) identified four stages where automation occurs: information acquisition, information analysis, decision selection, and action implementation. They also identified different levels within those stages. They predicted that out-of-the-loop effects increase when later stages are automated at higher levels. Automating what an operator sees is different from automating what an operator decides.

Then comes trust. Lee and See (2004) defined trust as an attitude. They argued that the important factor is not the amount of trust, but how well calibrated that trust is compared to actual capability. Hoff and Bashir (2015) added the concept of resolution. This is the ability to distinguish between situations where a tool is competent and where it is not. A person with constant high trust and a person with constant low trust are both poorly calibrated. Only the person with low trust appears cautious.

Complacency is the next stage. In Parasuraman and Manzey's (2010) synthesis, complacency is not an attitude or a character flaw. It is an observable reallocation of attention: the operator samples the automated channel less often than the situation warrants. Two aspects of this definition are especially important: First, the term only has meaning when compared to a baseline of how often an operator should check the system. Without a baseline, the word is an accusation rather than a measurement. Second, complacency is most common when there are multiple competing tasks. It is not caused by laziness. It happens when attention is limited, and the automation seems like the easiest place to save effort.

Automation bias is a separate issue. Mosier and Skitka (1996) describe this as a failure of decision-making rather than attention. It involves using automated output as a shortcut instead of looking for information. This is divided into errors of omission and errors of commission. Omission errors occur when an operator misses a problem because the automation did not report it. Commission errors occur when an operator follows a recommendation despite having evidence that contradicts it. Complacency is a failure to look, while bias is a failure to weigh evidence.

Reliance is the resulting behavior. Parasuraman and Riley (1997) used the terms use, misuse, disuse, and abuse. Misuse is overreliance. Disuse is the failure to use automation that would be helpful, often due to frequent false alarms. Abuse is the deployment of automation by management without considering human consequences. Most current discussions focus on misuse. The Microsoft literature review on overreliance by Passi and Vorvoreanu (2022) covers the human-computer interaction perspective, though it uses "overreliance" as a general term for the specific distinctions mentioned above.

Cognitive offloading is a slower process. Risko and Gilbert (2016) use this term to describe using the environment to reduce mental effort. This is the mechanism that causes skills to decline through lack of use. Offloading is normal and includes things like writing or using checklists. It becomes problematic when the offloaded skill is necessary for another task. For example, heavy GPS use is linked to a poorer understanding of one's environment. This process happens over months and years, so it requires different study methods than the other stages.

Accountability is the final stage. Madeleine Elish (2019) used the term moral crumple zone to describe how human operators are blamed for failures they could not prevent. In these cases, a human is required to be in the loop but lacks the actual power to change the outcome.

Two reasons to be skeptical

I should not present this as settled. Dekker and Hollnagel (2004) argued that several of these terms work as folk models: they explain by naming, and they are hard to falsify because they are inferred from the behavior they are supposed to explain. Why did the operator not monitor enough? Complacency. How do we know there was complacency? They did not monitor enough. We can define the constructs operationally through sampling rates, error types, and calibration curves rather than use them as labels for outcomes.

The second one is Dekker and Woods' (2002) substitution myth. Automation transforms the task and creates new coordination demands instead of removing a piece of human work and leaving the rest untouched. Any sentence of the form "the AI writes the code and the developer just reviews it" is an instance of the myth.

What transfers to LLMs and agents

Let's start with reliability. The classic complacency effects are strongest with automation that is highly and consistently reliable. Variable reliability tends to weaken them, because variability keeps you sampling. On that logic LLMs, which are inconsistent and occasionally wrong, should be relatively safe.
Initially, that reassured me. Then I noticed the assumption in it. Variability only protects you if the failures are detectable when you look. LLM failures are often not easily detectable: they are syntactically clean, idiomatic, and confidently commented. They fail while appearing competent.

Aviation automation produced rare and salient failures. LLMs produce frequent and camouflaged failures. If this is correct, the protective effect of variable reliability does not transfer. We should expect monitoring failures at reliability levels the literature would consider safe.

The second difference involves the specific task being monitored. Checking an altimeter is cheap, but reviewing a four-hundred-line code change is expensive. The cost of generating code has decreased, while the cost of verifying it remains high. A developer chooses how much effort to spend on comprehension based on these costs. This behavior relates to research on cognitive offloading rather than research on sustained attention. Our metrics should measure the depth of engagement instead of the time spent on a task.

The third difference is the most significant. Existing research has not addressed it. Calibrating trust requires fast and diagnostic feedback. A pilot knows within seconds if an aircraft followed an instruction. A developer might only learn about an agent's error weeks or months later during an incident handled by a different person. The original context is lost by then. I suspect that a developer's trust in a coding agent does not eventually match the tool's actual reliability. Trust will only match reality if we intentionally create a feedback loop.

The fourth difference is that code review is becoming the primary defense against agent errors. Bacchelli and Bird (2013) showed that code review is less effective at finding defects than people assume. It functions as a tool for knowledge transfer and social signaling. Relying on an ineffective process is a poor strategy.

One aspect of the aviation analogy is accurate. Complacency happens most often when there are competing task demands. Managing multiple agents at once is now often the recommended workflow. METR noticed a detail while redesigning their productivity experiment. Some developers could not accurately report their time because they switched to unrelated tasks while an agent worked. This condition causes complacency. It is currently being used as a strategy for productivity.

I will add a qualification regarding METR because people often cite their data incorrectly. Their first trial showed that experienced developers were 19% slower using early-2025 tools. Those developers believed they were 20% faster. The team is changing their experiment design due to significant selection effects. More recent data suggests different results regarding speed. The speed findings are debated and likely temporary. The gap in perception is the important finding. A study by Perry and colleagues (2023) supports this. Participants using AI assistants wrote less secure code but were more confident in its security. This is a failure of calibration.

What I would like us to measure

Sorting out vocabulary is worth the trouble if it changes what we do. Concretely:

Let us stop measuring overreliance as one thing. Complacency needs a behavioral monitoring measure against a normative baseline: review time per changed line, fixation patterns, and how far people navigate into called functions. Automation bias needs seeded defects and a split between omission and commission errors. Trust calibration needs reliability to be manipulated experimentally so that we can plot a calibration curve instead of reporting a Likert score. Deskilling needs longitudinal designs, and here we have essentially no data. The GPS and calculator analogies are suggestive but are not evidence.

For the study design itself, the guidelines we wrote for empirical studies involving LLMs cover reporting and reproducibility. This becomes acute when you manipulate model reliability as an experimental factor.

Two more things for the agenda. The step from autocomplete to chat to agent is a distinct shift. In the stages-and-levels framing, it moves us into the region where the literature predicts the sharpest out-of-the-loop effects. This is a testable prediction. We should also take disuse as seriously as misuse. Anyone who remembers why developers did not adopt static analysis tools will recognize what a noisy review bot does to a team's attention.

I would rather be argued with early than late on all of this. So two questions. Do you recognize the disuse pattern in your own team, where agent output is nominally reviewed and effectively ignored? And has anyone managed to close the feedback loop by attributing production incidents back to AI-authored changes well enough that trust could actually calibrate? If so, I would very much like to hear about it.

References

Bacchelli, A., & Bird, C. (2013). Expectations, outcomes, and challenges of modern code review. Proceedings of the 35th International Conference on Software Engineering (ICSE), 712–721.

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.

Baltes, S., Angermeir, F., Arora, C., et al. (2025). Guidelines for empirical studies in software engineering involving large language models. arXiv:2508.15503.

Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity. arXiv:2507.09089.

Dekker, S. W. A., & Hollnagel, E. (2004). Human factors and folk models. Cognition, Technology & Work, 6(2), 79–86.

Dekker, S. W. A., & Woods, D. D. (2002). MABA-MABA or abracadabra? Progress on human–automation co-ordination. Cognition, Technology & Work, 4(4), 240–244.

Elish, M. C. (2019). Moral crumple zones: Cautionary tales in human–robot interaction. Engaging Science, Technology, and Society, 5, 40–60.

Endsley, M. R., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394.

Hoff, K. A., & Bashir, M. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434.

Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R. (2013). Why don't software developers use static analysis tools to find bugs? Proceedings of the 35th International Conference on Software Engineering (ICSE), 672–681.

Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80.

METR (2026). We are changing our developer productivity experiment design. 24 February 2026.

Moray, N., & Inagaki, T. (2000). Attention and complacency. Theoretical Issues in Ergonomics Science, 1(4), 354–365.

Mosier, K. L., & Skitka, L. J. (1996). Human decision makers and automated decision aids: Made for each other? In R. Parasuraman & M. Mouloua (Eds.), Automation and Human Performance: Theory and Applications (pp. 201–220). Erlbaum.

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410.

Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253.

Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics – Part A, 30(3), 286–297.

Passi, S., & Vorvoreanu, M. (2022). Overreliance on AI: Literature review. Microsoft Technical Report MSR-TR-2022-12.

Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). Do users write more insecure code with AI assistants? Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2785–2799.

Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688.

Ruginski, I. T., Creem-Regehr, S. H., Stefanucci, J. K., & Cashdan, E. (2019). GPS use negatively affects environmental learning through spatial transformation abilities. Journal of Environmental Psychology, 64, 12–20.

Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006.