Research Governance

Automation Bias

Automation bias is the tendency for people to over-rely on, accept, or follow automated system outputs even when those outputs may be wrong, incomplete, or inappropriate.

Concept facts

Automation bias is the tendency for people to over-rely on, accept, or follow automated system outputs even when those outputs may be wrong, incomplete, or inappropriate.

Also known as
Automation overreliance, over-trust in automation, automation-induced complacency, inappropriate reliance.
Used in
Human factors, human-AI interaction, UX research, decision-support evaluation, safety-critical systems, service design, and risk review.
Interpret with
Trust calibration, confidence signal, human oversight, explainability, automated decision, algorithmic accountability, and user trust.

Plain-language explanation

Automation bias happens when people give too much weight to a system output because it comes from automation.

For example, a staff member may approve a system recommendation without checking evidence, or a user may accept an automated result because it looks official.

Automation bias is different from algorithmic bias. Algorithmic bias concerns biased system output or decision patterns. Automation bias concerns how people respond to automated output.

Why it matters

AI and automation are often presented as fast, objective, or efficient. This can lead users or staff to trust outputs more than they should.

Automation bias can cause wrong decisions, missed exceptions, poor escalation, service exclusion, and harm in high-stakes journeys.

In public-service, healthtech, fintech, telco, and development-sector contexts, automation bias can affect eligibility, fraud review, triage, case prioritization, and support decisions.

Use contexts

Automation bias is used when evaluating:

  • staff-facing decision-support tools
  • AI-assisted triage or prioritization
  • automated eligibility or verification systems
  • fraud detection and risk scoring
  • recommendation systems in high-stakes services
  • automated support routing
  • human oversight and review workflows
  • training, warning, or confidence-signal design

Application guidance

Evaluate whether users or staff understand the system’s limits.

Check whether the interface presents automated output as certain, final, or authoritative when it is not.

Design confidence signals, explanations, and review prompts carefully. Poorly designed signals can increase overreliance.

Test whether staff can detect wrong or questionable outputs.

Make human oversight meaningful. Reviewers need time, evidence, authority, and escalation routes.

Monitor decisions for patterns where automated output is accepted without adequate review.

Practical example

A clinic uses an AI-assisted system to classify referral urgency. Staff begin to follow the urgency label without checking referral notes because the system is usually correct.

A usability study shows that staff are less likely to challenge the label when the interface uses strong visual certainty. The service redesign adds uncertainty signals, evidence summaries, and escalation prompts.

The UX consequence is reduced patient-risk exposure, better staff decision-making, and lower overreliance on automated triage.

Interpretive boundaries

Automation bias is not the same as algorithmic bias.

It is also not the same as general user trust. It specifically concerns inappropriate reliance on automated output.

Automation bias can affect both experts and non-experts, especially under time pressure, workload, uncertainty, or high institutional trust in the system.

Applied at Userhub

Userhub evaluates automation bias where AI or automated outputs influence users, staff, or service decisions.

In UX Lab work, automation bias may be assessed through usability testing, staff workflow review, decision-support evaluation, and human-centered research.

See Userhub UX Lab for applied UX research and evaluation context.

Sources and references

Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. https://doi.org/10.1136/amiajnl-2011-000089

Lee, J. D., & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300233

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://www.nist.gov/itl/ai-risk-management-framework

Cite this entry

APA

Userhub. (2026). Automation Bias. UX Reference. https://userhub.com.bd/reference/automation-bias/