Common Pitfalls: What Most People Get Wrong
From over-reliance to privacy mistakes, learn the errors that even experienced users make — and how to build better habits.
Automation Bias: Over-Trusting What AI Produces
Automation bias is the well-documented human tendency to over-trust automated systems, especially when those systems sound confident. Studies in aviation, medicine, and finance consistently show that people defer to automated recommendations even when the automation is clearly wrong — sometimes even when they knew the correct answer before the automation provided its (incorrect) one. LLMs are particularly susceptible to causing automation bias because they are extremely fluent. Poorly written text signals 'check this carefully.' Beautifully structured, confident text signals 'this is authoritative.' The fluency and confidence of AI output is not correlated with its accuracy. A hallucinated medical fact reads just as well as a correct one. Critical thinking has to be applied consistently — not only when the output 'feels' wrong.
Fluent, confident-sounding AI output is not more accurate. Apply critical thinking uniformly, not only when something 'feels off.'
Key Takeaways
- ✓Automation bias causes people to over-trust AI output — fluency is not accuracy
- ✓In high-stakes domains, every specific AI claim needs verification from primary sources
- ✓Treat commercial AI tools as potentially public — never paste confidential, sensitive, or identifying information
- ✓Skill atrophy is a real risk — use AI to amplify skills, not replace developing them
- ✓The best AI users apply consistent critical thinking, not selective skepticism
🗂️ Real-World Cases
Two lawyers in New York submitted a legal brief to a federal court that contained citations to over half a dozen court cases. All of the cases were fabricated by ChatGPT. Neither lawyer verified the citations before submitting to the court.
The judge discovered the fabrications. Both lawyers faced sanctions, professional conduct hearings, and significant public embarrassment. The case became a landmark example of the real-world consequences of AI hallucination in professional contexts.
In any professional or high-stakes context, treat AI as a drafting assistant — never as a source of verified facts. Every citation, every statistic, every specific claim must be independently verified before it matters.
Multiple studies in 2023–2024 tested the accuracy of popular AI tools for medical symptom assessment. Several found error rates of 30–50% when tested against verified clinical diagnoses, with some cases where AI provided confidently wrong advice about serious conditions.
In documented cases, patients delayed seeking care for serious conditions because AI had assessed their symptoms as low-risk. Medical organizations issued guidance warning against using generalist AI tools for medical decision-making.
AI has no access to your physical body, no ability to run tests, and no malpractice liability. For medical concerns, consult qualified healthcare professionals. AI can help you understand general health concepts or prepare questions for a doctor — it should not be your diagnostic tool.