When OpenAI released o1 in late 2024, the narrative was irresistible: give the model time to think, and it gets dramatically smarter. Chain-of-thought reasoning — having the model work through problems step by step before answering — delivered stunning benchmark improvements. o3 subsequently destroyed records on ARC-AGI. DeepSeek-R1 showed the same gains could be reproduced open-source. Claude's extended thinking mode offered users a literal window into the model's deliberation process.

The engineering community rushed to adopt reasoning models for everything. And that's where things started going wrong.

Six months into widespread production deployment, a clearer picture is emerging: reasoning models are powerful but brittle in ways that standard benchmarks completely miss. They fail in predictable patterns that are actively costing companies in latency, cost, and accuracy. This piece breaks down exactly what's going wrong and how to think about when reasoning actually helps.

The Overthinking Problem Is Real

The most documented failure mode is what researchers call "overthinking" — where extended chain-of-thought reasoning actually degrades performance on tasks that should be simple. A model given a straightforward factual question will sometimes reason its way into the wrong answer by exploring possibilities that weren't worth exploring.

This happens because reasoning models are trained to generate plausible-looking deliberation, not necessarily correct deliberation. The chain-of-thought becomes a narrative the model tells itself, and narratives can drift. A well-trained non-reasoning model that pattern-matches to the right answer often outperforms a reasoning model that talks itself out of it.

The benchmark on which this is most visible is simple arithmetic and factual recall. On HumanEval-style coding benchmarks, reasoning models show consistent improvement. On questions like "what country is Paris in" or basic multiplication, reasoning can actually introduce errors that wouldn't appear with a direct response. The model considers edge cases that don't exist.

Latency That Breaks Real Applications

In controlled research environments, waiting 30-60 seconds for an answer is acceptable. In production, it's often a dealbreaker. Reasoning models have dramatically different latency profiles than standard models — and the variance is high, which is worse than just being slow.

Consider a customer support application where 95% of queries are handled in under 2 seconds by a standard model, but the team switches to a reasoning model hoping for better accuracy on complex edge cases. What they get instead: median latency jumps to 12 seconds, with P95 latency hitting 45+ seconds. The 5% of hard cases get better answers. The 95% of simple cases now feel broken.

The latency problem compounds with cost. Reasoning tokens — the internal chain-of-thought — are charged the same as output tokens on most providers. A reasoning model "thinking" through a simple query can generate 2,000-5,000 tokens of internal monologue before producing a 100-token answer. That's a 20-50x cost multiplier for the same output quality you'd have gotten from a direct response.

Teams that haven't done careful measurement often discover this only after their API bill spikes unexpectedly at scale.

The Calibration Gap

Standard LLMs have a well-understood calibration problem: they can express high confidence while being wrong. Reasoning models were supposed to fix this — more deliberation means better self-assessment, right?

In practice, reasoning models often have worse calibration on their own uncertainty. The extended thinking process creates an illusion of rigor. When a model reasons through five plausible hypotheses and selects one, it tends to present that answer with high confidence regardless of whether the reasoning was actually sound. The visible deliberation process makes the answer feel more trustworthy to both the model and the user — even when it isn't.

This is particularly dangerous in high-stakes applications. A model that says "I'm not sure" when it isn't sure is more useful than a model that shows its work while arriving at a confident wrong answer. Researchers at several AI labs have documented that reasoning models, when evaluated for calibration rather than accuracy, often perform similarly to or worse than their non-reasoning counterparts.

Adversarial Reasoning Injection

A newer failure mode that security-focused teams are increasingly worried about: reasoning models can be manipulated through their own thinking process in ways that standard models can't.

When a reasoning model processes a document containing carefully crafted text — say, a PDF submitted by a user — that text can include instructions that get incorporated into the model's chain-of-thought. Because reasoning happens in a privileged context (the model treats it as its own deliberation), injected reasoning can bypass safeguards more effectively than prompt injection into the standard input.

Early work on "reasoning injection" attacks suggests this is a genuine attack surface. The extended thinking capability that makes these models more powerful also creates a new vector. For any application where reasoning models process untrusted content — documents, web pages, user inputs — this needs to be on the security threat model.

When Reasoning Models Are Worth It

None of this means reasoning models are a mistake. They're genuinely transformative for the right problems. The issue is the mismatch between where they're being deployed and where they actually help.

Reasoning models excel at:

  • Hard mathematical and formal reasoning tasks — competition math, formal verification, complex proof steps. These are exactly the tasks where extended deliberation pays off and where simple pattern-matching fails.
  • Multi-step planning with explicit constraints — tasks where the model needs to track multiple interdependent requirements and reason about conflicts between them.
  • Code generation for non-trivial algorithms — not boilerplate, but genuinely complex algorithmic problems where working through edge cases matters.
  • Debugging with full context — when given a full stack trace and codebase context, reasoning models show substantially better root-cause analysis than direct-response models.
  • High-stakes single decisions — where you're making one important call and latency/cost are acceptable tradeoffs for accuracy.

Reasoning models are often worse for:

  • High-volume, low-stakes queries where speed matters
  • Simple factual retrieval
  • Creative tasks where constraints are loose (the model overthinks itself into safe, generic outputs)
  • Real-time conversational applications
  • Any task where cost-per-query matters at scale

The Routing Solution Most Teams Haven't Implemented

The practical answer — which a surprising number of production teams haven't implemented yet — is query routing: a fast, cheap classifier that evaluates incoming queries and sends them to reasoning or non-reasoning models based on predicted complexity.

This sounds obvious in retrospect. You don't send every customer support query to GPT-o3 the same way you don't send every task to your most senior engineer. A small trained classifier or even a lightweight LLM call can categorize query complexity with high accuracy, routing only the genuinely hard problems to expensive reasoning models.

Teams that have implemented routing typically see 60-80% cost reductions with equivalent or better overall accuracy — because the simple queries stop getting worse answers from overthinking models, and the hard queries actually get the reasoning power they need.

The Benchmark Illusion

Underlying all these failure modes is a deeper problem: the benchmarks that made reasoning models famous are systematically unrepresentative of production workloads.

ARC-AGI, MATH, and competition-programming benchmarks select for exactly the tasks where extended reasoning helps — hard, well-defined problems with clear right answers. Real production traffic is mostly simple, ambiguous, latency-sensitive queries interspersed with occasional hard problems. The ratio is probably 80/20 or 90/10 in favor of simple queries in most applications.

A model that improves 30% on MATH while getting slower and more expensive on everything else is not a 30% improvement for most applications. The AI field's benchmark infrastructure hasn't kept pace with the complexity of evaluating models for real deployment scenarios.

Some teams are now building private benchmark suites from their own production traffic — which is expensive and time-consuming but yields evaluation that actually predicts production performance. This is likely to become standard practice at serious AI engineering organizations.

What This Means for AI Engineering in 2026

Reasoning models aren't going away — they'll continue improving and the failure modes described here will be partially addressed in future versions. But the engineering discipline around deploying them needs to catch up to the hype.

The teams that are getting the most out of reasoning models in 2026 share a few characteristics: they measure before deploying, they build routing infrastructure, they evaluate on their own data rather than published benchmarks, and they treat reasoning models as a specialized tool rather than a universal upgrade.

The teams that are struggling have swapped their standard models for reasoning models wholesale, assumed benchmark improvements translate directly to production improvements, and are now dealing with cost overruns and latency complaints.

The lesson isn't that reasoning models are bad. It's that "AI that thinks" requires AI engineers who think carefully about when thinking is actually worth the cost.

Key Takeaways

  • Overthinking is a real failure mode — reasoning models can talk themselves into wrong answers on simple tasks
  • Latency and cost variance are production killers — measure before deploying at scale
  • Calibration often gets worse, not better — visible reasoning creates false confidence
  • Reasoning injection is an emerging security attack surface — take it seriously for document-processing applications
  • Query routing is the practical solution — send hard problems to reasoning models, simple ones to standard models
  • Benchmark improvements don't translate linearly to production — evaluate on your own traffic