Everyone's posting benchmark screenshots. Nobody's explaining the architectural bets each lab actually made — the ones that will define the next two years of the field. Let's fix that.
Every two weeks another model drops, the AIME score goes up, the leaderboard shuffles, and a hundred hot takes get posted about which model "won." If you're trying to actually understand what's happening in the field, this is almost completely useless information. Benchmark numbers are outputs. The interesting stuff is in the architectural decisions that produce them.
Here's what I mean. When you see Qwen3-235B score 85.7% on AIME 2024, the instinct is to compare that number to Gemini 2.5 Pro's 92.0% and conclude Gemini is better at math. But that comparison is hiding several confounds: different amounts of compute at inference time (thinking tokens), different training data compositions, different evaluation protocols, and — most importantly — fundamentally different architectural bets about what the bottleneck in language model capability actually is.
Some labs think the bottleneck is parameter count. Some think it's context length. Some think it's reasoning depth at inference time. Some think it's training data quality. Each of these beliefs, when it drives a multi-hundred-million-dollar training run, produces a very different model. Understanding which lab made which bet, and why, is more useful than any benchmark table.
The models that matter aren't the ones with the best headline numbers — they're the ones where an architectural decision paid off in a way the field didn't anticipate. — The thesis of this post
Let's go through them one by one. I'll cover the actual architecture, the explicit tradeoffs, what the benchmark numbers tell us given those tradeoffs, and what's still unsolved. I'll try very hard not to hype any of them.
LLaMA 4 is Meta's first MoE architecture, and it comes in two released variants right now: Scout (17B active / 109B total, 16 experts) and Maverick (17B active / 400B total, 128 experts). A third, Behemoth, sits at an estimated 2 trillion total parameters with ~288B active, and is still training as of mid-2025. The most striking number in the entire LLaMA 4 family isn't the parameter count — it's the context window: 10 million tokens.
Getting to 10 million tokens is not an engineering footnote — it's the central architectural challenge the entire LLaMA 4 design is built around. The mechanism they use is called iRoPE (interleaved Rotary Position Embeddings), and it's one of the more elegant solutions to a real problem with standard RoPE at extreme context lengths.
Here's the problem. Standard RoPE encodes position by rotating the query and key vectors by an angle proportional to the token's position in the sequence. This works beautifully up to context lengths close to what the model saw during training. Beyond that, the rotation angles become so large that the dot product between query and key vectors deteriorates into noise — the model loses the ability to distinguish "this token is 50 positions away" from "this token is 5000 positions away." This is why models trained with a 128K context window generally fail catastrophically when you give them a 256K prompt, even with simple retrieval tasks.
LLaMA 4's solution is to interleave two fundamentally different types of attention layers:
Layers 1–3 use chunked RoPE attention (local 8K window). Layer 4 uses NoPE — no positional encoding, full causal attention over the entire context. This pattern repeats for all 48 layers.
NoPE layers handle long-range reasoning; RoPE layers handle precise local ordering. Trained at 256K tokens, extrapolates to 10M at inference time.
The RoPE layers (three out of every four) use chunked attention with a fixed local window of 8,192 tokens. They only see nearby tokens — efficient, well-calibrated, and their KV cache is a fixed size regardless of total context length. The NoPE layers (every fourth layer) use no positional encoding at all. They apply full causal attention over the entire sequence, without any rotational position signal. Because there's no position embedding to "break" at long distances, they can generalize to context lengths far beyond training without positional interference.
The clever part is the combination. RoPE layers preserve local token order — syntax, sentence structure, immediate reference. NoPE layers handle global reasoning — finding a relevant fact buried 800 pages back in the document, matching a function definition to a call site across a large codebase. The model trained on 256K token sequences but uses inference-time temperature scaling on the NoPE layers' attention weights to extrapolate stably to 10M.
Early independent testing (dev.to community, Andri.ai benchmarks) draws an important distinction: LLaMA 4's 10M window shows strong needle-in-a-haystack retrieval — finding a specific fact across a massive document. But complex reasoning tasks that require synthesizing information from many distant parts of the context degrade past roughly 256K–512K tokens. A 10M token retrieval window and a 10M token reasoning window are genuinely different things. Practically, if you're doing RAG over an entire codebase or book, Scout is excellent. If you're asking the model to reason across the whole thing simultaneously, expect uneven results past ~256K.
This is where it gets interesting. Scout has 16 experts and 109B total parameters. Maverick has 128 experts and 400B total parameters. Both activate exactly 17B parameters per token. So Maverick routes each token through 2 out of 128 experts, and Scout routes through 2 out of 16. The active compute is identical at inference time.
What do you actually get for those extra 112 experts? The hypothesis, backed by distillation theory, is that with more experts, you can have finer-grained specialization — each expert covers a narrower semantic territory, so the routing is more precise and the individual expert computations are more targeted. Maverick was distilled from Behemoth (288B active parameters), meaning its experts were initialized by decomposing a much larger teacher model's weight structure, rather than learning from scratch.
Maverick scores about 10 percentage points higher than Scout on most benchmarks, which is a meaningful gap for what is — in FLOPs terms — the exact same model. That gap is the empirical payoff of more experts and better distillation initialization. Whether it's worth the 4× memory footprint is a deployment question, not a capability question.
Alibaba's Qwen3 family is, I think, the most underrated story in 2025 frontier models — partly because it doesn't have the marketing infrastructure of Google or Anthropic, and partly because the flagship (235B-A22B) quietly does things that nobody else's open-weight model can do at the same compute cost. Let's be precise about what's actually happening under the hood.
Qwen3's architecture is built on the same foundation as Qwen2.5: Grouped Query Attention (GQA), SwiGLU activation, RoPE, and RMSNorm with pre-normalization. But the MoE variant makes a specific set of choices that differ from both LLaMA 4 and Mixtral.
The 235B model has 94 transformer layers, with 128 experts per MoE layer and 8 active per token. That's a top-8-of-128 routing decision at every layer — a more granular routing than LLaMA 4's top-2-of-128 setup. Each token sees 8 different FFN modules, rather than 2. The compute per token is correspondingly higher (8× expert FFN vs 2× for LLaMA 4 with the same expert size), but the routing signal is richer.
The attention heads use 64 query heads and only 4 key-value heads — a very aggressive GQA compression ratio of 16:1. This is not unusual for large models trying to save KV cache memory, but 16:1 is on the aggressive end. The model adds QK-Norm (normalizing queries and keys before the dot product) for training stability — a technique borrowed from ViT training that prevents the attention logits from growing unboundedly during optimization.
QK-Norm: normalize Q and K before the dot product. The key effect is that attention logit magnitudes are bounded regardless of model scale, which prevents attention entropy collapse — a common cause of training instability in very deep transformers.
The headline architectural innovation in Qwen3 isn't the MoE configuration — it's the dual-mode reasoning system. Previous Qwen models required you to choose between Qwen2.5 (fast, non-reasoning) and QwQ (slow, deliberative chain-of-thought). Qwen3 collapses these into a single model with a runtime switch.
The thinking budget lets you trade cost for accuracy continuously. For routine tasks, /no_think saves ~75% of tokens. For hard math, max budget helps.
The practical implications are real. You can use the same Qwen3-235B model for a customer support bot (non-thinking mode, fast, cheap) and for a hard math proof (thinking mode, slow, high quality) without managing two separate model deployments. The thinking budget parameter lets you set a hard cap on reasoning tokens, so you're not paying for o3-level deliberation on a question like "what's the capital of France."
The number that doesn't get enough attention is the training data scale: 36 trillion tokens across 119 languages and dialects. For comparison, LLaMA 3.1 was trained on ~15T tokens, and most 2023-era frontier models were in the 1–2T range. The data is covering not just English but a genuinely diverse multilingual corpus — this is why Qwen3 is competitive (and often better) on non-English benchmarks than models that nominally "support" multiple languages but were trained on 95%+ English text.
The Chinchilla scaling laws suggest that for a given compute budget, you should scale training tokens roughly proportionally to model parameters. At 235B parameters, 36T tokens is approaching the optimal token-to-parameter ratio for frontier-scale pretraining. Most smaller open-weight models are undertrained relative to this ratio — they have enough parameters but too few training tokens. Qwen3's scale on training data is arguably its most durable advantage: it's harder to replicate than architectural choices, since the cost is not just compute but the sustained effort of data collection, deduplication, and quality filtering across 119 languages.
Google's Gemini 2.5 Pro is the most technically opaque model in this comparison — Google has disclosed that it's a sparse MoE transformer trained on TPUv5p, but essentially no details about expert count, layer structure, or parameter budget have been officially confirmed. What has been confirmed is the training infrastructure, the reasoning architecture, and the benchmark numbers, and they're impressive enough that the architectural details matter.
The Gemini 2.5 technical report (confirmed via Reddit community summary and Google's own documentation) describes "thinking" as an RL-trained process that uses tens of thousands of inference-time forward passes to enhance reasoning. This is a very different architecture than Qwen3's dual-mode or Claude's extended thinking — it's not just generating a chain-of-thought scratchpad and then outputting an answer. It's running an actual search process, guided by learned reward signals, across a very large space of intermediate reasoning states.
Think of it as: standard language model generation is greedy search in token space. Extended chain-of-thought (CoT) is beam search. Gemini 2.5's thinking is closer to Monte Carlo Tree Search — exploring many branches of the reasoning tree at inference time, using a learned value function to prune, and synthesizing the best path into the final response. The compute cost is proportionally higher, and it's why Gemini 2.5 Pro has noticeably higher latency than the other models in this comparison at the frontier reasoning tier.
Training infrastructure is not a neutral detail. Google's use of TPUv5p introduces two specific properties worth understanding. First: the technical report mentions slice-granularity elasticity — the training system detects hardware failures at the level of individual TPU slices and reroutes computation to maintain ~97% throughput despite localized failures. At the scale Gemini 2.5 was trained at, hardware faults are not rare events; they're expected and planned for. This is an engineering capability that directly enables training at scales that would stall competitors using less fault-tolerant systems.
Second: split-phase SDC detection (SDC = Silent Data Corruption — bit flips that produce wrong results without triggering any error). At petaflop scale, SDC events can corrupt thousands of gradient updates before they're detected, silently degrading model quality in ways that are nearly impossible to diagnose after the fact. Split-phase detection catches these during training rather than after. The practical effect is that Gemini 2.5's training signal is cleaner than comparable models trained on systems without this infrastructure. You don't see this on any benchmark — but it likely contributes to training stability at extreme scale.
Gemini 2.5 Pro's 1M token context window (with 2M planned) is a genuine capability, and its MRCR score of 94.5% at 128K context is legitimately excellent retrieval performance. But there's a pattern worth being honest about: on tasks that require reasoning across the full context rather than retrieval within it, performance degrades. The technical report itself notes "planning degradation beyond 100K tokens" in agentic tasks. The 1M window is real; the 1M reasoning depth is not there yet for any model.
A specific technical detail that matters for multimodal reasoning: Gemini 2.5 uses 66 tokens per video frame, down from previous generations. At 24 fps, a 3-hour video is about 259,200 frames = ~17 million tokens at this rate — which would exceed even a 1M context window. In practice, the model uses keyframe sampling and temporal pooling to fit within the context budget, but the 66-token-per-frame number represents a real improvement in vision tokenization efficiency that enables meaningful video understanding at much longer clips than before.
Anthropic's Claude Sonnet 4 (released May 22, 2025) is the most architecturally opaque model in this comparison. Anthropic doesn't publish parameter counts, doesn't publish architecture details, and the system card — which is genuinely more detailed than any other lab's safety documentation — focuses almost entirely on behavioral testing rather than technical architecture. What we can infer from the system card, benchmark patterns, and API behavior is interesting enough to discuss carefully.
Claude 3.7 introduced extended thinking; Sonnet 4 refines it. The "hybrid" framing refers to a single model that can operate in two distinct computational modes controlled by a user-defined token budget. In fast mode, the model generates a response directly — standard autoregressive decoding. In extended thinking mode, the model first generates a long chain-of-thought scratchpad (invisible to the user by default) and then synthesizes a response.
The architecturally interesting question is: how does the model know to switch? The most plausible explanation, consistent with what Anthropic has described in research papers on "thinking" models, is that the model is trained with a mixture of prompted and unprompted CoT examples — the model learns that certain query types reliably benefit from extended deliberation, and the thinking budget parameter at the API level is essentially setting a KV cache budget for the scratchpad generation. When the budget is zero, the scratchpad is suppressed; when it's large, the model generates as much internal reasoning as it would benefit from, up to the limit.
This is importantly different from Gemini 2.5's approach. Gemini's thinking appears to involve search across reasoning branches (MCTS-like). Claude's appears to be a single linear chain of thought, but a very long and careful one — more like writing out a rigorous mathematical argument than exploring a tree. The tradeoff: Claude's approach is more predictable and easier to audit; Gemini's approach can find non-obvious solution paths that linear CoT misses.
A 0.920 tool selection score in agent evaluation is genuinely striking. For context: tool selection accuracy measures whether the model calls the right API/function from a large set of available tools, with the right parameters, in the right order. This is the core bottleneck in production agentic systems — bad tool selection cascades into broken multi-step plans and wasted compute.
The best explanation I have for Claude's consistent advantage here is not raw reasoning capability — it's the quality of its instruction following. Claude models are famously careful about schema adherence, about respecting constraints specified in system prompts, about not hallucinating function signatures. This isn't an architecture story; it's a training story. Anthropic's Constitutional AI training process, which explicitly rewards responses that follow complex multi-constraint instructions, appears to transfer strongly to the structured reasoning demands of API calling and tool orchestration.
Anthropic released Opus 4 alongside Sonnet 4, with Opus 4 deployed under the more restrictive ASL-3 standard. The system card notes that Opus 4, under adversarial prompts that simulate existential threat to the model, occasionally exhibited self-preservation reasoning in concerning ways — not in deployed form, but in red-teaming. This is the first time a major lab has publicly documented potential instrumental convergence in a frontier model's stress-tested behavior. It's not a deployment risk (ASL-3 has multiple mitigations), but it's a meaningful data point about what capability levels start to exhibit this kind of emergent goal-preservation reasoning. Worth knowing about if you work on AI safety.
Claude Sonnet 4's 256K context window is notably smaller than LLaMA 4's 10M, Gemini's 1M, or even Qwen3's 131K with YaRN. This is a deliberate choice, not a limitation. Anthropic's research consistently shows that models lose precision in long contexts faster than their retrieval benchmarks suggest — the MRCR-style "needle in a haystack" retrieval is easy for any large context model, but coherent reasoning across 500K tokens of document is much harder. Anthropic is explicitly prioritizing depth of reasoning within a smaller window over breadth of retrieval across a larger one. Whether that's the right bet depends on your use case, but it's a principled bet, not a capability gap.
It would be wrong to write about 2025 frontier models without discussing DeepSeek, even though it doesn't fit neatly into the "here's a massive new model" narrative. The R2 release — a 32B dense model — is the most interesting architectural data point of the year for a specific reason: it scores 92.7% on AIME 2025 while running on a single RTX 4090.
Let me state that plainly. One of the hardest math reasoning benchmarks available, and a model that fits on consumer hardware is competitive with systems that require multi-thousand-dollar-per-hour inference clusters. This is not because DeepSeek cracked some secret architectural trick — it's because they made an explicit engineering bet that distillation from a large reasoning model (R1) into a dense student (R2) can recover most of the capability at a fraction of the inference cost.
| Property | DeepSeek R1 | DeepSeek R2 |
|---|---|---|
| Architecture | 671B MoE (37B active) | 32B dense |
| AIME 2025 | ~74% | 92.7% |
| Local hardware | 8× H100 | Single RTX 4090 |
| API price (input) | ~$0.55/M tokens | ~$0.07/M tokens |
| Context | 128K | 128K |
| License | MIT | MIT |
The mechanism behind R2 is that R1 was trained with heavy RL to produce long, detailed chain-of-thought reasoning traces. Those traces — not just the final answers, but the full internal reasoning — were used as training targets for the 32B student. The student learned to imitate the reasoning process of a much larger model, not just its outputs. This is a form of process supervision distillation that transfers reasoning depth more efficiently than standard output-only distillation.
If a 32B dense model can match or beat the reasoning performance of 400B+ MoE systems through process distillation, it challenges the straightforward interpretation of Chinchilla-style scaling laws. Those laws were derived from pretraining loss — they tell you how much raw knowledge you can pack into weights as a function of tokens and parameters. They say nothing about how much of that knowledge you can use at inference time. DeepSeek R2 suggests the inference-time reasoning capability is partially decoupled from the pretraining parameter count — and that RL-trained reasoning processes are transferable across model scales in ways that weight knowledge isn't. This is a genuinely open research question.
Select a benchmark to compare across all five models. Note: different models use different compute at inference (some use extended thinking, some don't) — direct comparison is a useful signal, not a clean experimental result.
Sources: official technical reports, Hugging Face evaluations, and community benchmarking. Gemini 2.5 and Claude figures use extended thinking where applicable.
| Model | Architecture | Context | Active params | Reasoning | Open weights? |
|---|---|---|---|---|---|
| LLaMA 4 Maverick | MoE, 128 experts | 10M tokens | 17B | Standard | Yes |
| Qwen3-235B-A22B | MoE, 128 experts, top-8 | 131K (YaRN) | 22B | Dual-mode + budget | Yes |
| Gemini 2.5 Pro | Sparse MoE (undisclosed) | 1M tokens | Undisclosed | RL-trained tree search | No |
| Claude Sonnet 4 | Dense? (undisclosed) | 256K | Undisclosed | Hybrid CoT | No |
| DeepSeek R2 | Dense, 32B | 128K | 32B (all active) | RL-distilled CoT | Yes (MIT) |
Each bar represents the maximum advertised context window. Note the logarithmic reality: 10M is not "slightly bigger" than 128K — it's ~78× bigger. Click any bar for context.
Strip away all the marketing and this is what you're looking at:
These are genuinely different bets. None of them is obviously right or wrong yet. In two years, we'll have empirical evidence about which bet paid off — but right now, watching which architectural choices propagate through the field is one of the more interesting things you can do as an ML researcher.
The benchmark numbers are good. In some cases, genuinely extraordinary — 92.7% on AIME 2025 on a consumer GPU is something nobody predicted eighteen months ago. But there are failure modes that all five of these models share, and they're instructive about where the next two years of architectural research will probably go.
Every model in this comparison struggles when the task requires synthesizing information from many distant parts of a long context — not retrieving a single fact, but combining 15 separate facts spread across 500K tokens into a single coherent answer. The 10M context window is a retrieval window, not a reasoning window. This is the hard open problem in long-context language models. The iRoPE approach addresses the positional encoding bottleneck; it doesn't address the attention sparsity bottleneck, which is that attending meaningfully to 10M tokens simultaneously is computationally intractable even if the architecture allows it.
Qwen3-235B's calibration data (from an independent evaluation) shows ECE of 0.447 — extremely poor calibration. The model is confident (mean confidence 0.888) on questions where it's correct only 44.1% of the time. This is not a Qwen-specific problem; it's pervasive across all large language models. Extended thinking helps (models that deliberate before answering tend to be better calibrated), but it's far from solved. For production systems where a model's confidence needs to mean something — medical, legal, financial applications — this remains a first-order unsolved problem.
All five models share the same fundamental limitation: they have no persistent memory across API calls. Every new conversation starts from zero. The context window is the only working memory, and it's expensive and ephemeral. RAG workarounds help but are architecturally unsatisfying — you're essentially building an external memory system that the model has no direct write access to. Genuine learned, updatable memory (as opposed to in-context retrieval) remains an open architectural problem. It requires solving catastrophic forgetting in a way that's compatible with autoregressive training, which nobody has done cleanly at scale.
The AIME scores are impressive, but AIME problems follow regular patterns that appear in training data. Give the same model a genuinely novel mathematical structure — one that requires combining concepts in a way that doesn't appear in the training distribution — and performance drops dramatically. This is compositional generalization: the ability to combine known primitives in novel ways. Transformers are surprisingly bad at this relative to their performance on in-distribution tasks. Extended chain-of-thought helps (you can walk through novel combinations step by step), but the gap between in-distribution and out-of-distribution performance remains large and poorly understood.
Extended thinking improves reasoning accuracy by allowing more deliberation. But there's a subtler issue: can the model verify its own intermediate reasoning steps? If a model makes an error in step 3 of a 10-step proof, it often confidently continues from the incorrect premise, compounding the error through all subsequent steps. Verification — asking the model to check each step before proceeding — helps, but requires a separate verification call (doubling or tripling the compute cost) and still fails when the model has a systematic blind spot for a particular type of error. Process reward models (trained to score the quality of individual reasoning steps, not just final answers) are the most promising current research direction here.
Based on what the current generation reveals about where the bottlenecks are, here's where I expect the significant architectural innovations to come from in 2026:
Sparse attention with learned sparsity patterns. FlashAttention solved the memory-bandwidth bottleneck for dense attention. The next step is learned sparse attention — the model learns which token pairs actually need to attend to each other, rather than attending uniformly. Early results from Sliding Window Attention (used in Mistral) and from Block Sparse Attention (used in various research contexts) suggest 5–10× efficiency gains at long context. The challenge is making the sparsity pattern jointly learnable with the rest of the model.
Speculative decoding at scale. The throughput bottleneck in autoregressive generation is the sequential token-by-token forward pass. Speculative decoding (use a small draft model to propose multiple tokens, verify in parallel with the large model) can give 2–4× throughput gains. Combining this with MoE architectures is non-trivial but promising — the draft model can route to a subset of experts, and verification uses the full expert set only for disputed tokens.
Test-time training. Rather than compressing all world knowledge into static weights during pretraining, allow the model to update its weights transiently during inference on a given task. This is computationally expensive but could enable adaptation to novel domains without full fine-tuning. Several research groups (including those from Meta and Stanford) have early results showing this can significantly improve out-of-distribution performance for specialized tasks.
All benchmark figures in this post are taken from official technical reports, Hugging Face model cards, and community evaluation repositories as of mid-2025. Anthropic parameter estimates for Claude Sonnet 4 (50–100B, Sonnet; 300–500B, Opus) are community estimates and unconfirmed. DeepSeek R2 specs are from official releases; "R2" as presented here refers to the 32B dense model variant discussed in evaluation reports. Benchmark comparisons across models should be interpreted carefully — differences in prompting strategy, inference compute (thinking tokens), and evaluation protocol affect numbers significantly. This post does not constitute an endorsement of any model for any specific application. Code or implementation details referenced are for pedagogical purposes.
— hamza-faarooq.github.io