The Efficiency Turn: Why This Week's Most Interesting AI Models Were About Speed, Not Size

Diffusion text models, sparse 500-billion-parameter open weights and a 4GB on-device agent all point the same way: AI's next frontier is how much intelligence you get per token, per second and per gigabyte.

TL;DR

Inception's Mercury 3 (19 October) is a diffusion language model that claims more than 1,000 tokens per second and quality on par with or better than GPT-6 Luna and Claude Haiku 5.5, at launch pricing of $0.10/$0.35 per million input/output tokens [1][2]. The benchmarks are self-reported; no independent tests had been published at the time of writing [2].
Reflection's Beam 1.0 (16 October) is an Apache 2.0 open-weight sparse mixture-of-experts (MoE) model: 501B total parameters, about 23B active per token, 1M-token context [4][5][6].
NVIDIA's Nemotron 3.5 Bolt E4B (23 October) packs a multi-turn conversational agent into roughly 4GB, aimed at RTX PCs, Jetson boards and robots [7].
Cordoval's view: these releases come from very different labs, but they share one design goal. The question used to be "how capable is the model?" Now it is "how much capability do I get for my latency, memory and money budget?" That changes which models win in production, and it changes how we should read benchmarks.

A busy fortnight, and a pattern within it

October 2026 has been crowded. Benchlm.ai's release tracker lists 42 confirmed releases from 31 providers this month alone [8]. Most coverage goes to the frontier: Google made Gemini 4 Argon generally available on 13 October, calling it its model for "deep work" and offering an industry-leading 1M unified input-output token limit [9]. Anthropic shipped Claude Haiku 5.5 on 7 October, which it describes as its cheapest, fastest and most capable small model. Anthropic says Haiku 5.5 costs about 75% less to run on average than Haiku 4.5, and it halved cache-read prices on Sonnet 5.5 [10].

Look at the past week specifically, though, and the most interesting releases aren't trying to top a leaderboard. They're each attacking a different cost:

Release	Date	What it attacks	Headline claim
Mercury 3 (Inception)	19 Oct	Latency	1,068 tok/s on NVIDIA GPUs [1]
Beam 1.0 (Reflection)	16 Oct	Compute per token, and ownership	501B total / 23B active, Apache 2.0 [5][6]
Nemotron 3.5 Bolt E4B (NVIDIA)	23 Oct	Memory and device footprint	Agent in ~4GB, up to ~80% fewer output tokens vs peers [7]

We'll go through each, then explain why we think they add up to a real shift.

1. Mercury 3: diffusion comes for text

Almost every chatbot you've used is autoregressive: it writes one token at a time, and each token depends on the ones before it. The Next Web's explainer puts the alternative well. A diffusion model "starts with rough, noisy output and refines many tokens at once, in parallel, until the text resolves" [2]. That is the same idea behind image generators, applied to language.

Inception's announcement makes three main claims [1][3]:

Speed: 1,068 tokens per second on NVIDIA GPUs, which it says is 4–8× faster than comparable speed-optimised models.
Quality: "on par or better than frontier speed-optimized models like GPT-6 Luna and Claude Haiku 5.5."
Scaling: more than 40× the parameters of Mercury 1, with benchmark intelligence "risen 3x" from Mercury 1 (mini) to Mercury 3, at a similar speed. The company states plainly that "the scaling laws that drove progress in autoregressive models hold for diffusion too" [1].

Context grows to 1M tokens, up from 260K in Mercury 2.5. List pricing is $0.20 per million input tokens and $0.70 per million output tokens, halved at launch to $0.10 and $0.35 [1][3]. The model is OpenAI-API compatible and available through Inception's API, OpenRouter, Baseten and Vercel's AI Gateway [1][4].

The caveat matters. The Next Web notes that "the lab's benchmarks are its own, and no outside group has tested Mercury 3 yet" [2]. Inception also hasn't said how long the launch discount will last [2].

Cordoval's view: The most important sentence in the launch isn't a benchmark number. It's the claim that diffusion follows scaling laws. If that holds up under independent testing, diffusion stops being a niche trick for fast autocomplete and becomes a credible second architecture for general-purpose language models. The commercial argument is also clever. Reasoning models "think" by generating long hidden token streams, so a system that produces tokens in parallel gains more from that trend, not less. (That is our inference from the architecture; Inception has made a similar argument publicly.) We'd hold off on any procurement decision until third-party evaluations arrive. Still, this is the first time a non-autoregressive text model has been positioned head-to-head against the major labs' small models rather than as a curiosity.

2. Beam 1.0: frontier scale you can own

Reflection, a newer US lab, published Beam's weights on Hugging Face on 16 October under the permissive Apache 2.0 licence [5][6]. According to the model card [6]:

501.3B total parameters, 23.4B active per token (sparse MoE)
52 layers (3 dense + 49 MoE), 256 experts, gated grouped-query attention plus sliding-window attention
1M-token context, text in and text out, developed predominantly for English
Knowledge cutoff of 30 June 2026
Weights in BF16, FP8 and NVFP4

Hugging Face's launch post adds that 8 experts are routed per token, reasoning effort has five settings from "low" to "max", reasoning can be kept in context across turns, and the release includes a speculative-decoding drafter (DFlash2) plus day-0 support in transformers, SGLang and vLLM [5].

On training, Reflection's earlier preview says Beam was pretrained on 23.8 trillion tokens. Its reinforcement-learning run "generated over 100 million rollouts on 10.5K NVIDIA GB300 GPUs over 4 weeks" [11]. The company presents the release as "an open operating system for agents", backed by more than 40 partners and integrations [4].

Cordoval's view: The interesting number here is 23 billion, not 501 billion. Sparse MoE means you pay to store a very large model but only compute a mid-sized one on each token. Add official FP8 and NVFP4 checkpoints and a speculative drafter, and the message is clear: Reflection isn't only competing on capability. It's competing on cost of ownership. For UK and European organisations with data-residency or sovereignty requirements, a permissively licensed model of this scale that can be self-hosted and fine-tuned is worth evaluating seriously. The usual caution applies. A 500B-parameter model still needs serious multi-GPU hardware even when quantised, so "own it" means "own it if you have a data centre budget", not "run it on a laptop."

3. Nemotron 3.5 Bolt E4B: an agent in 4GB

At the other end of the scale, NVIDIA released Nemotron 3.5 Bolt E4B on 23 October. It's an open small model built on Google's Gemma 4 E4B with Nemotron post-training, "optimized for speed and intelligence per GB in conversational agents" [7]. The target uses are voice and device assistants, in-game characters, and robotic or industrial interfaces running on GeForce RTX, Jetson and DGX Spark hardware [7].

The claims NVIDIA puts forward [7]:

About 40% higher average quality than its previous Nemotron 3 Nano 4B across its chosen conversational-agent benchmarks, and about 10% above other peers it evaluated.
Up to ~80% fewer output tokens than peers in reasoning-enabled comparisons.
A 64K context window, support for 35 languages, configurable reasoning, and a "conversational mode" for short, speech-ready replies.
At Q4_K_M quantisation on an RTX 5070, about 3.5GB VRAM, 548ms time to first token and 133 tokens/s on a 3K-token prompt.

NVIDIA's evaluation section is more candid than most launch posts. It argues that "a model that excels at agentic coding is often not the best fit for a fast home assistant", and that "current benchmarks only partially capture what makes these agents work well" [7]. Its own tables don't show a clean sweep either. On Tau2-Bench (multi-turn customer tasks), Bolt E4B scores 55.1 against 75.1 for Qwen 3.5 4B. On RULER at 64K, it scores 87.0 against 95.1 [7]. It wins clearly on instruction hierarchy (IHEval 84.8 vs 60.6) and grounding benchmarks such as GaRAGe (42.6 vs 24.7) [7].

Cordoval's view: "Fewer output tokens" is an underrated metric. For on-device and voice agents, every unnecessary token costs latency, battery and user patience. NVIDIA says it trained for brevity on purpose, increasing length penalties during RL to control verbosity [7]. That's a sign the industry is starting to treat conciseness as capability. We also welcome the mixed benchmark results. A vendor publishing tables where a rival beats it on some rows is more believable than one claiming to lead everywhere.

Why this is a turn, not a coincidence

Here's the argument in brief (all of it Cordoval's view).

1. Agents changed the economics. One chat reply might be a few hundred tokens. An agent working through a browser task, a codebase or a customer case can generate thousands of tokens across dozens of steps, many of them hidden reasoning. When usage multiplies like that, the cost per token and the time per step stop being rounding errors and become the product. Anthropic presents Haiku 5.5 as a subagent alongside its larger models and as a fit for browser use [10]. Every launch in our table names agents or real-time interaction as the target workload [1][4][7].

2. Three different routes, same destination. Inception changes the generation algorithm (parallel diffusion). Reflection changes which parameters do work on each token (sparse MoE plus low-precision formats). NVIDIA changes the training objective (brevity, instruction hierarchy, grounding) and squeezes the result into consumer memory. These are independent levers, so they can stack. A sparse, quantised, brevity-tuned diffusion model is a plausible 2027 product.

3. The frontier labs are pricing for it too. Google launched Argon at an introductory $2/$10 per million tokens with cached input 95% off [9]. Anthropic paired Haiku 5.5 with lower Sonnet cache-read prices [10]. Even at the top end, cost per unit of work is now a headline feature.

4. Benchmarks need to catch up. Most leaderboards still report a single quality score. The releases above ask to be judged on quality at a given latency, memory budget and price. NVIDIA explicitly plots quality against time to first token and VRAM [7]. Inception sells itself on a "Pareto frontier" of speed and intelligence [1]. We think the useful evaluation unit for 2027 will be something like "task success per pound per second", and anyone choosing models should start measuring that way now.

What we'd do with this, practically

If you're building on these models (Cordoval's recommendations):

Separate "think" from "talk". Route hard planning to a large model and high-volume or real-time steps to a fast one: a Haiku-class or diffusion model in the cloud, or a Bolt-class model on the device.
Test latency-sensitive paths with Mercury 3, but verify it yourself. The launch discount and free-token offer make it cheap to try [1]. Don't treat the self-reported comparisons as settled [2].
Put Beam on the shortlist if sovereignty matters. Apache 2.0 weights and official FP8/NVFP4 checkpoints make self-hosting realistic for organisations that already run GPU clusters [5][6].
Count tokens, not just accuracy. A model that is 2 points worse but 50% more concise may well be the better product.

The bottom line

This week's headline frontier race got the attention, but the more lasting story may be quieter. Labs are now competing to deliver the same intelligence faster, cheaper and in less memory. Diffusion text generation, very sparse open MoEs and brevity-trained small models are three separate bets on the same idea. If even one of them holds up under independent scrutiny, the AI stack of 2027 will look much less like "one giant model for everything" and much more like a carefully engineered portfolio.

We'll be watching for the first independent Mercury 3 evaluations and for Reflection's full technical report, and we'll report back.

— Cordoval Research

Search This Blog

Cordoval Research