The Test-Time Compute Paradigm Shift

Topics We'll Cover

• The o1/R1 Paradigm Shift
• Scaling Test-Time Compute vs Train-Time
• o1 / o3 — What's Public, What's Conjecture
• DeepSeek-R1 — Open Architecture & GRPO Training
• Long Thinking Traces & Max-Tokens Budget
• Cost & Latency Reality
• Eval Lift — Maths, Coding vs Chat Performance
• Reasoning Models in Agentic Loops
• What to Take Away

The o1/R1 Paradigm Shift

From 2017 to 2023, the dominant scaling law was simple: bigger pre-training runs → smarter models. GPT-4 is roughly 10× more compute than GPT-3.5. The curve flattened — not because the law broke but because frontier training runs hit hundreds of millions of GPU-hours and data scarcity.

OpenAI's September 2024 release of o1-preview introduced a second axis: how much compute you spend at inference time. The model is trained to think before answering — emitting a long private chain-of-thought (“thinking trace”) before producing its visible reply. The trace is hidden from the user but consumes tokens.

Scaling Test-Time Compute vs Train-Time

Snell et al. (Google DeepMind, 2024) — “Scaling LLM Test-Time Compute Optimally” — showed that for a fixed total FLOP budget, reallocating compute from training to inference can match or beat a larger model on hard problems. The crossover depends on problem difficulty: easy problems don't benefit, hard ones benefit enormously.

o1 / o3 — What's Public, What's Conjecture

OpenAI has published very little about o1's internals. The September 2024 system card and the December 2024 o1 technical report are the primary sources. Everything else is inference from behaviour and leaks.

import openai
client = openai.OpenAI()
resp = client.chat.completions.create(
    model="o3-mini",
    reasoning_effort="high",          # low | medium | high
    max_completion_tokens=16000,
    messages=[{"role": "user", "content": "Solve AIME 2024 II Problem 15."}]
)
usage = resp.usage
print(usage.prompt_tokens,
      usage.completion_tokens,
      usage.completion_tokens_details.reasoning_tokens)
# e.g.  42   14830   14620  (most tokens were thinking)

DeepSeek-R1 — Open Architecture & GRPO Training

DeepSeek-R1 (January 2025) is the first high-quality open-weight reasoning model. Its technical report is unusually detailed. The model family sits on top of DeepSeek-V3 (671B MoE, 37B active parameters per token), and the reasoning capability is entirely RL-induced — no supervised long-CoT imitation at the base stage.

GRPO — Group Relative Policy Optimisation

Standard PPO requires a value (critic) network, doubling memory. GRPO (DeepSeek-Math, 2024) eliminates the critic by estimating the baseline from a group of K sampled responses to the same prompt:

R1-Zero vs R1

Long Thinking Traces & Max-Tokens Budget

The thinking trace is the model's scratchpad. It is generated autoregressively before the final answer, typically enclosed in <think>…</think> tags (R1 convention) or as hidden reasoning_tokens (o-series). The model learns to use this space for hypothesis generation, arithmetic, checking sub-results, and recovering from errors.

import anthropic
client = anthropic.Anthropic()
resp = client.messages.create(
    model="claude-3-7-sonnet-20250219",
    max_tokens=16000,
    thinking={"type": "enabled", "budget_tokens": 10000},
    messages=[{"role": "user",
               "content": "Prove that sqrt(2) is irrational."}]
)
for block in resp.content:
    if block.type == "thinking":
        print("trace:", block.thinking[:200])
    else:
        print("answer:", block.text)

Cost & Latency Reality

Reasoning tokens are priced identically to output tokens. A single hard AIME problem with o3 at high effort can consume 20,000–60,000 reasoning tokens. At $15/M output tokens (o3 pricing as of 2025-Q1) that is $0.30–$0.90 per problem. For a 30-question AIME set: $9–$27. Batch API discounts (50%) bring this to ~$5–$14.

Eval Lift — Maths, Coding vs Chat Performance

The gains from reasoning models are highly domain-specific. AIME and competitive programming see dramatic lifts; conversational tasks see flat or negative results due to latency and over-thinking.

Reasoning Models in Agentic Loops

When a reasoning model is the planner in a multi-step agent, the interaction pattern changes significantly. The model's long thinking trace can incorporate tool results, plan sub-tasks, and revise the plan — all within a single generation. This collapses several turns of a classic ReAct loop into one.

What to Take Away

• A new scaling axis. Test-time compute joins train-time compute as a primary quality lever. The two are complementary and have different cost profiles.
• RL on verifiable rewards is the unlock. Correctness of maths answers and unit-test pass are automatic reward signals — no human raters needed at scale. GRPO makes this efficient by eliminating the critic network.
• DeepSeek-R1 is the open benchmark. Trained on DeepSeek-V3 MoE (671B/37B active) with pure GRPO + lightweight cold-start SFT. Weights publicly available in 1.5B–70B distilled sizes.
• Thinking tokens cost real money. Budget for 5–30k reasoning tokens per hard query; at o3 pricing that is $0.20–$1.20 per query. Use reasoning_effort: low or set a budget_tokens cap in production.
• Latency is the real constraint in interactive settings. 20k tokens at 60 tok/s ≈ 5.5 minutes to first visible token. Batch offline workloads; don't force o3 into a chat UX.
• Gains are domain-specific. AIME, Codeforces, GPQA: large lift. Chat, creative, simple QA: flat or negative. Pick the right tool.