A code model that learns to simulate execution, not just text. CWM 32B matches gpt-oss-20b at parity, exceeds gpt-oss-120b with test-time scaling — at 4× smaller parameter count. The world-modelling mid-training stage is the secret.
Linear hybrids and diffusion are generic-purpose architectural plays. CWM is different: it is a training innovation tied to a specific domain — code. The interesting question is whether the world-model trick generalises to other tasks, or whether code is uniquely suited because it has a free, executable, ground-truth simulator.
Standard code LLMs are next-token predictors over tokens of source code. They learn beautifully fluent code that follows the patterns in training data. They do not, in general, know what their code does when it runs. The model has read the function signature, the call sites, the docstring — but it has not run the program in its head.
x is after line 5.self.cache[key] is populated when line 12 executes.Real engineering tasks — debugging, multi-step refactors, fixing test failures — require knowing what the program does. The standard-LLM workaround is to give it a tool: let it run the code in a sandbox and read the output. CWM’s bet is that you can also internalise some of that capability so the model needs the sandbox less.
“A code world model learns to simulate what happens when the code runs by predicting program state evolution rather than just syntax patterns.” The clean-up of that statement: standard LLMs predict the next character; CWM predicts what the next character implies for the program’s state, and learns from the gap between its prediction and what really happened.
The training data has a new shape. For every code snippet, the training corpus also contains execution traces — structured records of variable values, function entry/exit, returned objects, side effects, exception paths.
# source
def tally(items):
counts = {}
for x in items:
counts[x] = counts.get(x, 0) + 1
return counts
# execution trace, interleaved at training time
<TRACE>
call: tally(items=['a', 'b', 'a'])
step 1: counts = {}
step 2: x = 'a'; counts = {'a': 1}
step 3: x = 'b'; counts = {'a': 1, 'b': 1}
step 4: x = 'a'; counts = {'a': 2, 'b': 1}
return: {'a': 2, 'b': 1}
</TRACE>The model is trained to predict both the source code and the trace tokens. At inference time, the model can choose to emit a trace before answering — or you can prompt it to. The execution intuition lives in the model, not just in a tool the model calls.
“Tokens can encode structured execution traces rather than plain text”. The structured-trace format is what gives the approach traction: ordinary CoT is a model’s guess about its own reasoning; an execution trace is the program’s actual behaviour. The training signal is far stronger.
The architecture is deliberately conservative: a dense decoder transformer, no MoE, no exotic attention, no recurrence. The novelty is in the training data and procedure — not the model. This is methodologically important: it isolates the contribution of the world-modelling stage.
| Property | CWM 32B | Notes |
|---|---|---|
| Total parameters | 32B (dense) | No MoE; every parameter active per token. |
| Architecture | Decoder-only transformer | Standard pre-norm; no architectural exotica. |
| Context length | 131 072 tokens | Long enough to fit a meaningful slice of a real repo plus traces. |
| Attention | Sliding-window | Local attention bands rather than full O(n²) at full context. |
| Training data | Code + execution traces | The differentiating ingredient; size is in the same league as comparable code-LLM training mixes. |
| Tokenizer | Standard BPE-style | Trace markers are dedicated special tokens. |
Real repos plus their traces are long. Full O(n²) attention at 131 k tokens would dominate inference cost. Sliding-window attention bounds each token’s attention span (e.g. the last few thousand tokens), which keeps inference tractable. The trade-off: information from very distant parts of the repo can’t be retrieved by attention — the model has to use the structured trace to carry that state forward, which is exactly what the training is teaching it to do.
The architectural conservatism is the methodological win. It means the SWE-bench gains can’t be attributed to clever attention tricks or routing networks — they have to be coming from the training data and procedure. That gives the result much more generalisability as evidence for the world-model thesis than a co-designed architecture would.
The training pipeline has four stages. The third — mid-training on world-modelling data — is the new one. Without it, you have a competent but unremarkable 32B code model. With it, you get the SWE-bench numbers on the next slide.
The cost of generating this data is non-trivial: you need a working sandbox, instrumentation, and curated input distributions. That investment is a large part of why the technique is hard to reproduce without significant engineering.
Mid-training, not pre-training, is where world-modelling lives. That choice means the model is not paying the cost of trace data on every pre-training token — it learns code first the standard way, then specialises into world-modelling once the basics are in place. Empirically this seems to work better than mixing trace data into the pre-training corpus from the start.
SWE-bench is the benchmark closest to what real coding agents do: take a real GitHub issue, produce a patch, see if the tests pass. The CWM numbers in the article are striking because they isolate capability per parameter.
| Model | Params | SWE-bench tier | Notes |
|---|---|---|---|
| gpt-oss-20b (mid reasoning) | 20B | Baseline | Standard code-LLM training; smaller but solid. |
| CWM 32B | 32B | Matches gpt-oss-20b | Same league at modestly larger size; the world-modelling is “earning back” the parameter overhead. |
| gpt-oss-120b (high reasoning) | 120B | Standard frontier-tier | 4× larger than CWM. The hard target. |
| CWM 32B + test-time scaling | 32B | Exceeds gpt-oss-120b | The headline result — with extended sampling, beats a 4×-larger model. |
The CWM at inference can sample multiple candidate patches, check each against the trace it predicts, and pick the one that survives. The execution-trace prediction is itself a verifier: a candidate that produces a trace inconsistent with the test expectations can be filtered before submission. The standard LLM doesn’t have this; it has to ask a tool to run the code, and pays latency for every check. The CWM does much of the verification in-model.
Two things buy capability: (a) the world-model gives sharper per-sample candidates than next-token-only, and (b) the model can reject its own bad candidates by examining the predicted trace. Each effect alone might be small. Together they explain the 4×-parameter advantage.
SWE-bench is one benchmark, on one task family. The result likely generalises to closely-related tasks (debugging, refactoring) and might not to distant ones (code review, design). Treat the headline number as a strong signal, not a proof of universal advantage.
If world-modelling can save 4× parameters on a code task, the same idea may save parameters on any task with a verifiable simulator. Mathematics has formal proof checkers. Database queries have query engines. Theorem proving has Lean. The CWM result is best read as evidence for a general approach, not just a one-off code optimisation.
Step through a small program. The left panel shows what a standard code LLM emits — just the next line of source. The right panel shows what a CWM emits — the next line plus its predicted state delta, and a verdict on whether the prediction matches actual execution. Watch how often the standard view goes through with no signal that something has gone wrong.
Press Inject bug. The standard model emits the same fluent code as before — no signal anything is wrong. The CWM’s state prediction diverges from actual execution, the verdict flips to red, and the model has internal evidence the candidate is bad before any test runs. That is the verification signal that lets test-time scaling do real work.
Predicting trace tokens means producing more tokens per task. Even with sliding-window attention, the model is generating substantially more text than a same-size autoregressive code model. Latency-sensitive completions (auto-complete in an editor) are a poor fit.
You need a sandbox, instrumentation, an input-sampling distribution, and pipeline plumbing to interleave traces with source. This is significantly more engineering than “feed the model The Stack”. The barrier to reproducing the technique is not weights, it is data.
The published evaluations are on Python-heavy SWE-bench-style tasks. The technique should generalise — the trace format is language-agnostic in principle — but published evidence beyond Python is thin. Treat the result as Python-strong, other-languages-plausible.
If the world-model trick works only when there is a clean executable simulator, then the technique applies to code, mathematics, formal verification, SQL, and a small handful of other domains. If it works for any domain where you can construct a structured trace — including domains where the trace is itself learned — the technique becomes far more general. Raschka explicitly flags this as an open question. The 2026 papers on this will tell us a lot.
For now: CWM is the right tool when you have a code-completion-or-debugging task, you want frontier-tier performance, and you don’t want frontier-tier inference cost. It is not yet the right tool for code review, code explanation, design conversations, or anything where the user wants to read what the model is “thinking” in plain prose.
Press Inject bug at different points and watch the CWM verdict flip. Note where the standard panel gives no signal — that is the cost the standard model pays for not knowing what its code does.
From your own commit history, take five “fix bug” commits. Walk through the buggy code and write down the variable state at each line manually. The exercise builds the same intuition the CWM is being trained to internalise.
Write a Python sys.settrace wrapper that produces structured traces in the CWM format from any input function. The amount of plumbing required is the data-engineering barrier the technique faces — doing it yourself makes that barrier viscerally clear.
If CWM weights are open, run it on the easiest tier of SWE-bench. Compare to a Llama-3 70B or comparable open code model. Look not just at pass-rate but at the shape of the failures — the CWM should fail on tasks where the trace itself was misleading; the standard LLM should fail more uniformly.
The CWM technical report is the primary source — pay attention to the data-construction section, which is the part that’s hardest to reproduce. The SWE-bench paper for context on what the benchmark actually tests.
A non-code domain getting the same treatment. Mathematics with Lean traces. SQL with query-engine traces. Whichever domain wins next will tell us whether CWM was a code-specific result or the start of a method.
Next time someone says “our code agent needs a bigger model”, ask: does it have a way to know whether its candidate is correct before we run the tests? If the answer is no, the problem may be a missing world-model, not a missing 100B parameters — and CWM is the proof-point that those are not the same lever.