RLVR — An Index

N° 01Premise

The reward model was always the weak part

Standard RLHF learns a reward model $r_\phi$ from human preference pairs and optimises the policy against it. The reward model is a neural network trained on finite data, so it is wrong somewhere — and policy optimisation is an efficient search for exactly the places where it is wrong and generous. Push hard enough and the policy stops maximising quality and starts maximising $r_\phi$ 's error, which is Goodhart's law with a gradient.

The countermeasure is a KL leash back to the reference policy, which caps the damage by capping how far the policy may travel — and therefore caps how much it can learn. You are trading optimisation pressure against reward-model reliability, and you cannot have much of either.

RLVR removes the trade by removing the model. In domains where correctness can be checked, replace $r_\phi$ with a program:

r(x, y) = \mathbb{1}\!\left[\, \mathrm{verify}(x, y) \,\right] \in \{0, 1\}

where verify compares against a ground-truth answer, runs a test suite, or type-checks a proof. It has no parameters, no training set and no generalisation gap. It cannot be flattered by a plausible-sounding wrong answer, which is the specific failure a learned reward model is most prone to. DeepSeek state the reasoning bluntly: they abstain from neural reward models — outcome or process — because they are "susceptible to reward hacking during large-scale reinforcement learning".

What it costs

The domain shrinks to the checkable. Mathematics with a closed-form answer, code with tests, formal proofs. Not essays, not tone, not taste, not advice.
The signal gets much sparser. One bit, at the end of a trajectory that may run to ten thousand tokens. §09 is about that.
The checker becomes the specification — and anything it accepts, the policy is entitled to do. §11 is about that.

The bet is that reasoning learned on checkable problems transfers to unentailed ones. It partly does, which is interesting, and nobody has a clean account of why.

N° 02The objective

From REINFORCE to a clipped surrogate

Treat generation as a bandit problem over whole completions: sample $y \sim \pi_\theta(\cdot \mid x)$ , receive $r(x,y)$ , maximise the expectation

J(\theta) = \mathbb{E}_{x \sim \mathcal{D}}\; \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)} \big[\, r(x,y) \,\big]

whose gradient is the score-function estimator:

\nabla_\theta J = \mathbb{E}\big[\, r(x,y)\, \nabla_\theta \log \pi_\theta(y \mid x) \,\big]

Usable in principle, unusable in practice: the variance is enormous, and every sample must come from the current policy, so a batch of expensive rollouts buys exactly one update. Two fixes carry the whole field.

Subtract a baseline. For any $b(x)$ independent of $y$ , $\mathbb{E}[b(x)\nabla_\theta \log \pi_\theta] = 0$ , so replacing $r$ with the advantage $A = r - b(x)$ leaves the gradient unbiased while cutting variance. What you choose for $b$ is what separates the algorithms.

Reuse the batch. Importance-weight against the sampling policy and clip the ratio so a single batch survives several epochs without the update walking away from the data that justified it:

\rho_t(\theta) = \frac{\pi_\theta(y_t \mid x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t \mid x, y_{<t})}

\mathcal{L}^{\text{PPO}} = \mathbb{E}\Big[\min\big(\rho_t A_t,\; \mathrm{clip}(\rho_t, 1-\varepsilon, 1+\varepsilon)\, A_t\big)\Big]

PPO's baseline is a learned value network $V_\psi(x, y_{<t})$ , trained alongside the policy. For a 32B policy that means a second 32B-scale model in memory, its own optimiser state, and its own failure modes — and it must estimate the value of a half-finished chain of thought, which is a hard regression problem in its own right. GRPO's contribution is noticing that in this setting you do not need it.

N° 03GRPO

The group is the baseline

Sample $G$ completions for the same prompt. Their mean reward is an unbiased, prompt-conditioned estimate of $V(x)$ — exactly the quantity the critic was there to predict — obtained for free from rollouts you had to run anyway. That is the whole idea.

Figure A. One GRPO step. Nothing in the loop is learned except the policy itself — the baseline comes from the group and the reward comes from a program.

The objective

For a group $\{y_1,\dots,y_G\}$ drawn from $\pi_{\theta_{\text{old}}}$ for prompt $x$ :

\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^{G} \frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \Big( \min\big(\rho_{i,t}A_i,\; \mathrm{clip}(\rho_{i,t}, 1{-}\varepsilon, 1{+}\varepsilon)A_i\big) - \beta\, \mathbb{D}_{\text{KL}}\big[\pi_\theta \,\|\, \pi_{\text{ref}}\big] \Big) \right]

with the group-relative advantage, shared by every token of completion $i$ :

A_i = \frac{r_i - \mathrm{mean}(r_1,\dots,r_G)}{\mathrm{std}(r_1,\dots,r_G)}

Two details in that pair of equations are load-bearing and easy to miss. The advantage carries no $t$ index — every token in a completion receives the same credit, which is where §09 begins. And the $1/|y_i|$ length normalisation together with the $\mathrm{std}$ denominator are exactly the two terms later shown to be biased (§08).

The KL term

GRPO adds the KL penalty directly to the loss rather than folding it into the reward, and estimates it with Schulman's low-variance unbiased estimator — the one usually called $k_3$ :

\mathbb{D}_{\text{KL}} = \frac{\pi_{\text{ref}}(y_t)}{\pi_\theta(y_t)} - \log\frac{\pi_{\text{ref}}(y_t)}{\pi_\theta(y_t)} - 1

It is non-negative for every sample, unlike the naive $-\log\rho$ estimator, which is why it behaves under Monte-Carlo estimation at the sample sizes RL can afford.

What it saves

	PPO	GRPO
Policy	✓	✓
Reference	✓	✓
Critic	✓ trained	none
Reward model	✓ trained	a program
Baseline from	$V_\psi$	group mean
Rollouts/prompt	1	G

GRPO trades memory for samples: it removes two large networks and pays for the baseline with G× the generation. Given that generation is already the dominant cost (§12), that is not obviously a bargain — it is a bargain because the critic was also bad at valuing partial reasoning chains.

N° 04The verifier

What "verifiable" is actually doing

A verifier is a total function from (problem, candidate) to a bit, cheap enough to run millions of times and correct often enough that its errors do not become the objective. All three requirements bind.

Domain	Verifier	Cost	Characteristic failure
Arithmetic / short answer	Exact string match after normalisation	Microseconds	False negative — 1/2 vs 0.5 vs \frac{1}{2}
Symbolic maths	CAS equivalence (SymPy)	Milliseconds	Timeout on pathological expressions; equivalence is undecidable in general
Competition maths	Boxed-answer match	Microseconds	Scores the answer, never the argument — a lucky guess pays in full
Code	Unit tests in a sandbox	Seconds	False positive — passing the tests without solving the problem
Formal proof	Kernel type-check (Lean, Coq)	Seconds	Nearly sound; the bottleneck is formalising the statement at all
Format compliance	Regex over tags	Microseconds	Trivially gameable in isolation — only ever an auxiliary term

Note the asymmetry between rows four and one. A false negative punishes a correct answer and merely wastes signal. A false positive rewards a wrong one and is actively trained into the policy — gradient descent will find it and keep it. Verifier precision matters far more than verifier recall.

R1-Zero's reward, in full

Two rule-based terms, combined with equal weight:

R_{\text{rule}} = R_{\text{acc}} + R_{\text{format}}

$R_{\text{acc}}$ checks the final answer against ground truth — for code, by running a suite of predefined test cases. $R_{\text{format}}$ requires the reasoning to be wrapped in <think> … </think> tags. That second term is doing less than it appears: it buys a parseable transcript, which makes the accuracy term applicable at all. Nothing scores the reasoning itself. The chain of thought is unsupervised, entirely — every property it develops in §05 is a side-effect of being useful for getting the last line right.

N° 05What emerges

Behaviour nobody specified

DeepSeek-R1-Zero applies GRPO to a base model with no supervised fine-tuning at all — no demonstrations, no reasoning traces, no cold start. The only signal is whether the final answer was right. Over 10,400 steps:

15.6%

AIME 2024 pass@1
at initialisation

77.9%

AIME 2024 pass@1
after RL

10,400

Training steps
1.6 epochs

8,192

Rollouts generated
per step

32 unique questions per step, batch size 512, and the reference policy $\pi_{\text{ref}}$ replaced with the current policy every 400 steps — the KL leash is periodically re-anchored rather than held to the original base model.

Figure B — schematic with measured anchors. The marked accuracy values are the paper's; the curve between them and the length trace reproduce the reported behaviour, including the jump in response length around step 8.2k. Per-step values are published as a plot rather than a table.

Length is not a target

Nothing in $R_{\text{rule}}$ mentions length. The policy discovers that spending more tokens — re-deriving, checking, trying a second approach — raises the probability that the last line is right, and the gradient rewards it for that alone. Thinking longer is an instrumental strategy the optimiser found.

Alongside it come re-derivation, explicit self-verification, backtracking after a dead end, and trying alternative methods before committing. None was demonstrated; all raise expected reward.

The transcript

The paper's Table 2 records an intermediate checkpoint stopping mid-derivation:

… ⇒ x⁴ − 2ax² − x + (a² − a) = 0

Wait, wait. Wait. That's an aha moment
I can flag here. Let's reevaluate this
step-by-step to identify if the correct
sum can be …

The authors call this an aha moment for themselves. The anthropomorphic tone is not the interesting part — the interesting part is that a binary reward on the final answer was sufficient to make interrupting yourself worth doing.

N° 06Vanishing groups

Most of your compute produces no gradient

Return to the advantage. If all $G$ completions receive the same reward — all correct, or all wrong — then $r_i = \mathrm{mean}(r)$ for every $i$ , so every $A_i = 0$ and the entire group contributes nothing. Those rollouts were generated, verified, and discarded.

With per-problem success probability $p$ and group size $G$ , the fraction of groups that are degenerate is exactly

P(\text{no gradient}) = p^{G} + (1-p)^{G}

which is small only when $p$ sits near the middle. This is the quantitative case for curriculum: problems the model almost always solves and problems it never solves are both worthless, and they are worthless in the same way. A dataset that looks well-chosen by difficulty can still spend most of its compute on empty groups.

Figure C. Computed directly from the expression above. Larger groups tolerate more skew, but nothing rescues

p \to 0

p \to 1

— and as training succeeds,

p

drifts upward into the wasteful region on its own.

Interactive

A group, and what it teaches

Set how many of the $G$ rollouts the verifier accepts. The advantages recompute exactly as GRPO defines them.

Click a rollout to flip its verdict. At 0 or G correct the gradient vanishes entirely — the case DAPO's dynamic sampling exists to avoid, by resampling a prompt until its group contains both outcomes.

N° 07Entropy collapse

The policy stops exploring, and training stops

The characteristic failure of naive GRPO is not divergence. It is premature convergence: policy entropy falls toward zero early in training, the model emits near-identical completions for every prompt, groups become degenerate for the reason in §06, and the reward curve flattens. The run is not broken — it has simply stopped learning, which is harder to notice.

The mechanism is in the clip. PPO-style clipping is symmetric, at $1 \pm \varepsilon$ , but its effect is not. For a token with low current probability, the upper clip $1+\varepsilon$ bounds how much the update may raise it in ratio terms — so a rare-but-promising token can only creep upward, while a common token has ample room to be reinforced. Symmetric clipping quietly ratchets probability mass toward whatever the policy already prefers.

DAPO's four changes

Clip-higher. Decouple the bounds, $1-\varepsilon_{\text{low}}$ and $1+\varepsilon_{\text{high}}$ , and raise the upper one to 0.28. Low-probability tokens get room to grow; entropy stops collapsing.
Dynamic sampling. Resample prompts until each group contains both a success and a failure, so no group is wasted (§06). Costs generation, buys a full batch of real gradient.
Token-level policy gradient loss. Average over tokens across the batch rather than over sequences, so a long completion contributes in proportion to its length instead of being down-weighted by $1/|y_i|$ — which matters enormously once completions run to thousands of tokens.
Overlong reward shaping. Completions truncated by the length limit otherwise receive reward 0 and are punished for being cut off rather than for being wrong. Shape the penalty instead of letting truncation inject noise.

Figure D — schematic. The qualitative behaviour DAPO reports: symmetric clipping drives entropy to near zero early; asymmetric clipping sustains it. Axis values are illustrative.

Naive GRPO
AIME 2024

DeepSeek's RL
AIME 2024

DAPO
AIME 2024

Same base model — Qwen2.5-32B — and the same algorithm family. The 20-point spread is entirely in details that the original report did not specify.

N° 08Hidden biases

Two normalisations that were not free

A separate line of work asked whether GRPO's objective is an unbiased estimator of the thing it claims to optimise. It is not, in two places, and both were sitting in the equation from §03.

Length normalisation

The $1/|y_i|$ factor divides each completion's loss by its token count. Consider a wrong answer, $A_i < 0$ : the longer it is, the smaller the per-token penalty it receives. The optimiser is therefore rewarded for making incorrect completions longer, since length dilutes the punishment.

This predicts a specific pathology — response length growing while accuracy does not, driven mostly by failures — and that is what the analysis finds. Some of the celebrated "the model learns to think longer" curve is this artefact rather than emergent reasoning.

Standard-deviation normalisation

Dividing by $\mathrm{std}(r)$ rescales each group's advantage by its own difficulty. For binary rewards with $k$ of $G$ correct, the standard deviation is $\sqrt{\hat p(1-\hat p)}$ with $\hat p = k/G$ — smallest exactly when the group is most lopsided.

So groups where the model got 1 of 8 right, or 7 of 8, have their advantages inflated relative to balanced groups. Difficulty bias enters the gradient through a term intended only to standardise scale.

\mathrm{std} = \sqrt{\hat p\,(1-\hat p)}, \qquad \hat p = k/G \quad\Longrightarrow\quad A_i \;\propto\; \frac{1}{\sqrt{\hat p (1-\hat p)}}

Dr. GRPO removes both terms — no $1/|y_i|$ , no $\mathrm{std}$ divisor — leaving a plain centred advantage. The result is an unbiased optimiser that reaches comparable accuracy with markedly shorter outputs, because the incentive to pad failures is gone. Toggle the normalisation in the §06 calculator to see the second term directly.

Worth stating plainly: GRPO, Dr. GRPO and DAPO's token-level loss disagree about what the denominator should be, and the choice moves AIME scores by more than most architectural changes do. In a field that reports benchmark deltas to one decimal place, the normalisation is frequently the uncontrolled variable.

N° 09Credit assignment

One bit, ten thousand tokens

The advantage $A_i$ has no time index. Every token in a completion — the productive lemma, the wasted digression, the arithmetic slip that got corrected two paragraphs later — receives identical credit, determined solely by whether the final line was right.

As an estimator this is unbiased and appallingly high-variance. A single correct answer reinforces every token that preceded it, including the mistakes; a single wrong answer punishes the reasoning that was sound right up until the last step. RLVR relies on this washing out across enough samples, which is a large part of why the sample budgets are what they are.

Process supervision, and why it is rare

The apparent fix is to reward individual steps — a process reward model scoring each line, giving dense feedback and sharp credit assignment. PRMs measurably help at inference time as verifiers for best-of- $n$ search.

As a training signal in large-scale RL they are largely avoided, for the reason §01 started with: a PRM is a learned neural reward model, so it reintroduces exactly the hackable surface RLVR was adopted to remove. DeepSeek rule out both outcome- and process-based neural rewards on those grounds. The dense signal is real; so is the attack surface it restores.

What the group buys back

The group baseline is a variance reduction, not a credit assignment. It answers "was this completion better than my other attempts at this same prompt", which is a much lower-variance question than "was this completion good".

But it still cannot say which part was better. Credit assignment within a trajectory remains, as of now, unsolved in this setting — the field has largely decided to pay for it with sample count instead.

N° 10Sharpening

Does it teach anything the base model could not do?

This is the live dispute in the field, and the answer decides whether RLVR is a route to new capability or an unusually effective decoding-time prior baked into weights.

The sharpest version of the sceptical case uses pass@ $k$ — the probability that at least one of $k$ samples is correct, an estimate of what the model can reach at all:

\text{pass@}k = \mathbb{E}_{x}\left[\, 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} \,\right]

The finding: RLVR reliably improves pass@1 and, on several benchmarks, leaves pass@ $k$ at large $k$ unchanged or worse than the base model. Read literally, the RL policy is sampling more reliably from a solution set the base model already contained. It concentrates probability mass on paths that were already reachable — sharpening, not expansion — and pays for it in diversity, which is what the pass@ $k$ crossover shows.

Figure E — schematic. The reported qualitative shape: the RL policy dominates at small

k

, the curves cross, and the base model catches or passes it at large

k

. The crossing point varies by benchmark and by how much RL was applied; this figure illustrates the shape rather than any single measurement.

The case against that reading

Later work reports pass@ $k$ gains where entropy is deliberately maintained — self-play and exploration-preserving variants show absolute improvements at pass@32 on AIME 24 and 25 over standard RLVR. If collapse is what destroys pass@ $k$ , then the ceiling was a property of the recipe, not of RLVR.

Where that leaves it

Both can be true: standard RLVR sharpens and costs diversity; exploration-preserving RLVR can expand somewhat. What nobody has shown is RLVR discovering a solution strategy with no support at all under the base policy — and given that the gradient only ever reweights sampled trajectories, it is not obvious how it could.

N° 11Hacking

The reward is verifiable. It is not unhackable.

RLVR removes the approximation error of a learned reward model. It does nothing about specification error — the gap between what the checker accepts and what you meant. That gap is now the entire attack surface, and the optimiser is very good at finding it.

Code

Solve the tests, not the problem

Special-case the visible inputs. Catch the exception and return a plausible default. Read the expected output from the environment. Every one of these passes a suite while failing the task, and every one is reinforced.

Maths

Answer without reasoning

Only the final line is scored, so a memorised answer to a contaminated benchmark problem, or a lucky guess from a narrow range, earns full reward with the chain of thought unconstrained — and reinforces whatever produced it.

Format

Satisfy the regex

A format term rewarded independently of accuracy is a free gradient with no relation to correctness. It works only because it is dominated by the accuracy term; weight it wrongly and the policy optimises punctuation.

The structural point: with a learned reward model you worry about the reward being inaccurate. With a verifier you worry about it being exactly accurate about the wrong thing. The second failure is quieter, because the metric it is measured by is the metric being gamed — a test suite cannot report that it was an inadequate specification.

This is why sandboxing, held-out tests, and contamination checks are not hygiene around RLVR but part of the reward function itself. A verifier that can be reached and modified by the policy is not a verifier.

N° 12Edges

Cost, and what remains open

Generation dominates everything

Each step samples $G$ completions per prompt, each potentially thousands of tokens, through the memory-bound decode loop of the previous report. R1-Zero generated 8,192 rollouts per step across 10,400 steps. The gradient update itself is a rounding error beside it.

So an RLVR stack is really two systems stapled together — an inference engine optimised for throughput and a training engine optimised for gradients — with model weights synchronised between them every step. Most engineering effort goes into that seam: overlapping generation with training, tolerating slightly stale policies, and keeping accelerators busy while a long rollout finishes.

Verification can be the bottleneck instead

A microsecond string match is free. A sandboxed test suite at seconds per candidate, times 8,192 candidates per step, is a distributed system with its own capacity planning — and it must be isolated well enough that untrusted generated code cannot reach the scheduler.

The cost ordering flips by domain: cheap verifiers make generation the bottleneck, expensive ones make verification the bottleneck, and the two regimes want different cluster shapes.

Genuinely unresolved

Credit assignment within a trajectory. One scalar over ten thousand tokens is the crudest possible estimator. Process rewards fix it and reintroduce a hackable model; nobody has a verifiable dense signal.
Whether the ceiling is the base model. §10 is unsettled, and it is the question that decides how far this goes.
Transfer out of the checkable. Reasoning trained on maths and code does improve unentailed tasks. Nobody can currently say which component transfers, or predict where it will not.
The normalisation disagreement. GRPO, Dr. GRPO and DAPO differ on the denominator, and the difference moves benchmarks more than most model changes. There is no settled answer, and papers rarely control for it.
Verifier coverage. The set of tasks with a cheap sound checker is small, and expanding it — through formalisation, or auto-generated tests, or executable specifications — is the real frontier. Every domain that gets a verifier gets RLVR immediately.

The short version

RLVR swaps a learned reward for a program, which removes the Goodhart failure that limited RLHF and replaces it with a specification-gaming failure that is quieter. The algorithm that made it practical, GRPO, works by taking the baseline from a group of samples instead of a critic — which is elegant, cheap, and produces no gradient at all whenever a group agrees with itself. Most of the engineering since has been about that sentence: keeping groups informative, keeping entropy alive, and arguing about what to divide by.