LLM 9 - RLHF, PPO, and GRPO

Welcome back. LLM 8 asked how we can tell whether a language model is correct, useful, safe, and efficient. This blog takes the next step: once evaluation reveals a behavioral gap, how can training move the model toward the behavior people actually want?

The central difficulty is that next-token prediction does not directly encode helpfulness, honesty, or harmlessness. Pretraining teaches a model what text is likely. Supervised fine-tuning teaches it to imitate selected responses. Neither objective automatically expresses the full ordering humans have in mind: one answer may be more relevant, safer, clearer, or more faithful than another even when both are plausible continuations.

Reinforcement learning from human feedback (RLHF) turns those comparisons into a training signal. We will build the pipeline without skipping the uncomfortable parts: where preference labels come from, how rankings become pairwise examples, what a reward model learns, why PPO needs a value model and a reference model, how clipping works, how reward hacking appears, and why a KL penalty is necessary. We will then replace expensive human comparisons with constitutional feedback and finish with DeepSeekMath’s Group Relative Policy Optimization (GRPO), which removes the value model and compares several answers to the same question.

This page preserves the complete 130-slide lecture. Repeated animation builds are consolidated into their final diagrams and explained step by step; the original slide deck and a slide-by-slide archive remain available at the end.

1. Alignment After Fine-Tuning

From project lifecycle to behavioral goals

A generative-AI project usually passes through four broad phases: define the scope, select a model, adapt and align it, and integrate it into an application. Prompt engineering, supervised fine-tuning, evaluation, and alignment with human feedback all sit inside the adaptation phase. They are related, but they solve different problems.

Prompt engineering changes the context without changing model weights. Supervised fine-tuning changes the weights so that desirable responses become more likely. Evaluation measures whether the resulting system meets a specification. Human-feedback alignment adds a preference objective when “good” cannot be captured by one reference string.

The fine-tuning goals in the lecture are deliberately broader than memorizing prompt–response pairs:

• understand prompts more reliably;
• complete tasks more accurately;
• produce more natural language;
• behave in ways that are helpful, honest, and harmless.

Prompt engineering, fine-tuning, evaluation, and human-feedback alignment are complementary parts of model adaptation. Course slide 7.

Why likelihood is not enough

A model can produce toxic language, respond aggressively, or provide dangerous instructions because the pretraining objective rewards statistical continuation, not intent. The HHH framework asks three separate questions:

• Helpful: does the response address the user’s legitimate need?
• Honest: does it avoid confident falsehoods and communicate uncertainty?
• Harmless: does it avoid creating unjustified risk?

These goals can conflict. A maximally literal answer to a harmful request may be “helpful” in a narrow sense and unacceptable overall. A refusal may be harmless yet unhelpful if it does not explain what safe assistance remains available. Alignment is therefore an ordering problem, not a binary filter.

Three failures illustrate why alignment must evaluate more than grammatical fluency. Course slide 9.

Evidence from human-feedback summarization already showed the value of this distinction. In the cited study, human preference for model summaries improved with scale when the model was trained from human feedback. The key comparison was not token-level similarity to one reference. People directly judged which summary was better.

Human-feedback training can improve preference beyond supervised imitation or pretraining alone. Course slide 11; adapted from Stiennon et al. (2020).

The RLHF pipeline begins with an instruction-tuned model and ends with a human-aligned model. Between them is reinforcement learning driven by feedback. The target is not “agree with every user.” It is to maximize useful and relevant behavior while reducing harm, misinformation, and dangerous assistance under a written alignment policy.

2. Reinforcement Learning for Language

Agent, environment, action, and reward

Reinforcement learning describes an agent interacting with an environment. At time , the agent observes a state , chooses an action , receives reward , and transitions to a new state . A policy assigns probabilities to actions. The objective is to maximize expected accumulated reward.

For a discounted horizon,

where controls how strongly later rewards matter. In an episodic task, the reward may arrive mainly at the end.

Tic-tac-toe makes the vocabulary concrete. The board is the state, a legal square is an action, a complete game is a rollout, and win/loss/draw outcomes produce rewards. The policy learns which move is preferable in each board position. The environment enforces legal transitions and returns the next board.

The final animation state of the tic-tac-toe example: state enters the policy, an action changes the environment, and the resulting reward closes the loop. Course slides 14–20.

Mapping the loop to an LLM

For language generation, the mapping is almost literal:

• agent/policy: the instruction-tuned LLM;
• state: the prompt plus all tokens generated so far;
• action: the next token selected from the vocabulary;
• environment: the evolving text context and termination rule;
• trajectory/rollout: a complete generated response;
• reward: a score assigned to the response, often by a learned reward model.

At step , the state is

where is the prompt and is the partial completion. The action is token , sampled from

After an end token or length limit, the trajectory receives a score. Unlike a board game, the environment does not usually reveal a natural reward after each word. That delayed and imperfect supervision is one reason language-model RL is difficult.

An LLM is the policy, its context is the state, vocabulary items are actions, and a reward model scores the completed rollout. Course slides 21–30.

The objective sounds simple—generate aligned text—but the reward is not available yet. We first need a dataset that teaches a model what people prefer.

3. Human Feedback and Reward Models

Building a preference dataset

Start with a prompt dataset representative of the deployment task. For every prompt , sample several completions from the instruction-tuned model. Diversity matters: if every completion is nearly identical, labelers cannot teach the reward model much about the boundary between weak and strong behavior.

Human labelers then rank the candidate responses under a specific criterion such as helpfulness. The criterion must be written down before annotation begins. “Choose the best response” is not enough when labelers disagree about whether safety, concision, factuality, tone, or instruction following has priority.

The lecture’s example begins with “My house is too hot.” A useful response suggesting air conditioning is ranked above a dismissive “There is nothing you can do,” while “It is not too hot” ignores the user’s report. The point is not the household advice; it is that ranking expresses a relative judgment among answers to the same prompt.

Several labelers rank the same model completions under an explicit helpfulness criterion. Course slide 33.

The annotation instructions in the lecture require labelers to:

1. rank responses from best to worst for the supplied prompt;
2. base the decision on correctness, relevance, informativeness, and usefulness;
3. treat nonsensical, irrelevant, or factually incorrect answers as low quality;
4. prefer concise, direct answers while recognizing that longer answers may be better when extra detail is necessary.

These instructions are part of the model. If the rubric changes, the meaning of the reward changes.

A preference dataset is only interpretable together with the rubric used to create it. Course slide 34.

From rankings to pairs

Suppose the responses are ranked . This ranking can be expanded into pairwise examples:

For each pair, one response is labeled chosen () and the other rejected (). Pairwise conversion makes the learning problem consistent even when prompts have different numbers of candidates.

A ranking becomes multiple chosen-versus-rejected comparisons for reward-model training. Course slide 35.

The first lecture checkpoint asks for the role of human labelers in RLHF. They do not identify weights or write every completion from scratch. Their core role is to score or rank prompt completions so that these comparisons can train the reward model. The correct choice is D.

Training the reward model

A reward model receives a prompt and completion and returns a scalar. A common Bradley–Terry preference model defines

where . The pairwise loss is

Only the score difference matters. Adding the same constant to every reward does not change the preference probability. This is why raw reward values should not be interpreted as universal moral units; they are learned coordinates for comparisons on the training distribution.

The reward model learns to assign a larger scalar to the preferred completion. Course slides 37–39.

After training, the reward model can score new prompt–completion pairs. The lecture visualizes it as a binary preference classifier: logits are converted to probabilities, and the preferred-class score becomes a reward signal. In practice, calibration, distribution shift, annotation noise, prompt leakage, and adversarial responses all matter. The reward model is a learned proxy for the rubric, not a perfect oracle.

The trained reward model supplies a scalar signal for model-generated text. Course slides 40–43.

4. PPO for RLHF

The rollout–update cycle

RLHF now has three main learned components: an instruction-tuned policy, a reward model, and a value model used by PPO. A fourth model—a frozen reference policy—will later constrain drift.

For each iteration:

1. sample prompts from the prompt dataset;
2. generate completions with the current policy;
3. score the completions with the reward model;
4. estimate returns and advantages;
5. update the policy while limiting how far it moves;
6. replace the old policy with the updated policy and repeat.

The lecture’s running example begins with “A dog is …”. Early completions receive rewards such as , , , and . After many iterations the policy discovers higher-scoring language. These numbers illustrate the feedback loop; they are not comparable across unrelated reward models.

The reward model evaluates rollouts and PPO updates the policy. Course slides 44–50.

PPO alternates between two phases. Phase 1 creates completions and collects experience. Phase 2 updates the model using those fixed trajectories. This separation makes the probability ratio meaningful: the numerator comes from the updated policy and the denominator from the policy that generated the data.

Value prediction and value loss

Let predict future return from state . A simple Monte Carlo target is

The value loss is

If the estimated future reward is and the observed terminal reward is , the prediction error is and the squared-error contribution is . The value model supplies a baseline: actions should be reinforced not merely when their outcome is positive, but when it is better than expected from the current state.

The value function learns to predict the return; its residual helps determine whether a sampled action was better or worse than expected. Course slides 55–57.

PPO implementations often use generalized advantage estimation (GAE),

trades variance against bias. The value network reduces variance, but for an LLM it is another model-sized component that consumes memory.

Probability ratios and clipping

Let be the policy that generated a token and the policy being updated. Their probability ratio is

means the updated policy makes that token more likely; means it makes the token less likely. Multiplying by advantage gives the desired direction:

• : increase the sampled token’s probability;
• : decrease it;
• a large : apply a stronger update;
• far from : the new policy has already moved substantially.

The clipped PPO surrogate is

Training maximizes , or equivalently minimizes . The minimum chooses the more conservative term. For a positive advantage, ratios above stop receiving additional benefit. For a negative advantage, ratios below stop receiving additional benefit. The interval acts as a trust region; the lecture uses a typical value near .

Clipping prevents one batch of noisy advantage estimates from causing an arbitrarily large policy step. Course slides 59–68.

Entropy and the complete objective

If the policy becomes too certain too quickly, it may collapse onto a narrow set of responses. Entropy measures distributional spread:

A low-entropy policy concentrates probability on a few tokens; a high-entropy policy retains more alternatives. When training minimizes a loss, an entropy term can be written

so minimizing it encourages exploration. Too much entropy, however, produces random or unfocused language. The coefficient must be tuned against task reward and stability.

Entropy distinguishes a narrow token distribution from a more diverse one; neither extreme is automatically desirable. Course slides 69–70.

One loss convention in the lecture is

The policy term improves reward, the value term trains the return baseline, and the entropy term controls diversity. After the update, the new model generates another set of trajectories and the two-phase cycle repeats.

The second lecture checkpoint asks what RLHF can improve. It can reduce toxicity and misinformation; it does not inherently make inference faster, enlarge the network with “preference parameters,” or guarantee interpretability. The supported choice is A.

5. Reward Hacking and KL Control

When the proxy becomes the target

Reward hacking occurs when the policy discovers a way to increase the learned reward without improving the intended behavior. The reward model is finite, trained on a limited distribution, and easier to optimize than the underlying human preference. Once the policy searches aggressively against it, small reward-model errors can become dominant.

The lecture’s toxicity example begins with “This product is …”. A clearly toxic completion receives . PPO then finds “okay but not the best” at , an exaggerated “the most awesome, most incredible thing ever” at , and finally a vacuous “Beautiful love and world peace all around” at . The reward rises while relevance and truthfulness deteriorate. The model has learned the reward model’s shortcut.

A proxy trained to dislike toxicity can be exploited by irrelevant, excessively positive text. Course slides 79–82.

This is an instance of Goodhart’s law: once a measure becomes a target, pressure on the measure can break its relationship with the real goal.

A frozen reference policy

To prevent uncontrolled drift, RLHF keeps a frozen reference model , usually initialized from the same instruction-tuned checkpoint. The updated policy is penalized for moving too far from it. A sequence-level objective can be written

At token level, many implementations shape the reward with

The coefficient controls the trade-off. If it is too small, the model can exploit the reward model and forget useful language behavior. If it is too large, the policy barely moves and alignment gains disappear.

The reference policy supplies a shift penalty; the lecture also shows parameter-efficient adapters so that the trainable update need not duplicate every weight. Course slides 83–91.

The reference model is frozen. The policy may be updated through all parameters or a parameter-efficient adapter. PEFT reduces trainable memory, but PPO still needs activations, optimizer state, the reward model, and often a value model. Memory efficiency is one motivation for GRPO later in the blog.

Evaluate after alignment

Alignment training must be followed by evaluation. The lecture compares toxicity scores on a summarization dataset before and after RLHF, with the illustrated score falling from to . The number is meaningful only under the evaluator and data slice used, but the experimental pattern is correct: preserve a held-out evaluation set, measure the target behavior, and check for regressions elsewhere.

The aligned model is evaluated on held-out summaries with the same toxicity evaluator. Course slides 92–94.

The TRL workflow summarizes the production loop as rollout, evaluation, optimization. A policy generates responses, a reward model evaluates them, and PPO optimizes log probabilities under reward and KL constraints. Parameter-efficient training can make a 20B-scale RLHF experiment possible on a 24 GB consumer GPU in the cited Hugging Face example, but hardware feasibility does not remove the statistical and safety problems above.

6. Scaling Feedback and Constitutional AI

The cost of human labels

Classical RLHF can require tens of thousands of preference labels. Human annotation is expensive, slow, and inconsistent across cultures and rubrics. It is also hard to expose annotators to every adversarial request the deployed model may encounter.

Constitutional AI replaces part of that repeated supervision with a written set of principles and model self-critique. The constitution does not eliminate human values; people still choose the principles. It changes how those values are scaled across prompts.

Constitutional AI trades repeated human comparisons for scalable feedback guided by explicit rules. Course slides 96–97.

Example principles in the lecture ask the model to choose a response that is most helpful, honest, and harmless; least harmful to humanity; attentive to legality, ethics, racism, sexism, toxicity, and danger; and similar in spirit to peaceful and ethical exemplars. A constitution should be auditable: each principle needs a rationale, priority, and process for resolving conflicts.

Written principles make the source of a critique explicit, although the principles themselves still require governance. Course slide 101.

Supervised self-critique and revision

The supervised stage is:

1. use red-team prompts to elicit problematic responses;
2. ask the model to critique its response under a selected constitutional principle;
3. ask it to revise the response;
4. fine-tune on the revised, safer responses.

In the lecture’s Wi-Fi example, an initial helpful model gives instructions for hacking a neighbor’s network. A critique identifies illegality, privacy violation, and harm. The revision refuses the request while explaining the problem and offering safe alternatives. The training target is the revised response, not the unsafe draft.

The supervised stage transforms a red-team response through principle-based critique and revision. Course slides 102–107.

RLAIF

The reinforcement-learning stage uses AI feedback (RLAIF):

1. generate multiple responses to red-team prompts;
2. ask a model which response better follows the constitution;
3. train a reward model on those preferences;
4. optimize the policy with RL;
5. obtain a constitutional model.

The complete pipeline combines supervised critique/revision with a preference model and reinforcement learning from AI feedback. Course slides 108–111.

RLAIF scales cheaply, but it can amplify blind spots shared by the feedback model and policy. Human audits, red teaming, disagreement analysis, and high-risk escalation remain necessary. The constitution is a specification, not a proof of safety.

7. DeepSeekMath

Data before algorithm

DeepSeekMath combines a large mathematical corpus, supervised examples, and reinforcement learning. The lecture highlights three stages:

1. Pretraining: 120B mathematical tokens, starting from DeepSeek-Coder-Base-v1.5 7B.
2. SFT: 776K examples spanning chain-of-thought, program-of-thought, and tool-integrated reasoning.
3. GRPO: roughly 144K questions, a subset of the SFT data, used for reinforcement learning.

The data pipeline starts with mathematical seed pages, trains a fastText classifier, recalls math-related Common Crawl pages, discovers related domains, annotates additional URLs with a language model, and iterates. Two findings emphasized in the lecture are that prior code training can improve mathematical reasoning and that arXiv papers were surprisingly ineffective as the primary reasoning corpus.

DeepSeekMath combines scalable web-data collection with pretraining, SFT, and GRPO. Course slide 114.

Reported results

The 7B RL model reported:

• GSM8K: ;
• MATH with chain-of-thought: ;
• MATH with tool-integrated reasoning: ;
• improvement over SFT: points on GSM8K and on MATH;
• Chinese CMATH improvement from to ;
• self-consistency with 64 samples: on MATH.

These are historical paper results under the paper’s models, datasets, prompting, and evaluation protocol. They should not be read as timeless rankings. Their importance here is causal: reinforcement learning improves the probability of producing correct solutions from the model’s existing distribution.

The reported 7B results motivate a closer look at how GRPO obtains RL gains without PPO's value model. Course slide 115.

8. GRPO Derivation

Why remove the value model?

PPO for language models commonly keeps four models in memory:

• policy model;
• frozen reference model;
• reward model;
• value or critic model.

The value model can be approximately as large as the policy, producing substantial memory and optimizer overhead. GAE also requires token-level temporal-difference estimates even though language rewards often arrive after the entire answer. KL shaping at every token further complicates the advantage calculation.

PPO needs policy, reference, reward, and value models; GRPO is designed to remove the critic. Course slide 116.

GRPO replaces the learned value baseline with a group-relative baseline. For one question , sample responses

and compute rewards . Normalize them within the group:

The baseline now comes from other answers to the same question. No critic is needed. This can save roughly 25–50% of model memory relative to a four-model PPO setup, depending on optimizer state, sharding, and implementation.

GRPO samples a response group, scores each response, and computes advantages relative to the group rather than a value network. Course slide 118.

Consider the lecture’s question “What is ?” and five answers . With binary rewards ,

Thus a correct answer has

while an incorrect answer has

The same raw reward has different meaning in a different group. A reward of is strong if the other answers average and weak if they average .

Correct responses receive positive relative advantage and incorrect responses negative advantage. Course slide 119.

The GRPO objective

For response , define the token ratio

A compact GRPO objective is

The outer average treats the sampled outputs symmetrically. The inner average prevents long responses from dominating solely because they contain more tokens. PPO-style clipping keeps the policy step bounded. KL regularization is included directly in the objective rather than modifying the task reward used to compute advantages.

GRPO combines group-relative advantages, token-level probability ratios, clipping, and direct KL regularization. Course slide 120.

Probability times advantage

The core policy-gradient term is the product of probability change and advantage. Four cases provide a useful sign check:

1. positive advantage, high probability: the output is good and already likely; reinforce it, but clipping may limit the update;
2. positive advantage, low probability: the output is good but rare; increase its probability strongly;
3. negative advantage, high probability: a poor output is too common; reduce its probability;
4. negative advantage, low probability: the poor output is already rare; clipping prevents an unnecessary extreme step.

Before trusting an implementation, verify that each probability–advantage case moves the policy in the expected direction. Course slide 121.

Clipping does not say that ratios outside the interval are impossible. It says the surrogate objective stops rewarding further movement beyond the trusted range. For , a ratio greater than is clipped downward; for , a ratio below is clipped upward inside the conservative term.

The sign of the advantage determines which side of the ratio interval is dangerous. Course slide 122.

Outcome and process supervision

Outcome supervision assigns one reward to a complete response. Every token shares the same group-relative advantage:

Process supervision assigns rewards to intermediate reasoning steps. A token’s advantage can accumulate normalized rewards from future steps:

The resulting signal can identify where reasoning goes wrong instead of treating every token in a failed solution as equally bad. In the reported experiment, GRPO with process supervision outperformed GRPO with outcome supervision.

Process supervision supplies step-level credit assignment and gave the stronger reported result. Course slide 123.

Iterative GRPO

One exploration stage samples a batch, generates outputs per question, computes rewards and group-relative advantages, and performs one policy update (). The lecture uses clipping , KL coefficient , and a reward-model replay buffer containing 10% historical data. At each outer iteration, the reference model is reset to the current policy and the reward model is updated as the policy’s output distribution changes.

Iterative GRPO co-evolves policy and reward models instead of treating the original preference model as permanently valid. Course slide 124.

9. Results and Interpretation

SFT, RFT, and GRPO as weighted likelihood gradients

The lecture unifies several methods through

The difference is the gradient coefficient and how training data are obtained:

• SFT: offline human-selected data; coefficient for every target token;
• rejection-sampling fine-tuning (RFT): offline sampled data filtered by a rule; coefficient for accepted answers and for rejected answers;
• GRPO: online samples; a model- or rule-based reward; continuous positive and negative coefficients from relative advantage.

SFT imitates. RFT keeps or discards. GRPO ranks degrees of quality and can explicitly push probability away from weak outputs.

SFT, RFT, and GRPO can be compared as different weights on log-probability gradients. Course slide 125.

Experimental hierarchy

On the reported GSM8K and MATH experiments, the hierarchy is

Online methods improve more strongly over time than offline rejection sampling. Continuous advantage coefficients outperform binary keep/discard coefficients, and process supervision supplies better credit assignment than one outcome reward. All methods improve on the SFT baseline, but the RL variants improve more.

GRPO with process supervision gives the strongest curves in the reported comparison. Course slide 126.

Two iterative rounds provide additional gains with diminishing returns. The lecture reports approximately

The reward model must evolve with the policy. Otherwise it is asked to score a distribution increasingly different from the one on which it was trained.

The second iteration remains beneficial, but the marginal gain is smaller. Course slide 127.

Maj@K versus Pass@K

The reported curves make an important distinction. Pass@K asks whether at least one of samples is correct; it estimates the boundary of what the model can produce under sampling. Maj@K asks whether the majority or self-consistent aggregate is correct; it measures how much probability mass lies on correct solutions.

RL increased Maj@64 on MATH from roughly to , while Pass@K changed little. This suggests that the SFT model already had many correct solutions somewhere in its support. RL redistributed probability toward them, making correct reasoning more reliable, rather than creating an entirely new ceiling of solvable problems.

RL improves the robustness of sampled outputs more clearly than the maximum capability revealed by Pass@K. Course slide 128.

Efficiency and future directions

The lecture’s DeepSeekMath configuration uses policy learning rate , reward-model learning rate , KL coefficient , group size , roughly 144K questions, and one policy update per exploration stage. Removing the value model reduces the architecture from four model roles to three: policy, reward, and reference.

GRPO's efficiency comes from removing the critic, not from eliminating the reward or reference model. Course slide 129.

The final directions are:

1. better data: out-of-distribution prompts and tree-search sampling;
2. more robust algorithms: noisy-reward handling and weak-to-strong learning;
3. better reward models: generalization, uncertainty, and process supervision.

DeepSeek-R1 later demonstrated emergent reasoning with GRPO-style training without an initial SFT stage, and variants such as DAPO and Dr. GRPO continue to refine optimization for reasoning models. The durable lesson is not that one optimizer solves alignment. It is that preference data, reward quality, regularization, credit assignment, sampling, and evaluation form one coupled system.

The complete pipeline: collect comparisons, train a reward proxy, optimize the policy conservatively, audit proxy failures, keep a reference constraint, update the evaluator as the policy changes, and verify behavior on held-out tasks. RLHF is not one loss function; it is a feedback system whose weakest component can dominate the result.

10. Complete Lecture Archive

Every slide is retained below. The main text consolidates repeated animation builds—for example slides 14–20, 21–30, 37–43, 44–50, 59–68, 75–78, 83–91, and 102–111—into their final state and explains the intermediate additions in order.

Slides	Covered above
1–12	Agenda, project lifecycle, model failures, HHH, human-feedback motivation, RLHF overview
13–30	RL loop, tic-tac-toe, LLM state/action/reward mapping
31–43	prompt sampling, annotation rubric, pairwise data, reward model training and inference
44–78	RL updates, PPO rollout, value loss, clipped policy loss, entropy, full objective, checkpoints
79–95	reward hacking, frozen reference model, KL penalty, PEFT, evaluation, TRL workflow
96–111	scaling feedback, constitutional principles, critique/revision, RLAIF
112–130	DeepSeekMath data and results, PPO limitations, GRPO derivation, supervision, experiments, future work

Download the original Lecture 9 PDF

Open the complete 130-slide archive

References

• Stiennon et al., Learning to Summarize with Human Feedback .
• Ouyang et al., Training Language Models to Follow Instructions with Human Feedback .
• Bai et al., Constitutional AI: Harmlessness from AI Feedback .
• Schulman et al., Proximal Policy Optimization Algorithms .
• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models .
• Hugging Face, Fine-tuning 20B LLMs with RLHF on a 24GB Consumer GPU .