Hugging Face’s Community Article, attributed to Karina Zadorozhny and published on January 19, 2026, explains how reinforcement learning is used to post-train large language models. The article focuses on REINFORCE, Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), three methods that differ in how they use rewards, control updates and compare generated answers.
What the model is doing during reinforcement learning
In the framework described by the article, the state st is the current context: the original user prompt plus every token generated so far. The action at is the next token. The policy πθ is the LLM itself, represented as a probability distribution over the vocabulary and conditioned on that state.
The reward usually applies to the entire trajectory, which means the completed response rather than one isolated token. A separate reward model learns human preferences or other reward signals and most often returns a scalar score. With on-policy learning, the LLM generates its own training data, those responses are scored, and the model parameters are updated using the resulting signal.
REINFORCE reveals the stability challenge
REINFORCE takes the most direct approach. The update for each token is weighted by the total reward received by the full trajectory, keeping the link between the completed answer and the parameter update straightforward.
That simplicity comes with a major limitation identified by the article: high variance and instability. The method provides a clear illustration of why a reward attached to a complete response can make policy-gradient training difficult to control.
PPO controls updates, but needs more memory
The article presents PPO as a policy-gradient method introduced by OpenAI in 2017. It uses Generalized Advantage Estimation to reduce variance in gradient estimates while maintaining low bias. PPO also relies on a critic, or value function, to estimate how valuable a state is.
Policy changes are restricted through clipping or an adaptive KL-divergence penalty. For PPO-CLIP, the usual ε value is 0.2, keeping the probability ratio between 0.8 and 1.2. This adds control over how far the policy moves during an update, but it increases the memory requirement.
Training may need the policy, a reference model, a large critic and often a reward model loaded at the same time. The resulting trade-off is concrete: PPO offers additional mechanisms for managing variance and policy changes, while its supporting models make the setup heavier.
GRPO replaces the critic with group comparisons
GRPO avoids a separate critic by using the average score from a group of responses as its baseline. The article says that a group generally contains 64 samples. For each answer, GRPO subtracts the group’s mean score and divides by the group’s standard deviation to calculate the advantage.
The loss also includes a KL penalty between the current policy and a frozen reference model. This penalty limits drift from the reference model, but the article distinguishes it from the trust-region KL penalty described for PPO. GRPO therefore removes the critic while retaining a reference-model constraint and depending on comparisons among several generated answers.
The article identifies DeepSeek-R1 and DeepSeek-V3 as models using this approach without a critic. That example places GRPO in the wider discussion of methods designed to reduce the infrastructure associated with PPO.
The choice changes the training requirements
The distinction between these algorithms matters because “reinforcement learning” does not describe one uniform post-training recipe. REINFORCE is exposed to high variance, PPO adds a critic and tighter controls on policy updates, and GRPO uses a group baseline instead of a critic while preserving a KL constraint against a frozen reference.
For readers navigating LLM training terminology, the comparison clarifies what each name represents in practice. The baseline, reward calculation and protection against policy drift all influence how the system learns from scored responses and what infrastructure that learning requires.
Karina Zadorozhny’s Community Article does not treat the three methods as interchangeable labels. Its conclusion is narrower but useful: the post-training algorithm determines both how an LLM learns from evaluated answers and the operational cost of that learning.