DeepSeek-R1

DeepSeek-R1 made the o1-style Reasoning Model pattern legible outside a closed lab: start from a strong base model, use verifiable reinforcement learning to elicit long Chain-of-Thought Reasoning, then distill those traces into smaller dense models. Its importance is not only that it was competitive with OpenAI o1 on several math and coding benchmarks, but that it exposed a reusable training pattern—especially Group Relative Policy Optimization—whose limits remain only partly understood.

Coverage note: verified through May 11, 2026.

1. Why DeepSeek-R1 mattered

DeepSeek-R1 was released into a moment when “reasoning model” mostly meant OpenAI o1: a closed model trained with large-scale reinforcement learning to spend more inference-time compute on internal reasoning before answering. OpenAI described o1 as using large-scale RL and chain-of-thought reasoning, with performance improving as both train-time and test-time compute increased; it reported strong AIME, Codeforces, and GPQA results, but did not release weights, traces, training data, or the training recipe in reproducible detail. OpenAI

DeepSeek’s contribution was different. The R1 report published an explicit two-track story: DeepSeek-R1-Zero, trained from DeepSeek-V3-Base using reinforcement learning without a prior supervised fine-tuning stage, and DeepSeek-R1, which added a small cold-start supervised dataset before RL and then used additional rejection-sampled data plus a second RL stage. DeepSeek also released R1-Zero, R1, and six dense distilled models based on Qwen and Llama checkpoints. arXiv

The result was not “fully open source” in the strict reproducibility sense: the model weights and code were permissively released, but the complete pretraining corpus, all intermediate data-generation choices, and the full RL production pipeline were not. Still, R1 moved frontier-style reasoning capability from “API-only closed model” toward “open-weights artifact plus partially documented recipe,” which was enough to trigger a wave of reproduction, evaluation, distillation, and deployment work in the open-source community. Hugging Face

2. The technical object: R1-Zero, R1, and the distills

The R1 family is best understood as three related artifacts rather than one model.

Artifact	Base	Training path	Main point
DeepSeek-R1-Zero	DeepSeek-V3-Base	RL directly from the base model, without supervised reasoning fine-tuning	Demonstrates that pure RL can elicit long reasoning, self-reflection, and strong math performance
DeepSeek-R1	DeepSeek-V3-Base	Cold-start SFT → reasoning RL → rejection-sampled SFT → general RL	Produces a more readable, usable, aligned reasoning model
R1-Distill models	Qwen and Llama dense bases	Supervised fine-tuning on R1-generated reasoning data	Transfers reasoning behavior into smaller, cheaper dense models

DeepSeek-V3-Base is not a small or ordinary dense model. It is a Mixture-of-Experts system with 671 billion total parameters and roughly 37 billion activated per token, using architectural components such as Multi-head Latent Attention and DeepSeekMoE; the V3 report says it was pretrained on 14.8 trillion tokens. arXiv

That matters because R1 was not “RL alone on a weak base.” It was RL applied to a very strong MoE base model with substantial math, code, and general pretraining behind it. Any serious account of R1 has to keep the base model in the causal graph.

3. DeepSeek-R1-Zero: pure RL as a reasoning elicitor

3.1 Setup

R1-Zero starts from DeepSeek-V3-Base and applies reinforcement learning with Group Relative Policy Optimization rather than first teaching the model to imitate human-written chains of thought. The prompt template asks the model to put reasoning inside <think>...</think> and the answer inside <answer>...</answer>. DeepSeek used two main reward types: accuracy rewards, such as answer checking for math or compiler-based verification for code, and format rewards, which enforce the requested structure. arXiv

DeepSeek explicitly avoided a learned neural reward model for this stage. The report says process reward models were hard to scale and vulnerable to reward hacking, while outcome-style rule rewards were simpler where answers could be verified. arXiv

This is the key Reinforcement Learning from Verifiable Rewards move: do not try to train a general judge for everything; instead, focus on domains where the environment can reliably say whether the final answer is correct.

3.2 Emergent behavior

R1-Zero’s most cited result is that pure RL changed the model’s reasoning behavior. DeepSeek reports AIME 2024 pass@1 improving from 15.6% to 71.0%, with majority voting reaching 86.7%. The report also describes longer thinking traces, spontaneous self-reflection, and exploration of alternative approaches emerging during training. arXiv

This was important because it weakened a common assumption: that a reasoning model must first be taught long chains of thought by supervised imitation before RL can work. R1-Zero suggested that, given a strong enough base model and verifiable rewards, RL can discover part of the behavior itself.

But R1-Zero also showed why pure RL was not enough. DeepSeek says R1-Zero suffered from poor readability and language mixing, which made the traces less useful to humans and less reliable as a production interface. arXiv

4. DeepSeek-R1: cold-start SFT plus RL

DeepSeek-R1 adds supervised structure before and after the main reasoning RL run.

The first addition is cold-start data: thousands of long chain-of-thought examples used to fine-tune the base model before RL. DeepSeek says these examples were produced through methods including few-shot prompting with long reasoning, prompting models to generate reflection and verification, collecting readable R1-Zero outputs, and manual refinement. arXiv

Then comes reasoning-oriented RL, similar in spirit to R1-Zero but starting from a model that already knows the intended answer format and has more readable reasoning behavior. DeepSeek also added a language-consistency reward to reduce language mixing. arXiv

After that RL stage, DeepSeek used the model to generate reasoning trajectories, filtered them with rejection sampling, and assembled roughly 600,000 reasoning-related examples. It also added roughly 200,000 non-reasoning examples for writing, factual QA, self-cognition, and translation. The resulting approximately 800,000-example dataset was used for another supervised fine-tuning stage, followed by a second RL stage for broader helpfulness and harmlessness. arXiv

The resulting model, DeepSeek-R1, is therefore not “R1-Zero with polish.” It is a multi-stage system:

1. cold-start supervised reasoning traces;

2. reasoning RL;

3. rejection-sampled reasoning data generation;

4. mixed reasoning and non-reasoning SFT;

5. broad RL for helpfulness, harmlessness, and general use.

That pipeline is one reason attribution is difficult. R1’s behavior is not the result of a single algorithmic trick; it is the result of a stack.

5. GRPO: the algorithmic pattern

5.1 From PPO to GRPO

Proximal Policy Optimization usually trains a policy model and a value model. In large-language-model RL, that value model can be expensive because it is often comparable in size to the policy. DeepSeekMath introduced Group Relative Policy Optimization, or GRPO, as a PPO-style method that removes the learned critic and instead estimates the baseline from a group of sampled responses to the same prompt. arXiv

For each question, GRPO samples a group of candidate completions from the old policy. Each completion receives a reward. The rewards are normalized relative to the group—subtracting the group mean and dividing by the group standard deviation—and that relative score becomes the advantage signal. The policy is then optimized with a clipped PPO-like objective plus a KL penalty to keep it near a reference policy. ar5iv

In informal terms: the model is not only rewarded for being correct; it is rewarded for being better than its siblings sampled from the same prompt.

5.2 Why GRPO fit R1

GRPO fits reasoning RL for three reasons.

First, reasoning tasks naturally support multiple attempts. A math problem can produce several candidate derivations; a code problem can produce several candidate programs. Group-relative comparison uses this multiplicity rather than treating each sample in isolation.

Second, many target domains have cheap, reasonably objective rewards. A math answer can often be checked. Code can be compiled or tested. That makes outcome rewards less speculative than learned preference rewards.

Third, removing the critic lowers memory and compute overhead relative to PPO. DeepSeekMath presents this as a resource advantage: GRPO avoids training an additional value-function model of comparable scale to the policy. ar5iv

This “data efficiency” claim needs precision. GRPO is more memory-efficient than critic-based PPO, and it can extract useful training signal from verifiable problems without human labels for every reasoning step. But it is not free: it samples multiple outputs per prompt, depends on high-quality prompts and reward functions, and consumes substantial generation compute. The efficiency is relative to a particular RLHF/RLAIF setup, not an absolute guarantee that reasoning capability is cheap.

5.3 The GRPO pattern

The reusable pattern is broader than the formula:

Component	Function
Strong pretrained base model	Supplies latent math, code, language, and instruction-following ability
Verifiable task distribution	Provides objective outcome rewards
Group sampling	Creates local comparisons among multiple candidate solutions
Relative advantage	Replaces a learned value model with group-normalized reward
KL regularization	Prevents the policy from drifting too far from the reference model
Long generation budget	Lets the model spend inference tokens on search, reflection, and correction
Rejection sampling	Converts successful traces into supervised training data
Distillation	Transfers expensive reasoning behavior into smaller models

This is why “GRPO” became shorthand in the open-source community for a family of reasoning-model recipes, even though the full R1 pipeline included much more than GRPO.

6. Architectural commonalities with o1 and divergences

DeepSeek-R1, OpenAI o1, and Anthropic’s later Claude reasoning models share a broad idea: use additional computation to reason before answering. They diverge sharply in openness, reasoning visibility, and architectural disclosure.

Axis	OpenAI o1	DeepSeek-R1	Claude reasoning models
Broad training idea	Large-scale RL for chain-of-thought reasoning	GRPO-style RL on DeepSeek-V3-Base, with cold-start SFT and later rejection-sampled SFT	Hybrid normal/extended-thinking model, with API-visible thinking controls
Raw reasoning visibility	Raw chain of thought hidden; user sees summaries	Reasoning emitted in visible <think>-style traces in many deployments	Claude 3.7 Sonnet introduced visible extended thinking and user/API control over thinking budget
Weights	Closed	Open weights, MIT-licensed DeepSeek release; distills also released subject to base-model licenses	Closed
Base architecture disclosure	Not publicly disclosed in comparable detail	DeepSeek-V3 MoE: 671B total, 37B active	Not publicly disclosed in comparable detail
Public recipe detail	High-level system card and blog; algorithm details not reproducible	Technical report describes GRPO, data stages, rewards, and distillation, but not full data/pipeline reproducibility	Product and safety documentation; less training-recipe detail
Main public benchmark framing	Competition math, Codeforces, GPQA, safety	Competition math, coding, GPQA, distillation	Real-world coding, agentic tasks, hybrid interaction

OpenAI’s o1 documentation says the model learns to think through a private chain of thought before answering and improves with more train-time and test-time compute. It also says OpenAI hides raw chains of thought, exposing model-generated summaries instead, partly because raw traces are valuable for monitoring and partly because of competitive considerations. OpenAI

DeepSeek diverged by releasing open weights and making the reasoning channel visible in the model’s natural output format. That visibility made R1 unusually useful for distillation and analysis, but it should not be confused with guaranteed faithful cognition. A generated chain of thought is still model text; it can be incomplete, post hoc, strategically shaped by training, or wrong even when fluent.

Anthropic’s Claude 3.7 Sonnet, released after R1, complicated the comparison. Anthropic described it as a “hybrid reasoning model” that can answer quickly or use extended step-by-step thinking, with API users able to control the thinking budget. Anthropic also emphasized real-world coding and agentic tasks more than contest math alone. Anthropic

7. Reported empirical results: R1 versus o1, Claude, and GPT-4o

DeepSeek’s R1 report compared DeepSeek-R1 against GPT-4o, Claude-3.5-Sonnet-1022, o1-mini, and o1-1217. The report states that o1-1217 results came from official reports rather than being independently rerun by DeepSeek, and it capped outputs at 32,768 tokens. Codeforces was evaluated through ten Div. 2 contests with expert-crafted tests. arXiv

7.1 Core benchmark table

Model	AIME 2024 pass@1	MATH-500 pass@1	Codeforces percentile	Codeforces rating
GPT-4o-0513	9.3	74.6	23.6	759
Claude-3.5-Sonnet-1022	16.0	78.3	20.3	717
DeepSeek-V3	39.2	90.2	58.7	1134
o1-mini	63.6	90.0	93.4	1820
o1-1217	79.2	96.4	96.6	2061
DeepSeek-R1	79.8	97.3	96.3	2029

Source: DeepSeek-R1 technical report, Table 4. arXiv

The table supports a narrow but important conclusion: on these reported math and competitive-programming metrics, R1 was roughly o1-class. It slightly exceeded o1-1217 on AIME 2024 pass@1 and MATH-500, while trailing slightly on Codeforces rating. It strongly exceeded GPT-4o and Claude-3.5-Sonnet-1022 on AIME, MATH-500, and Codeforces in DeepSeek’s reported setup. arXiv

The result should not be overgeneralized. Claude-3.5-Sonnet-1022 was not Anthropic’s later reasoning model. R1’s benchmark strengths were concentrated in math, code, and verifiable reasoning tasks. The R1 report itself notes that o1-1217 was stronger on Aider-Polyglot, a practical coding benchmark, even though R1 performed strongly on LiveCodeBench and Codeforces. arXiv

7.2 R1-Zero versus o1

R1-Zero’s comparison is especially informative because it isolates the pure-RL experiment more than the full R1 pipeline. DeepSeek reports:

Model	AIME 2024 pass@1	AIME 2024 consensus@64	MATH-500 pass@1	Codeforces rating
o1-mini	63.6	80.0	90.0	1820
o1-0912	74.4	83.3	94.8	1843
DeepSeek-R1-Zero	71.0	86.7	95.9	1444

Source: DeepSeek-R1 technical report, Table 2. arXiv

This table shows the split personality of R1-Zero. On math, it was very strong: MATH-500 exceeded the o1-0912 result, and AIME consensus@64 exceeded it as well. On Codeforces, however, R1-Zero’s rating lagged o1-mini and o1-0912. That makes R1-Zero a strong demonstration of verifiable-reward RL, not a universal proof that pure RL produces complete frontier reasoning behavior across all domains.

8. Independent evaluations and reproductions

Independent evaluation matters because reasoning benchmarks are sensitive to sampling budgets, prompt templates, contamination risk, tool use, answer parsing, and benchmark versioning.

8.1 Open-R1 reproduction work

Hugging Face’s Open-R1 project became one of the most important open attempts to reproduce and systematize the R1 recipe. Its repository includes SFT and GRPO training recipes and reports independent reproduction results for DeepSeek distill models. GitHub

Open-R1 also highlighted a reproducibility issue: DeepSeek’s paper used multiple responses per query to estimate pass@1, but did not specify the exact sample count for every benchmark. Open-R1 therefore estimated benchmark-specific sample counts—such as 64 for AIME, 4 for MATH-500, 8 for GPQA, and 16 for LiveCodeBench—and noted that discrepancies could be explained partly by sample-count choices and AIME’s high variance from only 30 problems. GitHub

The Open-R1 reproduction results were broadly close to DeepSeek’s reported distill results, often within one to three standard deviations. For example, Open-R1 reported MATH-500 scores for several R1 distills that were very close to DeepSeek’s paper values, while AIME had larger variance. GitHub

The takeaway is not that all R1 numbers are independently settled. The better conclusion is that the broad distillation story reproduced, while exact benchmark numbers remain sensitive to evaluation protocol.

8.2 Artificial Analysis and R1-0528

Artificial Analysis later evaluated DeepSeek-R1-0528, a May 2025 update. It reported a large increase in its Intelligence Index score and said the update had no architecture change: the model remained a 671B-total, 37B-active MoE, while improvements were attributed to post-training and reinforcement-learning progress. Artificial Analysis

That matters for the architecture-versus-training-data question. If a later R1 version improved substantially without changing the base architecture, that is evidence that post-training, data curation, reward design, and inference behavior can move benchmark performance materially even when architecture is fixed. It does not prove architecture is unimportant; it shows that architecture is not the only large lever.

8.3 Aider Polyglot practical coding benchmark

Aider’s independent coding leaderboard gives a different lens from contest programming. It reported original DeepSeek-R1 at 56.9% on Aider Polyglot, o1-2024-12-17 high at 61.7%, Claude 3.7 Sonnet thinking at 64.9%, and DeepSeek-R1-0528 at 71.4%. Aider

This illustrates three points. First, original R1 was not uniformly superior to o1 or Claude across practical coding harnesses. Second, later R1 post-training updates materially changed the picture. Third, “coding ability” is not a scalar: Codeforces, LiveCodeBench, SWE-bench, and Aider Polyglot measure overlapping but different behaviors.

9. Distillation: R1 traces as a capability substrate

One of R1’s most consequential results was not the 671B MoE itself, but the distillation of its reasoning traces into smaller dense models.

DeepSeek fine-tuned Qwen and Llama models on roughly 800,000 examples curated with R1, using supervised fine-tuning rather than RL for the distills. The released distilled models included Qwen-based 1.5B, 7B, 14B, and 32B models, plus Llama-based 8B and 70B models. arXiv

9.1 Distill benchmark results

Model	AIME 2024 pass@1	MATH-500 pass@1	GPQA Diamond pass@1	LiveCodeBench pass@1	Codeforces rating
GPT-4o-0513	9.3	74.6	49.9	32.9	759
Claude-3.5-Sonnet-1022	16.0	78.3	65.0	38.9	717
o1-mini	63.6	90.0	60.0	53.8	1820
R1-Distill-Qwen-7B	55.5	92.8	49.1	37.6	1189
R1-Distill-Qwen-14B	69.7	93.9	59.1	53.1	1481
R1-Distill-Qwen-32B	72.6	94.3	62.1	57.2	1691
R1-Distill-Llama-70B	70.0	94.5	65.2	57.5	1633

Source: DeepSeek-R1 technical report, Table 5. arXiv

The striking result is that the 32B distilled Qwen model exceeded o1-mini on AIME pass@1, MATH-500, GPQA Diamond, and LiveCodeBench in DeepSeek’s table, while still trailing o1-mini on Codeforces rating. The 14B distill was also competitive with o1-mini on several metrics. arXiv

This is why R1 mattered for access. A 671B-total MoE is expensive to serve. A 14B or 32B dense model can be run, fine-tuned, and inspected by a much wider community. Distillation converted frontier-model behavior into a form that was more usable for local deployment and downstream research.

9.2 Distillation versus small-model RL

DeepSeek also compared distillation with applying large-scale RL directly to a smaller model. In the paper’s Qwen-32B comparison, R1-Distill-Qwen-32B substantially outperformed R1-Zero-Qwen-32B on AIME, MATH-500, GPQA Diamond, LiveCodeBench, and Codeforces. DeepSeek concluded that distilling from a stronger teacher can be more effective than running RL directly on a smaller base, while pushing the frontier still requires stronger base models and larger-scale RL. arXiv

That conclusion is important for Reasoning Model Distillation. The smaller model is not independently discovering the full behavior. It is inheriting a distribution of reasoning traces from a stronger system. This makes distillation a capability amplifier, but also a dependency: the quality of the student depends on the teacher’s traces, filtering process, and coverage.

10. Reception in the open-source community

R1’s reception was unusually strong because it arrived with three things the community could immediately use: open weights, visible reasoning traces, and distilled dense models.

DeepSeek’s GitHub release says the R1 and R1-Zero weights were released with 671B total parameters, 37B activated parameters, 128K context length, and an MIT license permitting commercial use, modifications, derivative works, and distillation, while the distilled models remain subject to their respective Qwen and Llama base-model licenses. GitHub

The release therefore enabled several forms of community work:

Community activity	Why R1 enabled it
Local inference experiments	Open weights and dense distills made reasoning models runnable outside closed APIs
Distillation and fine-tuning	Visible traces gave researchers teacher data for smaller students
RL recipe reproduction	GRPO and verifiable-reward framing gave a concrete training pattern
Benchmark auditing	Open models allowed independent reruns under different prompts and sampling budgets
Systems work	Serving a 671B-total MoE forced optimization in inference engines and hosted APIs

Hugging Face’s Open-R1 project is a representative example. It provides recipes for SFT and GRPO training, reports reproduction benchmarks, and documents evaluation choices needed to compare against DeepSeek’s numbers. GitHub

The open-source reception also exposed a semantic problem: “open model” can mean several different things. R1 was open-weights and permissively licensed, but not fully open in the sense of releasing every dataset, data filter, reward script, intermediate checkpoint, and training run needed to reproduce it end to end. That distinction matters because the most valuable part of a reasoning model may be the data-and-training pipeline, not only the final weights.

11. Implications for frontier reasoning capability access

R1 changed the access frontier in four ways.

First, it made high-end reasoning behavior available without relying exclusively on closed APIs. Even when the full R1 model was expensive to host, the dense distills gave researchers and builders practical artifacts.

Second, it made reasoning traces available as training material. OpenAI’s o1 explicitly hides raw chain of thought, while R1-style models often emit it. That difference made R1 much more useful for distillation, auditing, and mechanistic curiosity, although the traces are not guaranteed faithful. OpenAI

Third, it lowered the barrier for RL experimentation. GRPO gave practitioners a critic-free method to try on verifiable tasks, and the Open-R1 ecosystem provided recipes for SFT and GRPO reproduction. ar5iv

Fourth, it sharpened the distinction between frontier capability and frontier infrastructure. R1 showed that frontier-like reasoning could be distributed as weights, but serving the largest model still required serious infrastructure. Artificial Analysis noted that R1-0528 remained difficult to host because a 671B-parameter model does not fit comfortably on a single 8×H100 node in native FP8. Artificial Analysis

The practical democratization came less from everyone running the full R1 model and more from three downstream effects: hosted providers competing on price, distills running locally, and open researchers adapting the recipe.

12. The unresolved attribution question: architecture, RL, or data engineering?

The hardest interpretive question is how much of R1’s performance came from:

• DeepSeek-V3’s MoE architecture and pretraining;

• math and code data quality;

• GRPO and rule-based RL;

• cold-start supervised traces;

• rejection sampling;

• distillation;

• inference-time token budget;

• benchmark-specific prompting and filtering.

The honest answer is that the public evidence does not support a clean decomposition.

12.1 Evidence for architecture and base-model strength

R1 starts from DeepSeek-V3-Base, a 671B-total, 37B-active MoE pretrained on 14.8T tokens. The R1 table shows DeepSeek-V3 already scoring 39.2 on AIME 2024 and 90.2 on MATH-500 before R1 post-training, far above GPT-4o and Claude-3.5-Sonnet-1022 on those two metrics in DeepSeek’s evaluation. arXiv

So architecture and pretraining clearly matter. RL did not create reasoning ability from nothing; it amplified and reorganized capabilities latent in a strong base model.

12.2 Evidence for RL

R1-Zero is the best evidence for RL as an independent lever. DeepSeek reports AIME 2024 pass@1 rising from 15.6 to 71.0 during RL, with longer reasoning and reflective behaviors emerging. That is too large to dismiss as mere prompting. arXiv

But R1-Zero’s weaknesses—readability, language mixing, weaker Codeforces score than o1—also show the limits of pure RL. The full R1 model needed cold-start data, filtering, and additional supervised training.

12.3 Evidence for data engineering

The R1 pipeline is heavy with data engineering: thousands of cold-start traces, language-consistency rewards, rejection sampling, 600,000 reasoning examples, 200,000 non-reasoning examples, and supervised fine-tuning on the resulting mixture. arXiv

DeepSeekMath, the earlier GRPO source, also demonstrates the importance of domain data. It built a 120B-token math corpus from Common Crawl using classification, deduplication, and iterative expansion, and combined that with GRPO to improve math reasoning. ar5iv

The distillation results further support the data-engineering view. Smaller dense models trained on R1 traces outperformed models that tried to learn through RL alone at comparable scale. arXiv

12.4 Evidence from later R1 updates

Artificial Analysis’s R1-0528 evaluation says DeepSeek improved its R1 model substantially without changing architecture, attributing the gains to post-training and RL improvements. That supports the claim that post-training can drive large changes after architecture is fixed. Artificial Analysis

But this does not prove that architecture is secondary. A fixed architecture can still be a necessary substrate. The better interpretation is conditional: given a strong MoE base, post-training and data engineering can move reasoning performance dramatically.

12.5 What evidence is missing

A decisive attribution study would require controlled ablations that are not publicly available:

Missing ablation	What it would test
Same RL recipe on weaker and stronger bases	How much GRPO depends on base-model capability
Same base with and without cold-start traces	How much readability and performance come from SFT
Same RL rewards with different prompt distributions	How much task selection matters
Same model with different inference budgets	How much performance comes from test-time compute
Same distill data across different student architectures	How much the student base matters
Full contamination audits	Whether benchmark familiarity leaked through pretraining or generated data

Open-R1’s evaluation notes already show that even reproducing pass@1 can require assumptions about sample counts and benchmark variance. GitHub

The most defensible conclusion is not “GRPO solved reasoning” or “architecture did all the work.” It is that R1 combined a strong MoE base, verifiable-reward RL, careful data construction, long inference traces, and distillation into a mutually reinforcing pipeline.

13. The R1 pattern as a design template

R1’s long-term influence may be as a template rather than a single model.

The template looks like this:

1. Pretrain or obtain a strong base model with enough math, code, and language competence to explore useful solution paths.

2. Choose verifiable domains where outcome rewards are cheap and reliable.

3. Use group-relative RL to reward better candidates without training a full value model.

4. Allow long reasoning traces so the model can search, backtrack, and self-correct.

5. Add cold-start data when pure RL produces unreadable or unstable behavior.

6. Rejection-sample successful traces into a supervised corpus.

7. Distill the corpus into smaller dense models for access, cost reduction, and downstream specialization.

This is a practical recipe for Self-Improving Systems in the narrow sense: the model generates candidate reasoning traces, external verifiers select better traces, and the selected traces become training data. It is not autonomous general self-improvement. The verifier, task distribution, filtering rules, and base model remain externally designed.

14. Limitations and failure modes

R1’s success should not obscure several limitations.

14.1 Verifier dependence

GRPO works best where rewards are reliable. Math answers, code tests, and formal checks are natural targets. Open-ended research, philosophy, product design, and ambiguous real-world tasks are harder because the reward function is contestable.

14.2 Trace faithfulness

Visible reasoning is useful, but not equivalent to faithful cognition. A model can produce a plausible trace that omits decisive internal heuristics, rationalizes a guessed answer, or follows learned stylistic conventions. OpenAI’s decision to hide raw chain of thought and Anthropic’s visible extended thinking represent different product and safety tradeoffs, not a settled scientific answer about trace truthfulness. OpenAI

14.3 Benchmark overfitting

Reasoning models are often evaluated on small or highly recognizable benchmarks. AIME has only 30 problems per year, and Open-R1 noted high variance in AIME reproduction. Benchmarks remain useful, but small score differences should not be treated as stable capability orderings. GitHub

14.4 Distillation narrows the behavior

Distilled models can be surprisingly strong, but they inherit a filtered teacher distribution. They may imitate the surface form of reasoning without the same search depth, robustness, or out-of-domain competence as the teacher. This is especially important when deploying small distills as if they were full R1.

14.5 Open weights are not open science

R1 made a major contribution to access, but without full data provenance and training logs, outside researchers cannot fully audit the sources of capability, contamination risk, or safety behavior. Open weights are valuable; they are not the same as full reproducibility.

15. Bottom line

DeepSeek-R1 should be read as a systems result. The headline was that an open-weights model reached o1-class performance on major math and coding benchmarks, but the deeper contribution was a public pattern: strong base model, verifiable rewards, GRPO, long visible reasoning, rejection-sampled traces, and distillation.

R1-Zero showed that pure RL can elicit long reasoning from a strong base model. R1 showed that production-grade reasoning benefits from cold-start supervision, data filtering, and additional alignment. The distills showed that reasoning traces themselves can become a portable capability substrate.

The unresolved question is not whether architecture or training data mattered. Both did. The live research question is how to apportion credit among base-model scale, MoE design, pretraining corpus, verifiable RL, cold-start traces, rejection sampling, and inference-time compute—and how far the same pattern extends beyond domains where correctness can be mechanically checked.

Companion entries

Core theory: Reasoning Model, Inference-Time Scaling, Chain-of-Thought Reasoning, Reinforcement Learning from Verifiable Rewards, Group Relative Policy Optimization, Proximal Policy Optimization, Process Reward Models

Model architecture: Mixture-of-Experts, DeepSeekMoE, Multi-head Latent Attention, Multi-Token Prediction, Sparse Activation

Training practice: Cold-Start Supervised Fine-Tuning, Rejection Sampling, Reasoning Model Distillation, Synthetic Data, Reward Hacking, KL Regularization

Evaluation: AIME Benchmark, MATH-500, Codeforces Evaluation, LiveCodeBench, Aider Polyglot Benchmark, Benchmark Contamination, Pass@k and Consensus Evaluation

Open-source and access: Open Weights vs Open Source, Open-R1, Local Reasoning Models, Model Serving for MoE Systems, Frontier Capability Access

Counterarguments and open questions: Architecture vs Data in Frontier Models, Trace Faithfulness, Hidden Chain-of-Thought, Verifier Dependence, Distillation Limits