DeepSeek-R1 is an open-source language model built on DeepSeek-V3-Base that's been making waves in the AI neighborhood. Not only does it match-or even surpass-OpenAI's o1 model in numerous criteria, however it also comes with totally MIT-licensed weights. This marks it as the very first non-OpenAI/Google design to provide strong reasoning capabilities in an open and available manner.

What makes DeepSeek-R1 particularly interesting is its openness. Unlike the less-open methods from some market leaders, DeepSeek has released a detailed training method in their paper. The design is likewise extremely economical, with input tokens costing simply $0.14-0.55 per million (vs o1's $15) and output tokens at $2.19 per million (vs o1's $60).

Until ~ GPT-4, the typical wisdom was that better models needed more data and calculate. While that's still valid, models like o1 and R1 show an alternative: inference-time scaling through reasoning.

The Essentials

The DeepSeek-R1 paper provided multiple models, however main among them were R1 and R1-Zero. Following these are a series of distilled designs that, while fascinating, I will not discuss here.

DeepSeek-R1 utilizes two significant ideas:

1. A multi-stage pipeline where a little set of cold-start information kickstarts the design, followed by massive RL.

1. Group Relative Policy Optimization (GRPO), a reinforcement learning approach that depends on comparing multiple model outputs per timely to avoid the need for a separate critic.
   
   R1 and R1-Zero are both reasoning designs. This essentially implies they do Chain-of-Thought before answering. For the R1 series of models, this takes kind as believing within a tag, before addressing with a final summary.
   
   R1-Zero vs R1
   
   R1-Zero uses Reinforcement Learning (RL) straight to DeepSeek-V3-Base with no monitored fine-tuning (SFT). RL is used to enhance the design's policy to make the most of benefit. R1-Zero attains excellent accuracy but often produces confusing outputs, such as blending multiple languages in a single reaction. R1 repairs that by incorporating limited supervised fine-tuning and multiple RL passes, which improves both correctness and readability.
   
   It is fascinating how some languages may express certain ideas much better, which leads the design to select the most meaningful language for the job.
   
   Training Pipeline
   
   The training pipeline that DeepSeek published in the R1 paper is profoundly fascinating. It showcases how they produced such strong reasoning models, and what you can get out of each stage. This consists of the issues that the resulting designs from each phase have, and how they resolved it in the next phase.
   
   It's fascinating that their training pipeline differs from the normal:
   
   The usual training method: Pretraining on large dataset (train to predict next word) to get the base design → monitored fine-tuning → choice tuning via RLHF R1-Zero: Pretrained → RL R1: Pretrained → Multistage training pipeline with multiple SFT and RL phases
   
   Cold-Start Fine-Tuning: Fine-tune DeepSeek-V3-Base on a couple of thousand Chain-of-Thought (CoT) samples to guarantee the RL procedure has a decent starting point. This gives a good design to begin RL. First RL Stage: Apply GRPO with rule-based rewards to enhance thinking accuracy and format (such as forcing chain-of-thought into thinking tags). When they were near convergence in the RL process, they relocated to the next step. The outcome of this action is a strong reasoning model however with weak basic abilities, e.g., bad formatting and language blending. Rejection Sampling + basic data: Create brand-new SFT information through rejection tasting on the RL checkpoint (from step 2), combined with monitored information from the DeepSeek-V3-Base model. They collected around 600k high-quality reasoning samples. Second Fine-Tuning: Fine-tune DeepSeek-V3-Base again on 800k total samples (600k reasoning + 200k general jobs) for more comprehensive abilities. This step resulted in a strong reasoning design with general capabilities. Second RL Stage: Add more reward signals (helpfulness, harmlessness) to improve the last design, in addition to the thinking benefits. The outcome is DeepSeek-R1. They likewise did model distillation for several Qwen and Llama designs on the thinking traces to get distilled-R1 models.
   
   Model distillation is a technique where you use an instructor design to enhance a trainee design by generating training information for the trainee model. The instructor is usually a larger design than the trainee.
   
   Group Relative Policy Optimization (GRPO)
   
   The basic concept behind using support learning for LLMs is to tweak the model's policy so that it naturally produces more accurate and useful answers. They used a reward system that inspects not only for correctness but likewise for correct format and language consistency, so the design gradually finds out to favor reactions that meet these quality requirements.
   
   In this paper, they encourage the R1 design to create chain-of-thought reasoning through RL training with GRPO. Instead of adding a separate module at inference time, the training process itself pushes the design to produce detailed, detailed outputs-making the chain-of-thought an emerging habits of the enhanced policy.
   
   What makes their method particularly fascinating is its dependence on straightforward, rule-based benefit . Instead of depending on costly external models or human-graded examples as in standard RLHF, the RL used for R1 uses basic criteria: it might give a higher reward if the response is right, if it follows the expected/ formatting, and if the language of the answer matches that of the prompt. Not depending on a reward design also implies you don't have to invest time and effort training it, and it does not take memory and compute away from your main design.
   
   GRPO was introduced in the DeepSeekMath paper. Here's how GRPO works:
   
   1. For each input prompt, the design produces various reactions.
2. Each action receives a scalar benefit based on elements like precision, format, and language consistency.
3. Rewards are adjusted relative to the group's efficiency, basically measuring how much better each action is compared to the others.
4. The model updates its method slightly to prefer responses with greater relative benefits. It only makes slight adjustments-using techniques like clipping and a KL penalty-to make sure the policy does not stray too far from its original habits.
   
   A cool aspect of GRPO is its versatility. You can use simple rule-based reward functions-for instance, awarding a perk when the design properly uses the syntax-to guide the training.
   
   While DeepSeek used GRPO, you could use alternative approaches rather (PPO or PRIME).
   
   For those aiming to dive deeper, Will Brown has composed rather a nice execution of training an LLM with RL utilizing GRPO. GRPO has actually also already been contributed to the Transformer Reinforcement Learning (TRL) library, which is another great resource. Finally, fraternityofshadows.com Yannic Kilcher has a fantastic video explaining GRPO by going through the DeepSeekMath paper.
   
   Is RL on LLMs the path to AGI?
   
   As a final note on explaining DeepSeek-R1 and the approaches they've presented in their paper, I want to highlight a passage from the DeepSeekMath paper, based upon a point Yannic Kilcher made in his video.
   
   These findings suggest that RL boosts the model's general performance by rendering the output distribution more robust, to put it simply, it appears that the enhancement is credited to improving the right action from TopK rather than the enhancement of basic abilities.
   
   In other words, RL fine-tuning tends to shape the output circulation so that the highest-probability outputs are more most likely to be appropriate, even though the general capability (as determined by the variety of right responses) is mainly present in the pretrained model.
   
   This recommends that support learning on LLMs is more about refining and "shaping" the existing distribution of responses instead of endowing the design with completely new capabilities. Consequently, while RL strategies such as PPO and GRPO can produce significant efficiency gains, there appears to be a fundamental ceiling identified by the underlying model's pretrained knowledge.
   
   It is uncertain to me how far RL will take us. Perhaps it will be the stepping stone to the next big milestone. I'm thrilled to see how it unfolds!
   
   Running DeepSeek-R1
   
   I've utilized DeepSeek-R1 by means of the main chat interface for different issues, which it appears to fix all right. The extra search performance makes it even nicer to utilize.
   
   Interestingly, o3-mini(-high) was released as I was composing this post. From my preliminary screening, R1 appears more powerful at mathematics than o3-mini.
   
   I also rented a single H100 by means of Lambda Labs for $2/h (26 CPU cores, 214.7 GB RAM, 1.1 TB SSD) to run some experiments. The main goal was to see how the design would perform when released on a single H100 GPU-not to thoroughly evaluate the model's capabilities.
   
   671B through Llama.cpp
   
   DeepSeek-R1 1.58-bit (UD-IQ1_S) quantized design by Unsloth, with a 4-bit quantized KV-cache and partial GPU offloading (29 layers working on the GPU), running via llama.cpp:
   
   29 layers appeared to be the sweet area provided this configuration.
   
   Performance:
   
   A r/localllama user explained that they were able to overcome 2 tok/sec with DeepSeek R1 671B, without utilizing their GPU on their regional gaming setup. Digital Spaceport composed a complete guide on how to run Deepseek R1 671b totally locally on a $2000 EPYC server, on which you can get ~ 4.25 to 3.5 tokens per second.
   
   As you can see, the tokens/s isn't quite bearable for any serious work, however it's enjoyable to run these large models on available hardware.
   
   What matters most to me is a combination of effectiveness and time-to-usefulness in these designs. Since reasoning designs need to believe before addressing, their time-to-usefulness is normally higher than other designs, but their usefulness is likewise usually higher. We need to both make the most of usefulness and minimize time-to-usefulness.
   
   70B by means of Ollama
   
   70.6 b params, 4-bit KM quantized DeepSeek-R1 running via Ollama:
   
   GPU usage shoots up here, as anticipated when compared to the mainly CPU-powered run of 671B that I showcased above.
   
   Resources
   
   DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [2402.03300] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models DeepSeek R1 - Notion (Building a fully local "deep scientist" with DeepSeek-R1 - YouTube). DeepSeek R1's dish to reproduce o1 and the future of thinking LMs. The Illustrated DeepSeek-R1 - by Jay Alammar. Explainer: What's R1 & Everything Else? - Tim Kellogg. DeepSeek R1 Explained to your grandma - YouTube
   
   DeepSeek
   
   - Try R1 at chat.deepseek.com. GitHub - deepseek-ai/DeepSeek-R 1. deepseek-ai/Janus-Pro -7 B · Hugging Face (January 2025): Janus-Pro is an unique autoregressive framework that unifies multimodal understanding and generation. It can both comprehend and create images. DeepSeek-R1: Incentivizing Reasoning Capability in Large Language Models by means of Reinforcement Learning (January 2025) This paper introduces DeepSeek-R1, an open-source thinking model that rivals the performance of OpenAI's o1. It presents a detailed method for training such designs using massive support learning strategies. DeepSeek-V3 Technical Report (December 2024) This report talks about the application of an FP8 mixed accuracy training structure confirmed on an exceptionally large-scale design, attaining both sped up training and lowered GPU memory use. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (January 2024) This paper looks into scaling laws and provides findings that facilitate the scaling of massive models in open-source configurations. It presents the DeepSeek LLM project, committed to advancing open-source language designs with a long-term point of view. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence (January 2024) This research study presents the DeepSeek-Coder series, a variety of open-source code models trained from scratch on 2 trillion tokens. The designs are pre-trained on a high-quality project-level code corpus and utilize a fill-in-the-blank task to boost code generation and infilling. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (May 2024) This paper presents DeepSeek-V2, a Mixture-of-Experts (MoE) language design identified by economical training and effective reasoning. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (June 2024) This research introduces DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that attains efficiency equivalent to GPT-4 Turbo in code-specific tasks.
   
   Interesting occasions
   
   - Hong Kong University replicates R1 results (Jan 25, '25). - Huggingface reveals huggingface/open-r 1: Fully open recreation of DeepSeek-R1 to duplicate R1, completely open source (Jan 25, '25).
5. OpenAI researcher verifies the DeepSeek team individually discovered and utilized some core ideas the OpenAI group used en route to o1
   
   Liked this post? Join the newsletter.