DeepSeek MIT License vs. Attention is All You Need

How DeepSeek trained a frontier-class model on a fraction of the budget — RL-first training, distillation and FP8, compared line by line against the 2017 Transformer.

Queries and edits by Kelly Nyland in collaboration w/ Chat GPT

How did DeepSeek train their model on a fraction of the budget and time?

Here’s how DeepSeek achieved significant efficiency and performance in training their models, according to their MIT License, compared to the original Transformer (as described in Attention Is All You Need):

Resources:

FileREADME_Deepseek.md PDFAttention is All You Need.pdf

Comparison to the Transformer (Attention Is All You Need)

Feature	Transformer (2017)	DeepSeek
	Explain this to me in simple terms.	Explain this to me in simple terms.
Core Training Approach	Supervised training with labeled data (e.g., machine translation tasks).	Reinforcement learning (RL) without initial SFT; fine-tuning via distilled data for scalability.
	Think of this robot like a kid learning in school. It needs a teacher to give it specific examples and tell it, “This is the correct answer.” It learns by copying those examples over and over.	This robot skips the teacher at first. Instead, it plays games where it gets rewards for good behavior (like solving puzzles). Later, it learns extra tricks by looking at smaller, simpler robots to figure out how to work faster.
Efficiency in Training	Required extensive training on GPUs for days; limited by sequential computation (e.g., self-attention).	RL drastically reduced training time; distillation produced smaller, efficient models.
	Imagine this robot having to study for weeks in a library with super-slow computers. It learns one step at a time, which takes forever, especially for big tasks.	This robot learns faster because it skips the library phase and starts playing games to get better right away. Then it shrinks itself down into smaller, smarter versions (like saving a cheat sheet), which makes it much faster.
Model Parallelism	Attention layers were parallelized but computationally intensive for long sequences.	Optimized parallelism with 128K token contexts and reinforcement-based efficiency.
	When the Transformer robot works on a big, messy problem, it can split the task into chunks and work on it at the same time (parallel). But if the problem is really big, like reading a super-long book, it still struggles because each chunk needs extra power.	This robot is much better at handling huge problems because it’s designed to work on longer pieces all at once (like reading a long book without getting tired). It uses smarter teamwork to solve tasks more efficiently.
Handling Long-Range Dependencies	Addressed via self-attention with complexity, which became costly for long inputs.	Improved token-to-token scaling via RL-based reasoning, enabling better handling of long dependencies.
	Imagine this robot trying to remember the beginning of a really long story. It keeps forgetting important details because it uses a system that works well for short stories but struggles with long ones.	This robot has a better memory system. It uses reasoning (like thinking step by step) to remember important details from the start, middle, and end of long stories. It’s like having a notebook to keep track of everything.
Distillation	No direct emphasis on distilling models for smaller scales.	Systematic distillation to create high-performance smaller models like Qwen-1.5B, -7B, and Llama-based.
	This robot doesn’t focus on becoming smaller or teaching smaller versions of itself how to do the same job.	DeepSeek creates mini-versions of itself (like baby robots) that are just as smart but smaller and faster. These baby robots save space, energy, and time.
Generalization	Focused on translation and text tasks, with limited extensions to other tasks.	Applied CoT reasoning and RL, enabling generalization across math, code, and multilingual benchmarks.
	This robot is really good at specific tasks, like translating languages or summarizing text. But if you ask it to solve math problems or write code, it might struggle because it wasn’t trained for that.	DeepSeek is like a jack-of-all-trades robot. It’s trained to think through problems step by step (chain of thought), so it can handle math, coding, languages, and even solving tricky puzzles.

Highlights of DeepSeek’s Innovations:

RL-First Training: RL eliminated dependency on human-annotated datasets, a significant cost-saving and flexibility-enhancing feature.
Distillation Pipeline: Creating high-performing smaller models enhanced scalability and deployment opportunities.
Computation Reduction: Improved methods for long-context tasks and token efficiency cut computational resource requirements.

Key Steps Taken by DeepSeek

Large-Scale Reinforcement Learning (RL):

DeepSeek-R1-Zero: They bypassed supervised fine-tuning (SFT) initially, directly employing RL to train their base models. This incentivized reasoning behaviors like chain-of-thought (CoT), self-verification, and reflection.

What is Chain of Thought (CoT)?

Chain of Thought (CoT) refers to a reasoning technique used in large language models (LLMs) where the model generates intermediate steps or a structured reasoning process before arriving at a final answer. It mirrors how humans often solve complex problems by breaking them down into smaller, logical steps.

Key Features of Chain of Thought:

Step-by-Step Reasoning:

Instead of directly producing an output (e.g., a simple “yes” or “42”), CoT prompts the model to “think out loud,” explaining each step in the reasoning process.

Example:

Question: If John has 3 apples, gives 1 to Mary, and then buys 5 more, how many does he have now?

CoT Reasoning: “John starts with 3 apples. He gives 1 to Mary, leaving him with 2. Then he buys 5 more, so he now has 2 + 5 = 7 apples.”

Final Answer: 7.

Improved Problem-Solving:

It helps the model tackle tasks like math problems, logical reasoning, multi-step queries, and ambiguous instructions.

Increased Transparency:

CoT allows humans to understand how the model arrived at its answer, making the process more interpretable and debuggable.

How CoT Works in Practice

Prompt Engineering:

CoT often starts with specifically designed prompts, such as “Think step by step” or “Explain your reasoning.”

Example Prompt: “Please reason step by step and include all calculations in your response.”

Training with Examples:

Models trained with CoT rely on datasets containing reasoning steps or logic chains to encourage step-by-step thinking.

Benefits of CoT in LLMs

Handles Complex Queries: Models can solve multi-step problems that are difficult for traditional direct-response models.
Reduces Errors: By reasoning step by step, CoT minimizes the likelihood of logical mistakes.
Enhanced Generalization: CoT allows models to perform better in tasks they were not explicitly trained for, such as novel math or logic problems.

What is Supervised Fine-Tuning (SFT)

Supervised Fine-Tuning (SFT) is a technique used in machine learning, particularly for training large language models (LLMs), where a pre-trained model is further trained on a specific dataset with labeled examples. This process aligns the model’s output to desired behaviors by providing explicit guidance on how to respond in particular scenarios.

Key Aspects of Supervised Fine-Tuning:

Starting Point:

The model begins with a pre-trained base, typically trained on large-scale, general-purpose datasets like books, articles, and websites.

Pre-training focuses on predicting the next token in a sequence, enabling the model to learn general language patterns and knowledge.

Fine-Tuning Objective:

SFT narrows the model’s focus by training it on task-specific or domain-specific labeled data.

Examples include datasets with:

QA pairs for question answering tasks.
Code snippets for programming tasks.
Step-by-step reasoning examples for Chain of Thought (CoT) tasks.

Labels as Guidance:

Each example in the fine-tuning dataset includes both an input and a correct output (label), teaching the model the correct behavior for specific scenarios.

Example:

Input: “Write a summary of the following paragraph.”
Label (Output): A short, well-structured summary of the given paragraph.

Loss Function:

During SFT, the model minimizes the difference between its predicted output and the labeled target using a loss function, such as cross-entropy loss.
This helps the model adjust weights to produce responses closer to the desired outputs.

Benefits of SFT:

Specialization:

Aligns the model to specific tasks, domains, or datasets.
Example: Fine-tuning GPT models for medical diagnoses using healthcare datasets.

Improved Performance:

Enhances accuracy and relevance on tasks where the pre-trained model might otherwise struggle or provide generic responses.

Human-Preferred Outputs:

Ensures the model produces outputs that align with human expectations, such as polite tone or consistent logic.

Challenges of SFT:

Data Dependency:

Requires high-quality labeled data, which can be time-consuming and costly to curate.

Poor-quality labels can bias or degrade the model’s performance.

Overfitting:

Fine-tuning on narrow datasets may reduce the model’s ability to generalize to unseen tasks or domains.

Computational Costs:

Training large models with SFT can be computationally expensive, particularly for massive datasets.

Supervised Fine-Tuning in Contrast to DeepSeek’s RL Approach:

DeepSeek reduces reliance on SFT by employing Reinforcement Learning (RL), which allows models to learn reasoning capabilities directly by maximizing reward signals rather than relying on labeled data.

SFT is used later in their pipeline (after RL stages) to seed specific reasoning and non-reasoning capabilities.

Efficiency: RL allowed DeepSeek to focus on discovery and reinforcement of reasoning capabilities without additional human-annotated fine-tuning steps, saving time and computational costs.

Optimized Pipeline for Model Training:

Two-Stage RL Process: Used sequential RL phases to refine reasoning and align the model with human preferences.

Cold-Start Data Inclusion: Unlike the initial RL-only phase, DeepSeek-R1 introduced pre-RL fine-tuning (cold-start data) in DeepSeek-R1, resolving issues like repetition and language mixing.

What is Cold-Start?

Cold Start refers to the challenge of initializing a system, process, or model when there is little or no prior data, context, or user interaction to guide its performance. This term is commonly used in machine learning, recommendation systems, and software development.

Cold Start in Machine Learning (ML):

In ML, a cold start typically refers to the situation where:

A model or system begins with no prior training or knowledge.
It lacks historical data or labeled examples to learn from.
It initially struggles to perform well until sufficient data is gathered or the model is fine-tuned.

Cold Start in Supervised Learning:

When training a model for a specific task (e.g., summarization or question answering), a cold start occurs if:

There’s no pre-trained model to build upon.
You train the model from scratch using a limited dataset.

This can result in:

Slower learning: The model must learn everything from the ground up.
Higher resource costs: Training from scratch is computationally expensive.
Suboptimal performance initially: Until sufficient training occurs, the model may perform poorly.

Cold Start in DeepSeek’s Context:

DeepSeek’s mention of cold-start data involves strategies to address the lack of initial data or performance issues:

Cold-Start Data for Pre-RL Fine-Tuning:

DeepSeek introduced labeled data or training steps before reinforcement learning (RL) to create a baseline of reasoning and language generation skills.
This step prevents common issues such as repetition, incoherent outputs, or incomplete reasoning when starting from scratch.

Advantages of Cold-Start Fine-Tuning:

Stabilizes RL Training: By seeding the model with basic reasoning and language skills, the subsequent RL process starts with a stronger foundation.
Improves Generalization: It allows the model to avoid learning purely from trial-and-error behaviors (which RL alone might emphasize).

Real-World Analogy:

Think of training a model with cold-start data like giving someone a basic instruction manual before they figure out a task through experimentation. Without the manual, they’d struggle to even start.

DeepSeek’s Cold Start vs OpenAI’s GPT Training

Feature	DeepSeek Cold-Start	OpenAI GPT (Transformer)
Purpose	Provides initial reasoning & coherence before RL.	Pre-training learns general patterns from scratch.
Data Requirements	Uses curated seed data to kickstart training.	Requires vast, diverse datasets for pre-training.
Effectiveness	Stabilizes RL while improving downstream performance.	Focuses on broad, generalizable language learning.

What are pre-RL methods?

DeepSeek effectively tackled the cold start problem by carefully incorporating labeled data or supervised training steps before their reinforcement learning (RL) phase to establish a baseline for reasoning and language generation. This foundational approach stabilized their RL process and improved overall performance. Here’s how they likely achieved this:

Steps Taken to Introduce Labeled Data or Training Steps:

Curated Cold-Start Dataset:

DeepSeek likely used a curated dataset tailored to the reasoning tasks they aimed to improve. These datasets would contain high-quality labeled examples for tasks such as:
- Logical reasoning.
- Math problems (step-by-step reasoning).
- Code generation with correct syntax.
- Multilingual language understanding.

By exposing the model to structured examples early, it learned the basics of generating coherent and logical outputs.

Supervised Fine-Tuning (SFT):

The model underwent supervised fine-tuning on these labeled datasets before transitioning to RL.

Purpose of SFT: To seed the model with basic reasoning skills, ensuring it could handle structured tasks like step-by-step problem solving.

Example:

Input Prompt: “Solve this equation: 2x + 3 = 7. Show all steps.”
Expected Output (Label): “Step 1: Subtract 3 from both sides → 2x = 4. Step 2: Divide by 2 → x = 2.”

Focus on Chain-of-Thought (CoT):

The labeled data likely included examples encouraging step-by-step reasoning (CoT), teaching the model to break down complex tasks.
This method ensured that the model could logically navigate reasoning problems, even during the early stages of training.

General and Task-Specific Training:

The pre-RL stage could have been divided into:

General Training: Using diverse datasets for math, logic, and language generation to create broadly applicable reasoning capabilities.
Task-Specific Training: Focused on particular domains (e.g., math benchmarks, coding problems) that DeepSeek prioritized.

Mitigation of Common Model Issues:

By including labeled examples, DeepSeek resolved issues like:

Endless Repetition: Labeled data helped the model learn when to conclude outputs.
Language Mixing: Clear examples in target languages reduced confusion between languages.
Poor Readability: Exposure to human-preferred structured outputs improved clarity and fluency.

Why This Worked Effectively:

Stabilized Reinforcement Learning:

RL models typically rely on trial-and-error to optimize for reward signals. Starting with a cold-trained model could result in erratic behaviors.
By introducing SFT beforehand, the model entered RL with a stronger foundation, allowing it to focus on refining rather than reinventing core reasoning skills.

Transfer Learning Advantages:

By fine-tuning on labeled datasets, the pre-trained model’s vast general knowledge was transferred and adapted for specific reasoning tasks.
This transfer learning approach saved computational resources while improving task-specific performance.

Efficient Reward Shaping:

During RL, models learn based on reward signals. A pre-trained baseline ensured the model could already produce near-correct outputs, making it easier to fine-tune reward systems for specific behaviors.

Hybrid Model Design:

DeepSeek used the pre-trained model as a seed for reasoning and non-reasoning capabilities.
The early exposure to labeled tasks created robust foundations that RL could build upon, ultimately achieving state-of-the-art results.

Comparisons to Other Approaches

Aspect	DeepSeek Cold-Start Approach	Traditional RL Without SFT
Starting Point	Seeded with labeled data and basic reasoning capabilities.	Models begin with random behaviors.
Training Efficiency	Faster convergence during RL due to stronger baseline.	Longer training time due to instability.
Output Quality	Coherent, logical, and human-aligned from early training.	Erratic and incomplete outputs during early stages.
Stability	Reduced issues like repetition, mixing, and incoherence.	More prone to errors and instability early on.

Conclusion:

DeepSeek’s strategy of introducing labeled data or training steps before RL was highly effective because it:

Created a strong baseline for reasoning and language tasks.
Reduced computational costs by stabilizing RL training.
Enhanced the model’s ability to generalize across tasks.

Model Distillation:

Distilling Reasoning Patterns: By distilling larger models’ reasoning patterns into smaller ones, DeepSeek significantly reduced model sizes while maintaining performance. For example:

Distilled models (e.g., DeepSeek-R1-Distill-Qwen-32B) showed state-of-the-art results, outperforming OpenAI-o1-mini.
Impact: This step reduced both computational and storage costs without sacrificing performance.

Efficient Hardware Utilization:

Their base models supported extremely long context lengths (e.g., 128K tokens), optimized for benchmarks requiring large-scale outputs.

They leveraged multi-stage parallelization techniques to improve efficiency further.

Open-Sourcing & Collaboration:

They open-sourced their distilled models and training pipeline, allowing broader collaboration and rapid iteration within the research community, benefiting from external optimizations.

So why is everyone freaking out?

Excerpt from Ben Thompson, Stratechery

I think there are multiple factors. First, there is the shock that China has caught up to the leading U.S. labs, despite the widespread assumption that China isn't as good at software as the U.S.. This is probably the biggest thing I missed in my surprise over the reaction. The reality is that China has an extremely proficient software industry generally, and a very good track record in AI model building specifically.

Second is the low training cost for V3, and DeepSeek's low inference costs. This part was a big surprise for me as well, to be sure, but the numbers are plausible. This, by extension, probably has everyone nervous about Nvidia, which obviously has a big impact on the market.

Third is the fact that DeepSeek pulled this off despite the chip ban. Again, though, while there are big loopholes in the chip ban, it seems likely to me that DeepSeek accomplished this with legal chips.

Source

…

This is one of the most powerful affirmations yet of The Bitter Lesson: you don’t need to teach the AI how to reason, you can just give it enough compute and data and it will teach itself!

What is Distillation?

Distillation in the context of DeepSeek refers to a process where a large model’s knowledge is compressed into a smaller, more efficient model, without losing too much of its performance. DeepSeek used this approach to make smaller, faster versions of their powerful models while maintaining high accuracy. Here’s how it works and why it’s significant:

What is Model Distillation?

The Big Model (Teacher):

DeepSeek starts with a large, highly trained model like DeepSeek-R1. This model is powerful but requires a lot of computational resources to run, making it less practical for everyday use.

The Small Model (Student):

The goal is to create a smaller, lighter version of the big model (e.g., DeepSeek-R1-Distill-Qwen-32B) that can run faster and use less memory while still performing well.

The Process:

The large model (teacher) generates predictions on various tasks (e.g., solving math problems, answering questions).

The small model (student) is trained to mimic these predictions by learning from the outputs of the teacher model, rather than directly from the labeled data.

This allows the student to “inherit” the reasoning abilities and insights of the teacher without needing the same massive training data.

Why Distillation is Important for DeepSeek:

Efficiency:

The smaller models are much faster and cheaper to run. They require less memory, making them more accessible for use in real-world applications, such as mobile devices or resource-limited environments.

Scalability:

By offering smaller versions, DeepSeek can deploy their technology to a broader audience, including researchers, businesses, and developers who might not have access to supercomputers.

Knowledge Transfer:

Distillation ensures that the powerful reasoning skills of the big model (developed through Reinforcement Learning) are passed down to smaller models without requiring those smaller models to go through the same expensive training process.

Practical Usability:

Smaller models are easier to integrate into applications like chatbots, coding assistants, or search engines. For example:

A smaller model might run smoothly on your laptop or phone, while the original large model would require expensive cloud infrastructure.

How DeepSeek Distillation Compares to Traditional Transformers:

Feature	Traditional Transformer Models	DeepSeek Distillation
Purpose	Focus on building large, standalone models for each task.	Compressing large models into smaller, general-purpose ones.
Resource Usage	Requires significant computational power for inference.	Smaller models use less power and run faster.
Knowledge Transfer	Rarely focuses on systematically teaching smaller models.	Explicitly distills reasoning skills and task knowledge.
Performance Tradeoff	Smaller models typically lose significant accuracy.	DeepSeek’s smaller models maintain competitive accuracy.

Example: Why Distillation Matters

Let’s say you have a teacher (the big DeepSeek model) who’s amazing at explaining math, languages, and science. This teacher creates cheat sheets (distilled models) for students, so they can learn everything faster without having to study as much as the teacher did. Even though the students are smaller, they’re still incredibly smart and can handle most problems almost as well as the teacher.

What makes DeepSeek V3 special?

Source

DeepSeek-V3 vs. Models Like GPT

1. Faster Thinking with DualPipe

DeepSeek-V3 uses a unique scheduling system called DualPipe, which improves how it processes data. Traditional models like GPT process tasks sequentially, like a factory where one step has to finish before the next begins. DualPipe overlaps steps in the process—like starting the next step before the first one is fully done. This reduces “idle time,” making the model faster and more efficient at handling large workloads.

2. More Efficient Memory Use (FP8 Training)

DeepSeek-V3 employs FP8 (8-bit floating point) training, which significantly reduces the memory and energy needed for its calculations. This is like compressing information without losing its accuracy. In contrast, GPT uses higher precision formats like FP16, which require more resources. This innovation allows DeepSeek-V3 to perform complex tasks at a fraction of the computational cost.

3. Handling Long Contexts

DeepSeek-V3 can work with much longer inputs—up to 128K tokens—compared to GPT, which struggles with much shorter contexts (typically 32K). Imagine trying to write a summary of a novel versus a short story. DeepSeek-V3 keeps track of all the important details in the novel, whereas GPT might forget parts of the story.

4. Modular Expertise

Instead of treating all tasks the same way, DeepSeek-V3 uses Mixture of Experts (MoE) layers. This means it activates specific “experts” within the model based on the problem, like consulting a mathematician for a math problem or a coder for programming. GPT doesn’t have this modular design, so it has to rely on the same general knowledge for everything, which can limit its efficiency and performance on specialized tasks.

5. Predicting More at Once

While GPT predicts one word at a time, DeepSeek-V3 predicts two tokens at once using a technique called Multi-Token Prediction (MTP). This speeds up how quickly it generates text and improves its efficiency without sacrificing accuracy.

6. Smarter Feedback Loop

DeepSeek-V3 uses advanced reinforcement learning techniques like self-rewarding to improve itself. For example, when solving a math problem, it doesn’t just rely on a human to tell it if the answer is correct; it can evaluate its own solution using internal checks. GPT typically requires human feedback to refine its performance.

7. Optimized for Future Hardware

DeepSeek-V3 is designed to fully utilize next-gen hardware like NVIDIA’s H800 GPUs. It also introduces improvements in communication between GPUs, making it more scalable and efficient for larger systems. This forward-thinking design ensures that it can continue to perform well as technology advances.

8. Superior Performance Across Benchmarks

DeepSeek-V3 consistently outperforms GPT and other models on a range of tests:

Math and Coding: It achieves significantly higher accuracy on complex math and programming tasks.

Long-Context Understanding: It excels at reading and reasoning through long documents or multi-step problems.

Language Diversity: Its training data includes more languages and specialized tasks, giving it a broader range of knowledge.

Key Differentiators from GPT

Feature	DeepSeek-V3	GPT (e.g., GPT-3/4)
Architecture	Mixture-of-Experts (MoE)	Dense Transformer
Efficiency	DualPipe, FP8 training, fine-grained quantization	BF16 or FP32, standard parallelism
Inference Speed	Multi-Token Prediction (MTP), speculative decoding	Next-token prediction
Long-Context	Handles 128K tokens via YaRN	Typically limited to 32K
Data Optimization	FIM strategy, diverse multilingual corpus	Next-token prediction corpus
Training Costs	Optimized for lower GPU hours	Higher computational costs
Math and Code	Outperforms on GSM8K, HumanEval	Competitive but underperforms
Alignment	Self-rewarding, constitutional AI	PPO with human feedback

Other stuff I’ve written lately:

DeepSeek MIT License vs. Attention is All You Need

How did DeepSeek train their model on a fraction of the budget and time?

Comparison to the Transformer (Attention Is All You Need)

Highlights of DeepSeek’s Innovations:

Key Steps Taken by DeepSeek

So why is everyone freaking out?

What makes DeepSeek V3 special?

Tell me whatyou’re working on.

Tell me what
you’re working on.