The DeepSeek Series: A Technical Overview

Motivation & Overview

The authors set out to answer an important question: Given a fixed
compute budget for pre-training, how do we choose the scale of the model
and how much training data to use? Prior studies (e.g. Chinchilla vs.
GPT-3) differed on the ratio between these two factors. DeepSeek-LLM
addresses that by measuring scale in a different way. Earlier work
measured scale in terms of how many parameters were in the model,
DeepSeek-LLM instead measured scale as non-embedding FLOPs/token They then found they could predict
computation with:

$$
C = M \times D
$$

where $C$ is the compute budget, $M$ is non-embedding
FLOPs/token, and $D$ is data size.

This more granular representation helps them predict how a 7B or 67B model
might train on 2T tokens of bilingual data.

Training Instability

A central concern they grapple with is training instability (sudden irrecoverable
divergences in the training process), which can often manifest in
large-scale language models—especially those with mixture-of-experts
or very long contexts.

By carefully tuning learning rates, batch sizes, and other
hyperparameters , DeepSeek-LLM
demonstrates that stable large-scale training is achievable, but it
requires meticulous design of the architecture of the transformer model
together with the infrastructure of the High Performance Computing (HPC)
data center used to train it. This interwoven design of both architecture and
infrastructure is called HPC Co-Design.

Data Quality & Model Scale

A point the authors make is about how data quality shifts the optimal
ratio—i.e., higher-quality data can justify a bigger model for the same
number of tokens. You can intuit this by imagining two scenarios:

• Scenario A: You have a 100-billion-token corpus full of duplicates,
  spammy text, or incomplete sentences. The model might not glean much new
  knowledge because the data is partly redundant or low-value.
• Scenario B: You have a carefully curated 100-billion-token corpus with
  broad coverage of code, math, multi-lingual dialogues, factual text, etc. Each
  token is more “information-rich,” so the model can “afford” to use more
  parameters without hitting diminishing returns prematurely.

In other words, when data is denser in useful information, scaling the
model further pays off because each parameter can learn from richer
signals.

Key Takeaways

• Hyperparameter Scaling: They propose simple power-law fits to pick batch
  size and learning rate as compute $C$ grows.
• Bilingual Data: They train two base sizes (7B, 67B) on 2T tokens
  covering English/Chinese, then do Supervised Fine Tuning (SFT) and a simpler
  preference-based alignment called Direct Preference Optimization (DPO).
• Results: The resulting DeepSeek-LLM67B “Outperforms LLaMA-2 70B” on
  math/coding tasks, illustrating how HPC co-designed approaches can keep training
  stable while efficiently pushing scale.

The seeds planted here – scaling laws and infrastructure for extremely large
training – will reappear in subsequent works.

Expanding the Model While Reducing Memory

Where DeepSeek-LLM mostly explored high-level scale tradeoffs,
DeepSeek-V2 dives into specifics of Transformer architecture overhead. Two
big obstacles in large LLMs are:

1. Attention KV Cache: Storing Key/Value vectors for thousands of tokens
   is memory-intensive.
2. Feed-Forward Computation: Typically the largest consumption of FLOPs
   in a Transformer.

To tame both, they propose:

1. Multi-Head Latent Attention (MLA): compresses Key/Value vectors to
   reduce memory.
2. DeepSeekMoE: a sparse Mixture-of-Experts approach that
   activates a fraction of the feed-forward capacity per token.

Multi-Head Latent Attention (MLA)

Attention is the process by which the model decides
which tokens in the input stream to pay attention to when it’s trying to
predict the next token. In standard attention, each token’s Q/K/V
can be as large as $d_{model}$ times the
number of heads. MLA folds them into smaller “latent”
vectors:

$$
\quad
\mathbf{c}_{t}^{KV} = W^{DKV}\mathbf{h}_t,
\quad
\mathbf{k}_{t}^{C} = W^{UK}\mathbf{c}_t^{KV},
\quad
\mathbf{v}_{t}^{C} = W^{UV}\mathbf{c}_t^{KV},
\quad
$$

Where $c_{t}^{KV}$ is the compressed latent vector for keys and values.
$W^{DKV}$ is the down-projection matrix, and $W^{UK}, W^{UV}$ are the
up-projection matrices for keys and values, respectively. In simpler terms:

1. Replaces the standard QKV computation by using low rank factorization to
   turn one matrix of dim (in, out) into two matrices of (in, rank) and
   (rank, out)
2. Project the compressed KV latent vector for each head to get the full K
   and V head corresponding to each Q head
3. Cache the compressed KV latent vector instead of each of the KV heads in
   full, and compute the KV heads on the fly from the latent vector.

DeepSeekMoE: Sparsely Activated FFNs

Next, they adopt a Mixture-of-Experts (MoE) in the feed-forward
blocks. Mixture-of-Experts (MoE) is a design technique that logically
splits the model into separate areas (experts) each having specialized parameters
for different domains of knowledge. Because each token is only routed to the experts
most relevant to it, MoE can drastically reduce the compute required compared to a fully
dense approach. This approach ties directly into the HPC co-design arc, as each expert
can reside on different GPU devices. By limiting cross-device communication
(e.g., device-limited routing), MoE effectively scales to extremely large parameter
counts without incurring prohibitive memory or data-transfer costs.

DeepSeek uses more fine-grained experts than previous models, dividing them into
two kinds:

• Shared Experts handle universal patterns for every token.
• Routed Experts handle specialized sub-problems, chosen dynamically via
  gating.

During training, they consider Auxiliary Loss to ensure
balanced usage so no expert collapses (i.e. is never used).

They further limit cross-device routing with
a “device-limited routing” scheme – instead of allowing any token to access any expert,
DeepSeekMoE selects a limited number of devices ($M$) per token, and performs expert
selection only within these devices. The basic process is as
follows:

• Identify top $M$ devices that contain experts with the highest affinity
  to the token
• Perform top $K_r$ expert selection within these $M$ devices
• Assign the selected experts to process the token

Without device-limited routing, MoE models can generate excessive
communication overhead which is incompatible with the hardware limitations
imposed on the DeepSeek team. In addition, MoE models typically risk uneven
expert utilization, where some experts are overused while others remain
inactive. To prevent this, DeepSeekMoE introduces three balancing loss
functions:

• Expert-level Balance Loss ($L_{ExpBal}$):
• Ensures uniform distribution of tokens across experts to prevent expert
  collapse
• Uses a loss function based on softmax scores of token-expert
  affinity
• Device-level Balance Loss ($L_{DevBal}$):
• Ensures workload is evenly distributed across devices
• Communication Balance Loss ($L_{CommBal}$):
• Balances incoming and outgoing token routing to each device

Training & Outcomes

DeepSeek-V2, with ~236B total params (21B activated), is pre-trained on
8.1T tokens. They do Supervised Fine Tuning (SFT) on 1.5M instruction samples,
then reinforcement learning (RL) for alignment. The end result:

• Inference and training are both faster and cheaper (MLA + sparse
  experts)
• They remain stable at scale

This paper is really when iteration gains due to HPC Co-Design start to
become apparent. By designing the model architecture with the training
infrastructure in mind, and implementing a training regime that considers the
realities of the hardware (e.g. low interconnect speeds on H800s), the team
was able to lay the foundation for their most notable breakthrough.

Scaling MoE to 671B While Preserving Efficiency

Building on V2, DeepSeek-V3 further extends sparse models to 671B
parameters (37B activated), training on 14.8T tokens in under 2.8M H800
GPU hours. The authors credit extensive HPC co-design:

Lastly, we emphasize again the economical training costs of
DeepSeek-V3, summarized in Table 1, achieved through our optimized
co-design of algorithms, frameworks, and hardware.

— DeepSeek-V3 Tech. Report, p.5

The major novelties are:

1. Refined MLA
2. Refined DeepSeekMoE
3. Co-Designed Training & Inference
   Frameworks

Refined MLA

Multi-Head Latent Attention was introduced in V2 to reduce KV cache overhead.
In V3, it is further refined with several new features:

• Improved RoPE Handling: V2 only partially decoupled keys, but V3 extends
  the concept for more stable 128K context. They track a “decoupled shared key”
  that reduces numerical drift in extremely long generations.
• Joint KV Storage: V2 stored compressed keys and values separately. V3
  merges them into a shared compressed representation to further reduce memory
  traffic during multi-node inference.
• Layer-Wise Adaptive Cache: Instead of caching all past tokens for all
  layers, V3 prunes older KV entries at deeper layers. This helps keep memory
  usage in check when dealing with 128K context windows.

Together, these MLA refinements ensure that while DeepSeek-V3 can attend across very long sequences,
the memory overhead remains manageable. While many popular LLMs cap out around 4K to 32K tokens, 128K pushes
that envelope significantly, enabling the model to process or reference an entire large document in a single pass.
This bigger context window puts added pressure on GPU memory—hence the importance of refining MLA
to keep overhead in check.

Refined DeepSeekMoE: Auxiliary-Loss-Free, Higher Capacity

On the MoE side, DeepSeek-V3 drops the auxiliary-loss approach from V2.
Instead of an explicit penalty term, each expert acquires a dynamic bias
$b_i$. If an expert is overloaded at a step, $b_i$ decreases; if
underloaded, $b_i$ increases. The gating decision then adds $b_i$ to the
token’s affinity:

$$
s’_{i,t} = s_{i,t} + b_i
$$

Key Improvements:

• No Token Dropping: V2 occasionally dropped tokens if certain experts got
  overloaded, but the new bias-based method keeps everything.
• More Activated Experts: They raise the number of routed experts from 6
  to 8 per token, improving representational power.
• Higher Stability: By removing auxiliary losses, they avoid potential
  interference with the main training objective, focusing purely on the
  intrinsic gating signals plus bias adjustments.

Hence, the final feed-forward module is a combination of a small set of
shared experts plus up to 8 specialized experts chosen adaptively.

Co-Designed Frameworks: FP8, DualPipe, and PTX Optimizations

Scaling an MoE model to 671B demanded HPC-level solutions for training and
inference. The authors emphasize:

Through the co-design of algorithms, frameworks, and hardware, we
overcome the communication bottleneck in cross-node MoE training, achieving
near-full computation- communication overlap.

— DeepSeek-V3 Tech. Report, p.5

FP8 Mixed Precision

They adopt an FP8 data format for General Matrix Multiplications (GEMMs),
halving memory. The risk is reduced numeric range so they offset it
with:

• Block-wise scaling (e.g., 1×128 or 128×128 tiles).
• Periodic “promotion” to FP32 after short accumulation intervals to avoid
  overflow/underflow.

DualPipe Parallelism

They propose DualPipe to overlap forward/backward computation with the MoE
all-to-all dispatch. It rearranges pipeline stages to ensure that network
communication (particularly across InfiniBand) is hidden behind local matrix
multiplications.

PTX-Level & Warp Specialization

To fully exploit InifiniBand(IB) and NVLink:

• They tune warp-level instructions in PTX (a level lower than CUDA),
  auto-tuning the chunk size for all-to-all dispatch.
• They dynamically partition Streaming Microcontrollers into communication vs.
  compute tasks so that token dispatch never stalls local GEMM.

As a result, training costs were cut to 2.8M H800 GPU hours per run – low
for a 14.8T token corpus.

Outcomes

The resulting DeepSeek-V3 excels at code, math, and some multilingual
tasks, outperforming other open-source LLMs of similar scale. Deep HPC
co-design (FP8, DualPipe, PTX-level optimization) plus refined MLA/MoE
implementation achieve extreme scale with stable training.

Emergent Reasoning Behaviors Through RL-Only

All prior DeepSeek releases used Supervised Fine-Tuning (SFT), plus
occasional Reinforcement Learning (RL). By contrast,
DeepSeek-R1-Zero tries an extreme: no supervised warmup, just RL from
the base model. They adopt Group Relative Policy Optimization (GRPO),
which:

1. Samples a group of old-policy outputs ${o_1, …, o_G}$
2. Scores each with a reward (in this case, rule-based)
3. Normalizes the advantage $A_i$ by group mean/stdev
4. Optimizes a clipped PPO-like objective

The reward function for the R1 models is rule-based – a simple weighted
sum between 2 components

• Accuracy Reward – if the task has an objective correct answer (e.g. a
  math problem, coding task, etc.), correctness is verified using mathematical
  equation solvers for step-by-step proof checking, and code execution & test
  cases for code correctness verification
• Format Reward – the model is rewarded for following a structured
  reasoning process using explicit reasoning markers <think></think> and
  <answer></answer>

The relative advantage $A_i$ for a given output is calculated as:

$$
A_i = \frac{r_i – mean(\{r_1, r_2, …, r_G\})}{std(\{r_1, r_2, …, r_G\})}
$$

where $r_i$ is the reward calculated for the given output. The model’s
policy is updated to favor responses with higher rewards while constraining
changes using a clipping function which ensures that the new policy remains
close to the old.

In so many words: the authors created a testing/verification harness around
the model which they exercised using reinforcement learning, and gently guided
the model using simple Accuracy and Format rewards. In doing so, emergent
reasoning behaviors were observed:

• Self-verification where the model double-checks its own answers
• Extended chain-of-thought where the model learns to explain its
  reasoning more thoroughly
• Exploratory reasoning – the model tries different approaches before
  converging on an answer
• Reflection – the model starts questioning its own solutions and
  adjusting reasoning paths dynamically

R1-Zero is probably the most interesting outcome of the R1 paper for
researchers because it learned complex chain-of-thought patterns from raw reward
signals alone. However, the model exhibited notable issues:

• Readability Problems: Because it never saw any human-curated language
  style, its outputs were sometimes jumbled or mixed multiple languages.
• Instability in Non-Reasoning Tasks: Lacking SFT data for general
  conversation, R1-Zero would produce valid solutions for math or code but be
  awkward on simpler Q&A or safety prompts.
• Limited Domain: Rule-based rewards worked well for verifiable tasks
  (math/coding), but handling creative/writing tasks demanded broader
  coverage.

Hence, the authors concluded that while “pure RL” yields strong reasoning
in verifiable tasks, the model’s overall user-friendliness was lacking. This
led them to DeepSeek-R1: an alignment pipeline combining small cold-start
data, RL, rejection sampling, and more RL, to “fill in the gaps” from R1-Zero’s
deficits.

Refined Reasoning Through SFT + RL

DeepSeek-R1 addresses R1-Zero’s limitations by injecting a small amount
of supervised data before RL and weaving in additional alignment steps.