# What Is Model Training and How Does It Actually Work?

> The core stages of model training, from pre-training on trillions of tokens to alignment techniques like RLHF and DPO for instruction-following.

Source: https://camnangai.com/en/engineering/what-is-model-training  
Published: 2026-08-23

**Model training** is the process of optimizing a transformer decoder network to predict the next
  token in a sequence using a massive corpus of data.

What you are really running is a stochastic computation graph whose only objective is to minimize a
loss function, so that the model's internal parameters are refined until the system generates
statistically plausible continuations of any input string you hand it. That single loop is what
carries raw compute all the way to a deployed assistant, through an orchestrated sequence of
pre-training, supervised fine-tuning, and preference alignment.

As an engineer, you have to read this not as a single event but as a series of hardware-intensive
stages, because each one demands its own architectural decisions, from managing high-performance
interconnects in multi-node clusters to tuning the token-to-parameter ratio. Strip the marketing
language away and what remains is mechanical, defined by data quality, scaling laws, and whatever
the underlying silicon will tolerate.

![A vast block of raw data compressed through layers of computation into an AI model that can answer questions](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/01-infographic-en.webp)

## What is model training?

Model training begins with a transformer decoder network, the standard architecture for modern
[large language models](https://camnangai.com/en/knowledge/what-is-an-llm). The process starts with
[tokenization](https://camnangai.com/en/knowledge/what-are-ai-tokens) and embedding mapping, where raw text is decomposed
into discrete units and converted into fixed-length vectors. Those embeddings then travel through a
series of transformer layers, each one a self-attention mechanism sitting alongside a parallel fully
connected network. In a decoder-only setup the layers mix information across the sequence, so the
processed embeddings that come out represent the contextual meaning of the input rather than a bag
of words.

![The training loop: predict the next token, compare it against the true one, measure the error, backpropagate and update the weights](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/02-en.webp)

The prediction step is narrower than most people expect, because the network looks only at the
output embedding of the last token in the partial sequence. That embedding is projected through a
linear transformation and a softmax to produce a probability distribution over the entire
vocabulary, and by sampling from it the model selects the next token and appends it to the string,
repeating the process autoregressively. This is the whole trick behind a model extending a prompt
into a coherent multi-token answer: it feeds its own output back in, and it never plans further
ahead than one token.

Underneath sits a single mathematical objective, which is to minimize the negative log-likelihood
loss function. Every partial sequence in the corpus passes through the model, the predicted
distribution is compared against the ground-truth token, and the gap between them drives a parameter
update through backpropagation. Nobody hands the model a grammar or an arithmetic table, so its
weights absorb the statistical regularities and reasoning patterns of the training data only because
predicting the next token well turns out to require them.

## Where does training data come from, and how is it cleaned?

High-scale training needs massive text corpora, which you typically aggregate from web-scraped
sources like Common Crawl, from specialized academic collections such as arXiv and PubMed, and
increasingly from synthetic data generation (SDG). Modern SDG uses a Generate-Critique-Filter
pipeline, where a teacher model generates raw samples, a reward model critiques them for attributes
like helpfulness or correctness, and a final filter keeps only the high-quality outputs for the
training set.

![A five-station data cleaning funnel running from raw web data down to the clean corpus used for training](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/03-en.webp)

Everything after that is unglamorous cleanup, and it decides how good the model gets. Unicode fixing
repairs garbled sequences, while language identification isolates the target monolingual data.
Heuristic filtering then prunes the set with rule-based metrics, including word count filters,
boilerplate string removal, and n-gram repetition filters that catch artificially generated or
low-quality content. Safety is its own stage, because you have to deploy a model like the AEGIS
Safety Model to sort content into critical risk categories and keep the training set compliant.

Deduplication is where the engineering gets hard, and it comes in exact, fuzzy, and semantic
flavors. Exact deduplication is a hash lookup, whereas fuzzy deduplication uses MinHash and
Locality-Sensitive Hashing (LSH) to find near-duplicates. The step people underestimate is what
follows, because you then have to turn the resulting similarity matrix into a graph and pull out its
connected components, or you delete part of a duplicate cluster and keep the rest. Semantic
deduplication goes further again, using embedding models and k-means clustering to strip out
documents that say the same thing in different words.

The last gate is compliance, which means PII redaction and task decontamination. Personally
Identifiable Information is substituted or eliminated to protect privacy, while task decontamination
runs a systematic n-gram search for test-set leakage from benchmarks like MMLU or GSM8K. Skip it and
the model memorizes specific evaluation answers, which produces exactly the misleading performance
numbers you will later have to explain.

## Pre-training: what does a model learn from trillions of tokens?

Pre-training is where broad syntax, general world knowledge, and reasoning structures come from. By
consuming trillions of tokens the model learns the likelihood of token sequences, so that it picks
up, for instance, the relationship between "moon" and "astronauts." What you want out of this stage
is a foundational base model that understands how language works, rather than a specialized
assistant.

Scaling laws are what keep that stage compute-efficient. The Chinchilla findings establish that
model size and training tokens should scale equally, targeting roughly 20 tokens per parameter, so a
70B model should be trained on 1.4T tokens to be compute-optimal. However, modern open-weight models
are frequently trained well past that point of optimality, because engineers push the data-intensive
ratio higher to squeeze more performance out of smaller architectures and make them more capable for
their size once deployed.

![The Chinchilla balance: parameter count and training tokens scaling in step for compute-optimal training](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/04-en.webp)

The reason a training pass is affordable at all is masked self-attention. Causal masking stops each
position from looking ahead at ground-truth tokens, and that single architectural constraint lets
you train all T tokens in a sequence of length T simultaneously. Every output embedding in the
transformer predicts the subsequent token, so one forward pass yields as many loss terms as there
are tokens, while the autoregressive integrity that generation depends on is left intact.

## How does supervised fine-tuning turn a raw model into an assistant?

Supervised Fine-Tuning (SFT), or instruction tuning, adapts a pre-trained model to follow user
intent. Unlike the unstructured text used in pre-training, SFT relies on high-quality labeled
prompt-response pairs, which teach the model to adopt a specific persona, follow formatting
constraints such as structured JSON, or handle multi-turn dialogue. Pre-training is about learning
language, whereas SFT is a behavioral shift, because what the model is learning here is how to act
as a conversational agent.

![How LoRA works: the base weight block stays frozen while only two small low-rank matrices beside it are trained](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/05-en.webp)

Because fine-tuning a model with billions of parameters would otherwise blow through your memory
budget, you reach for Parameter-Efficient Fine-Tuning (PEFT). LoRA (Low-Rank Adaptation) is the
standard approach, since it freezes the original pre-trained weights and injects small, trainable
low-rank matrices into the transformer layers, which cuts both the number of trainable parameters
and the GPU memory requirement without significant performance loss. QLoRA pushes further by
quantizing the base model to 4-bit precision, and that is what lets you fine-tune a massive model on
a single high-end GPU rather than a cluster.

## How do RLHF, DPO and GRPO teach a model how to behave?

Reinforcement Learning from Human Feedback (RLHF) aims a model at preferences nobody can write down
as a rule. The pipeline runs in three steps: initial SFT, then a reward model (RM) trained on human
rankings through a Bradley-Terry scalar reward model, then policy optimization with Proximal Policy
Optimization (PPO). The Kullback-Leibler (KL) penalty is not decoration here, because it keeps the
RL policy from drifting too far from the base SFT model, and without it the policy discovers that
fluent-sounding gibberish scores wonderfully with the reward model.

![A comparison of three alignment methods: RLHF with a reward model and PPO, DPO without a reward model, and GRPO comparing a group of answers](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/06-en.webp)

Direct Preference Optimization (DPO) is the simplified alternative, and it bypasses the reward model
entirely. Instead it uses a loss function based on the log ratio of the model's probabilities for
preferred versus dispreferred responses, so by treating the policy itself as the reward model, DPO
ends up more stable and more computationally efficient than the PPO-based RLHF pipeline.

For complex problem-solving, Group Relative Policy Optimization (GRPO) is increasingly the
interesting one. Rather than evaluating responses against an absolute scalar, GRPO generates
multiple candidate responses for the same prompt and compares them within a group, so the model
learns from the relative quality of these reasoning paths. That group-based relative scoring
reinforces the better logical steps, which is why it improves consistency on mathematics and
multi-step reasoning.

## How do you know a trained model is any good?

Evaluation runs on standardized benchmarks, each with a job. MMLU measures general knowledge, while
mathematical reasoning is tested through GSM8K and MATH, and coding performance is evaluated on
HumanEval and MBPP. You should also look at ARC for reasoning and at Humanity's Last Exam (HLE),
which currently shows low accuracy for even frontier models, and that gap is the honest reading of
the distance between these systems and expert human performance.

Metrics vary by task, so multiple-choice tests use raw accuracy while coding tasks use the pass@k
metric, defined as the probability that at least one of k generated samples passes functional unit
tests. Open-ended conversation usually falls to "LLM-as-a-judge", where a frontier model grades a
smaller model's output. I would stay skeptical of every one of these scores, because data
contamination lets benchmark questions leak into the training corpus, which converts a memorization
score into a reasoning score, and the leaderboard will not tell you which one you are reading.

![Data contamination: benchmark questions leaking into the training set, so a high score reflects memory rather than reasoning](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/07-en.webp)

## What does it actually cost to train a model?

Compute requirements for frontier models are growing at 4-5x per year, which makes large-scale
pre-training an industrial operation rather than a software project. GPT-4 is estimated at 2e25
FLOP, and Gemini Ultra at 5e25 FLOP, so scaling to those levels takes massive infrastructure
investment and specialized data engineering before a single training step runs.

Once your compute requirements exceed the 1e22 FLOP threshold, you move from simple GPU clusters to
dedicated hardware environments, because those clusters need high-performance interconnects and
multinode multi-GPU scaling simply to hold throughput. At that scale you are also trading precision
against memory, since 4-bit or 8-bit quantization is often what makes a model viable for real-world
inference and deployment at all.

## Should you train your own model?

For most engineering teams, pre-training a frontier model from scratch is off the table, because the
compute costs and the data engineering overhead are not the kind of thing you grow into. The return
sits in fine-tuning an existing open-weight model instead, and with PEFT techniques like LoRA that
means one GPU, a modest set of labeled data, and a base model somebody else paid to train.

![Three options for a company, cheapest to most expensive: RAG, fine-tuning with LoRA, and training a model from scratch](https://camnangai.com/articles/huan-luyen-mo-hinh-ai-la-gi/08-en.webp)

If you do fine-tune, prioritize data quality over quantity, because the current landscape favors
smaller, highly specialized models adapted through SFT and DPO over massive general-purpose models
for specific enterprise tasks. The decision you will actually spend your time on is the
hardware-centric one, between the VRAM capacity your training run needs and the quantization
precision your production inference can live with.

## References

- [Training and fine-tuning large language models — RBC Borealis](https://rbcborealis.com/research-blogs/training-and-fine-tuning-large-language-models/)
- [Mastering LLM Techniques: Text Data Processing — NVIDIA Technical Blog](https://developer.nvidia.com/blog/mastering-llm-techniques-data-preprocessing/)
- [Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)
- [Illustrating Reinforcement Learning from Human Feedback (RLHF) — Hugging Face](https://huggingface.co/blog/rlhf)
- [Supervised Fine-Tuning — Hugging Face LLM Course](https://huggingface.co/learn/llm-course/chapter11/1)
- [A Technical Deep Dive into the Essential Stages of Modern Large Language Model Training, Alignment, and Deployment — MarkTechPost](https://www.marktechpost.com/2026/04/15/a-technical-deep-dive-into-the-essential-stages-of-modern-large-language-model-training-alignment-and-deployment/)
- [Training compute of frontier AI models grows by 4-5x per year — Epoch AI](https://epoch.ai/publications/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year)
- [30 LLM evaluation benchmarks and how they work — Evidently AI](https://www.evidentlyai.com/llm-guide/llm-benchmarks)
- [Training open-weight models is becoming more data intensive — Epoch AI](https://epoch.ai/data-insights/training-tokens-per-parameter)