M MiniMind / English deep dive

Repository translated into English

Train a tiny LLM. Understand the whole machine.

A visual deep dive into jingyaogong/minimind: an end-to-end project for building, training, aligning and serving a small language model from scratch.

MiniMind-3 mainline64M dense / 198M-A64M MoEApache-2.0

01 / plain English

MiniMind is a guided tour of the LLM stack.

It keeps the model close enough to modern practice to be useful, but small enough to train, inspect and modify on personal hardware.

The point

Learn by building

Move beyond “call an API” or “attach a LoRA” and see what pretraining, SFT, RL and serving actually do.

The payoff

Own the loop

Change the model, data, tokenizer, reward or runtime and observe the consequences.

The catch

Small ≠ capable

This is an educational and experimental platform, not a frontier-model replacement.

Why it matters

The black box gets smaller when you can read the whole path.

The real value is not benchmark theatre. It is a concrete mental model of how a language model becomes a system.

Teaching

LLM fundamentals

Tokenisation, causal attention, training loops, fine-tuning and evaluation.

Prototyping

Small private models

Explore domain behaviour when a frontier API is too opaque, large or expensive.

Research

Controlled experiments

Change one ingredient at a time: architecture, context, reward or tools.

Product

Edge ideas

Use a compact sandbox for low-latency or on-device assistant concepts.

Section break

From tokens to tools: the learning map

MiniMind is best understood as a sequence of increasingly opinionated loops.

Repository anatomy

One repo, several layers of the stack.

model/

Architecture

Config, Transformer blocks, attention, RMSNorm, RoPE, SwiGLU, MoE and tokenizer assets.

trainer/

Learning loops

Pretraining, SFT, LoRA, DPO, PPO, GRPO, distillation and agentic training.

dataset/

Data interface

JSONL-oriented loaders and templates for text, conversations and tool-use examples.

scripts/

Product surface

Conversion, tool-call evaluation, an OpenAI-compatible server and Streamlit demo.

Translation note: trainer means the executable learning pipeline, not a high-level framework that hides the loop.

The central idea

Every stage changes what the model can do.

Start with next-token prediction. Then teach interaction, preference, tools and multi-step behaviour.

PretrainLearn language patterns from text.

SFTLearn roles, instructions and formatting.

PreferenceDPO or reward-driven RL shifts behaviour.

Tool useCall, observe, continue.

Agentic RLOptimise trajectories against task rewards.

Mental model

The model is repeatedly shaped by data, objectives and feedback.

Engineering lesson

Data format and chat templates are part of the model interface.

Product lesson

Capability is weights + prompt format + tools + runtime.

Model architecture

A modern decoder-only Transformer, kept intentionally legible.

Token IDs6,400-token vocabulary

Embeddings768-dimensional state

RMSNorm → attention → residual

RMSNorm → SwiGLU / MoE → residual

× 8 Transformer blocks

LM headTied word embeddings

Next tokenCausal language modelling

Dense

64M

8 layers, 768 hidden size, 8 query heads and 4 key/value heads.

MoE

198M total / 64M active

Four experts with top-1 routing. More capacity, plus routing and kernel overhead.

Inputs shape outputs

The “small model” trick is partly a data and vocabulary decision.

Tokenizer

MiniMind

6,400

Qwen2

151,643

A compact vocabulary reduces embedding and output-layer cost. The trade-off is less mature tokenisation across languages.

Training data

JSONL as the common language

Pretraining uses text → next token. SFT uses conversation templates, including roles, thinking markers and tool calls.

// pretrain_t2t_mini.jsonl {"text": "Transformer models learn patterns from context."} // SFT / tool-use shape {"conversations": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ]}

Important: max_seq_len means tokens, while dataset names may refer to character length. Padding and truncation are efficiency-versus-context decisions.

Mechanism

Small does not mean simplistic.

The model includes several design choices you will meet in larger systems.

Pre-NormRMSNormSwiGLUGrouped-query attentionRoPEKV cacheYaRN scaling

Grouped-query attention

8 query heads, 4 KV heads

More query heads than key/value heads can reduce the memory footprint of cached keys and values during generation.

RoPE + YaRN

Position is a controllable interface

The config advertises a 32,768-position limit and includes an inference-time YaRN option. Treat long-context gains as something to measure, not assume.

Inference

Cache the past, generate the next token

Past key/value states avoid recomputing the entire prefix on every step.

Beyond SFT

Feedback turns a language model into a behavioural system.

DPO

Preferred vs rejected

Preference pairs shift the policy towards the preferred answer without a separate reward-model training phase.

trainer/train_dpo.py

PPO / GRPO / CISPO

Score sampled behaviour

The policy generates candidates, receives a reward and updates towards higher-scoring outputs.

trainer/train_ppo.py · train_grpo.py

Agentic RL

Optimise trajectories

The agent can call tools over multiple turns, observe results and receive delayed task-level rewards.

trainer/train_agent.py

The loop

PromptTask + tools

Roll outResponse / call

ExecuteEnvironment returns

RewardScore trajectory

UpdateIncrease useful behaviour

Read the evidence properly

A brilliant learning substrate, with deliberate limits.

What the repo demonstrates

✓A complete path from token IDs to generation and serving.

✓Several post-training objectives implemented in native PyTorch.

✓Compatibility routes into Transformers, vLLM, Ollama and llama.cpp.

✓Evaluation hooks including C-Eval, CMMLU and English tasks.

What it does not prove

!A 64M model is a frontier-model replacement.

!Long-context support guarantees long-context quality.

!Reward gains cannot trade off against general capability.

!Reported cost and duration transfer to every setup.

Published benchmark snapshot

MiniMind-3 / C-Eval

24.89

MiniMind-3 / ARC

28.49

Reported in the repository README, not reproduced here. The project describes the comparison as rough because its data scale and training mix differ from larger models.

Section break

Build the smallest useful path first.

Inference → mini training → controlled extension. That order keeps feedback fast and failure modes legible.

Before you start

Choose your path by what you want to learn.

Path A / fastest

Run

Download a checkpoint. Use the CLI, Streamlit demo or API.

No trainingBest first hour

Path B / core

Train

Use the mini JSONL datasets to reproduce pretraining and SFT.

GPU recommendedBest for learning

Path C / research

Extend

Add LoRA, DPO, RL, tools, agentic rollouts or distillation.

Experiment designBest for builders

Reference environment

The README lists Python 3.10.16, CUDA 12.2, Ubuntu 20.04 and an RTX 3090-based reference setup. Your environment may differ.

First health check

import torch print(torch.cuda.is_available())

Implementation guide / 01

Get a response before you change the model.

Clone and install

Use the dependency file in a fresh environment.

Download a checkpoint

Use ModelScope or Hugging Face for a released model.

Run the CLI

Load it with eval_llm.py and test a few prompts.

Add a UI last

Launch Streamlit or the API after the baseline works.

# clone git clone --depth 1 https://github.com/jingyaogong/minimind cd minimind # install pip install -r requirements.txt # run a released checkpoint python eval_llm.py --load_from ./minimind-3 # optional WebUI cd scripts streamlit run web_demo.py

Keep one known-good checkpoint and command. Every experiment should return to that baseline.

Implementation guide / 02

Reproduce the learning loop in two stages.

Stage 1 / pretraining

Teach language patterns

Place pretrain_t2t_mini.jsonl in ./dataset, then train the base checkpoint.

cd trainer python train_pretrain.py # output: out/pretrain_*.pth

Stage 2 / SFT

Teach interaction

Add sft_t2t_mini.jsonl to teach roles, instruction following, thinking markers and basic tool-call patterns.

cd trainer python train_full_sft.py # output: out/full_sft_*.pth

Make it restartable

The repository documents checkpoint saving and resume support. For a long run, use --from_resume 1 and keep checkpoints with experiment logs.

Implementation guide / 03

Change one axis, then measure the delta.

Data axis

Private domain SFT

Swap in a small, licensed dataset with a held-out evaluation set.

Model axis

Dense vs MoE

Compare capacity, active parameters, throughput and quality. Don’t assume fewer active parameters means lower wall-clock cost.

Behaviour axis

Tool use or DPO

Start with a narrow task, explicit success criteria and an inspectable reward or preference signal.

Example experiment

1Hypothesis: explicit tool schemas improve valid-call rate.

2Change: add 200 licensed examples, no architecture change.

3Measure: valid JSON, correct tool, task success, latency.

4Inspect failed calls and reward shifts, not only aggregate score.

# test the resulting checkpoint python eval_llm.py --weight full_sft # broader benchmark route lm_eval --model hf \ --model_args pretrained=/path/to/model \ --tasks ceval-valid,arc_easy \ --apply_chat_template

Implementation guide / 04

Turn the checkpoint into a usable system surface.

Local demo

Streamlit chat

Inspect generations, thinking display, tool selection and multi-turn calls.

cd scripts streamlit run web_demo.py

Application integration

OpenAI-compatible API

Expose a familiar interface to a chat UI or application, with reasoning content and tool-call support.

python scripts/serve_openai_api.py

Runtime options

Use the ecosystem

Convert or serve through Transformers, vLLM, Ollama or llama.cpp where the format and versions support it.

Production caution

Wrap it with tests

Treat tool schemas, timeouts, refusal behaviour, logging and provenance as application code.

Before you share results

Validate the system, not just the loss curve.

Minimum viable eval

✓Can it follow the expected chat template?

✓Does it produce valid output on held-out prompts?

✓Does tool calling remain syntactically and semantically valid?

✓Does a change improve task success without breaking unrelated cases?

✓Can another person reproduce the environment and command?

Experiment record

model: minimind-3 / dense data: sft_t2t_mini + domain_delta change: tool-call examples seed: 42 hardware: [GPU, VRAM] metrics: [valid_call, success, latency] known_failures: [...]

This is the difference between a nice demo and an evidence-bearing learning project.

The short version

MiniMind makes the LLM pipeline small enough to hold in your head.

Start with inference, reproduce the mini training path, then run one controlled experiment. Treat the repo as a laboratory, not a magic box.

ReadableModifiableMeasurable

Sources / translated from

GitHub repository and English README ↗Model implementation: model/model_minimind.py ↗Training scripts: trainer/ ↗Serving and tool-use scripts: scripts/ ↗Apache-2.0 licence ↗

Claims and numbers are translated or synthesised from the repository as inspected on 19 September 2026. Timings, costs and benchmark figures are repository claims; this deck does not reproduce the runs.

End of deck

Build the small thing you can actually understand.

The next useful experiment is probably one dataset, one objective and one metric.

MiniMind deep diveEnglish translation + implementation guide