How SEA-LION turns a general-purpose language model into a more capable, more culturally grounded open model for the region.
The authors start with Llama 3.1 8B-Instruct and Gemma 2 9B, continue training them on a carefully mixed corpus, then add multilingual instruction tuning, preference alignment and model merging.
English, Chinese, Indonesian, Vietnamese, Malay, Thai, Burmese, Lao, Filipino, Tamil and Khmer.
A deliberate mix of SEA-language, English and code data, rather than SEA-only training.
Llama-SEA-LION-8B-IT and Gemma-SEA-LION-9B-IT, released under the MIT licence.
English-centric models can look strong on general benchmarks while still underperforming on SEA-specific language, knowledge and cultural understanding.
The reported models lead their comparison set on the paper’s SEA-focused benchmarks, especially Gemma-SEA-LION-9B-IT.
The authors release models and training artefacts to support reproducibility and future SEA-specific applications.
Customer support, product help and buying journeys that can handle local languages, code-switching and regional context.
Translation, explanation and access layers for communities that are poorly served by English-first interfaces.
The two recommendation cards are plausible applications, not experiments reported in this paper.
The interesting part isn’t a single dataset. It’s the training system: data mix, sequencing, model lineage, alignment and evaluation all matter.
We’ll unpack what changed at each stage and what the benchmark results can, and can’t, tell us.
There are two broad moves: teach the base model more SEA-language knowledge, then rebuild useful instruction-following behaviour on top.
The paper reports a 200-billion-token continued-pretraining budget. Small-scale experiments led to this starting mix:
Open corpora include Dolma, FineWeb, The Stack v2, SEA-LION-Pile, CommonCrawl and public-domain sources. SEA documents are filtered with a fastText language classifier and cleaned with Trafilatura.
The team ran small-scale CPT experiments with a 10B-token budget and varied English, code and SEA proportions before selecting the 55/25/20 mix.
About 9.5M mostly English instruction pairs for reasoning, maths and code.
7,298,828 samples across 11 languages, including synthetic SEA-language instruction data.
SEA preference data is used with SimPO to encourage helpful, relevant and informative responses.
The paper describes a tension: SEA-focused tuning can improve local performance but may erode English capability or instruction following. The pipeline keeps returning to the base lineage through model merges.
Interpretation: the model is treated as a portfolio of capabilities that needs balancing, not as a single score to maximise.
Holistic SEA benchmark used here across NLU, NLG, NLR, NLI, instruction following and multi-turn chat. The reported tables cover four languages: Indonesian, Vietnamese, Thai and Tamil.
Zero-shot NLU evaluation across 131 data subsets, 7 tasks and 31 SEA indigenous languages.
English-facing checks across IFEval, BBH, MATH, GPQA, MuSR and MMLU-PRO.
The authors could not reproduce the SEACrowd NLG benchmark because of maintenance issues, so their SEACrowd experiment is NLU-only. That matters when interpreting “multilingual quality” as a whole.
Use the toggle to switch between the two instruct model families. Values are the SEA-HELM average from Table 3.
Gemma-SEA-LION-9B-IT: +14.02 points against Gemma-2-9B-IT on the reported SEA-HELM average.
Llama-SEA-LION-8B-IT: +20.36 points against Llama-3.1-8B-Instruct on the reported SEA-HELM average.
On the paper’s Open LLM Leaderboard comparison, the final SEA-LION instruct models remained competitive in English, maths and reasoning tasks.
The authors report gains in English instruction following and SEA-HELM after the first instruction-tuning stage.
The second stage produces a further increase in SEA-HELM over Stage 1 in both model families.
Merging brings in strengths from related instruct models and is used to mitigate catastrophic forgetting from fine-tuning.
The paper reports broader SEA MTBench gains, alongside minor degradation on some instruction-following and task-specific scores.
Model merging is not a magic final polish. In this paper, it is an empirically selected balancing operation whose value depends on the model lineage, task mix and desired trade-offs.
A practical adaptation of the paper’s logic, sized for a real product team or research prototype.
Define the target, build the data, continue pre-training, post-train, evaluate, then deploy with guardrails.
Pick 1-3 languages, one user segment and 3-5 tasks. For example: Indonesian + Malay customer support, product troubleshooting and summarisation.
Create native-speaker prompts with expected answers, cultural edge cases, code-switching and refusal cases. Keep a private test split.
target = {
"languages": ["id", "ms"],
"tasks": ["support", "summarise", "classify"],
"must_not_regress": ["English safety", "citation fidelity"],
"golden_set": "versioned, native-speaker reviewed"
}Prioritise consented, licensed or public-domain text. Record source, date, language, domain and usage rights.
Run language identification, deduplication, quality checks, PII screening and document-level contamination checks.
Use the paper’s 55/25/20 ratio as a hypothesis, then tune the mix against your golden set rather than copying it blindly.
Ship a dataset card, a manifest and a reproducible sampling recipe with every training run.
Start from an open instruct model and expose it to the regional corpus. Track language loss and general capability loss separately.
Use general instruction data first, then multilingual and task-specific data. Keep the evaluation set untouched.
Only add preference optimisation or model merging when the evals show a specific behavioural problem to fix.
For a one-GPU proof of concept, QLoRA/LoRA can reduce compute and memory. That is a practical adaptation, not the SEA-LION paper’s reported full-model fine-tuning recipe.
Native-speaker review, task success, cultural fit, fluency and code-switching.
Instruction following, reasoning, maths, code and English regression checks.
Hallucination, refusal, privacy, harmful content, prompt injection and sensitive-domain tests.
Ship only when regional performance improves without unacceptable regression on general capability, safety or latency.
Done condition: every score has a dataset version, sample count, language breakdown and human-review note.
Use language ID and task routing where a smaller specialist model is enough. Keep a stronger fallback for uncertainty.
Log model, dataset and prompt-template versions. Make user-visible claims traceable to the right source.
Track quality by language, dialect, task, latency and refusal behaviour. Refresh the golden set as real usage exposes gaps.
SEA-HELM covers four languages in the reported tables. SEACrowd covers more languages but only with classical NLU datasets in this study.
The authors call for a broader SEA benchmark that includes LLM-specific tasks across all covered languages.
The authors argue the technique should generalise to other sizes, but this paper doesn’t establish that empirically.
A stronger ablation programme would isolate language mix, synthetic data, alignment and each merge decision across languages and tasks.
Better regional performance comes from data, training order, evaluation and capability balancing together.
The 55/25/20 mix and repeated merges are attempts to improve SEA capability without throwing away useful general behaviour.
If the benchmark misses dialect, culture, task type or language, the model can look better than it is for real users.
SEA-LION: Southeast Asian Languages in One Network
Raymond Ng et al. · arXiv:2504.05747v4 · 30 October 2025
sea-lion.ai
Project landing page and model collection referenced by the paper.
Hugging Face · AI Singapore
Training data, checkpoints and model releases referenced in the paper.
Use ← → or the buttons to navigate. Press Home/End to jump. On touch screens, swipe left or right. Expand details for caveats and implementation notes.
This deck explains the paper. It doesn’t reproduce training, benchmark scores or production safety validation.