AI Research · Language Models · From Scratch
MyLLM (S1-LLM)
AI Research · Language Models · From Scratch
A large language model built entirely from first principles — own BPE tokenizer, own data pipeline, own decoder-only Transformer, own training loop and inference engine — trained from random initialization with no pretrained weights, no external LLM APIs and no borrowed vocabularies.
- 8
- Technologies
- 1
- Domain
- 6
- Capabilities
- 1
- Public links
Who it's for
Engineers, researchers and hiring teams who want proof of understanding LLMs at the implementation level — every component built, tested and verified rather than imported.
Where it fits
A research codebase and training ground today; architected to carry the same pipeline from a 1M-parameter correctness gate up a scaling ladder as data and hardware allow.
The hard part
The problem
Using an LLM teaches you nothing about how one works. The only way to genuinely understand tokenization, attention, training dynamics and inference is to build every piece yourself — which means resisting the shortcut of pretrained weights, hosted APIs and imported vocabularies at every step.
Key constraints
Three rules are non-negotiable and enforced in code: no pretrained LLM weights (the model starts from random initialization), no external LLM APIs anywhere in the training, data or serving path, and no pretrained tokenizer. The allowed foundation stops at PyTorch, CUDA and NumPy. Hardware is a single 8 GB laptop GPU, so every stage must stream with bounded memory.
Architecture overview
The system is built in verified phases: a reproducibility foundation (config fingerprints, seeded RNG capture, experiment tracking), then a four-stage streaming data pipeline — ingestion with full provenance, quality scoring, exact/normalized/near deduplication, and corpus audit with a document-disjoint train/val/test split. Next comes the own-trained BPE tokenizer, then a decoder-only Transformer, training engine and inference layer, gated by a mandatory overfit test before any real training run.
Asynchronous pipeline → event-sourced knowledge
Own BPE tokenizer
append-only · time-travel queries · conflict detection
From-scratch Transformer
AI suggests with evidence — a person approves, edits or rejects
How it works
What's actually in it.
- 01Streaming, disk-backed data pipeline: ingestion → cleaning → deduplication → corpus audit, each stage a self-contained config-driven CLI.
- 02Near-duplicate detection via MinHash + LSH implemented from scratch in NumPy only.
- 03Full provenance on every document — source, license, SHA-256 hashes and lineage from raw file to final snapshot.
- 04Bit-for-bit deterministic reruns: same seed + config fingerprint reproduces the corpus exactly.
- 05Protected eval sources are never leaked into training; contamination is measured and reported.
- 06Mandatory overfit gate: a ~1M-parameter model must drive loss to ~0 on a tiny corpus before any real training is allowed.
Engineering decisions
The from-scratch rule is enforced by code, not by promise
A 'built from scratch' claim is easy to make and easy to quietly break. Here NO_PRETRAINED_WEIGHTS is a guard enforced at every checkpoint-loading boundary and covered by tests — any code path that tries to load foreign weights is a defect by definition. The same discipline runs through the data: every document carries source, license, hashes and lineage, and every stage must balance its accounting (input = retained + removed + errors) or the run fails.
The trade-off
Starting from random initialization on an 8 GB laptop GPU is enormously slower to any usable capability than fine-tuning an open model — and the first trained model (~10M parameters) will not be competitive with anything. That is accepted deliberately: the milestone is scientific, proving the full pipeline end to end, with a scaling ladder (1M overfit gate → 10M v0.1 → upward) climbed only as each rung is verified.
Technical highlights
- Zero pretrained weights, APIs or vocabularies — enforced in code and covered by tests.
- 70M-word source deduplicated in streaming mode with ~327 MB peak RSS on laptop hardware.
- Honest status reporting: the Phase 5 audit flags the current corpus as synthetic and not yet training-ready, recorded as an explicit gap.
Claims & evidence
280 tests passing, with ruff and mypy clean across 140 files.
Measured · S1-LLM test suite and per-phase verification reports
Data pipeline verified on 113.7M raw words across sources, distilled to a 16.5M-word balanced corpus with a bit-for-bit reproducible fingerprint.
Measured · Phase 5 verification report (corpus_0002)
Phases 0–5 (charter, foundation, full data pipeline) are implemented and verified; the tokenizer and model phases are next, not yet built.
Qualitative · S1-LLM phase status
Every statement above carries its evidence class — nothing is invented.
Tech stack
- Python 3.12
- PyTorch
- NumPy
- Own BPE tokenizer
- MinHash + LSH (from scratch)
- YAML-driven configs
- pytest
- ruff + mypy
Want to go deeper on MyLLM (S1-LLM)?
I'm happy to walk through the architecture and the decisions behind it.