I tried to replace a 120B LLM with a 0.5B model. Here are the numbers.
Distill a large LLM into a small, cheap model for one task, and measure whether it is good enough.
Exact match on 80 hand-labeled tickets
Real results from the repository, not a simulation. Hover or tap a bar.
A small browser version with a deterministic fake model — no API key, nothing leaves your device. The real project is in the repo.
What it solves
An expensive LLM call on every support ticket adds up. A small model you run yourself could be cheaper — if it is good enough. Most teams guess; this pipeline measures.
How it works
- Large models write synthetic tickets and two teachers label them; only tickets where both agree on all 5 fields are kept as training data (435 clean examples from 784 requested seeds).
- A small student (Qwen2.5-0.5B) is fine-tuned with LoRA on 384 synthetic examples, on a CPU, in about one hour (57 minutes).
- A router sends each ticket to the student first and falls back to the teacher when the answer is invalid or the student's confidence is below a threshold.
The honest verdict
Fine-tuning took the student from unusable (1.2%) to 46.2% exact match, but that is not enough to replace the teacher (75.0%), and on a CPU it was not even cheaper than the same teacher at its paid list price ($0.70 vs $0.29 per 1,000 tickets; the $0.70 uses an assumed $0.68/h VM price). With the router the student keeps 21% of the tickets and the overall score is 73.8% (76.5% on the tickets it kept); on 80 tickets the 95% interval is about ±7–10 pts. A stronger paid teacher raises the ceiling (Claude Haiku 5.5: 83.8% alone, 82.5% routed; Claude Opus 5.5: 88.8% and 85.0%), but the student was not retrained on its labels, and now it is the student that holds the router back. Next step: a bigger student on a GPU (config included, not run yet).
Next demoCRM Copilot →