Distill Globally, Adapt Locally: Reasoning Distillation and Product-Type Test-Time Training for Scalable Trade-Up Recommendation
A compact classifier distilled from LLM rationales handled product-pair decisions without inference-time LLM calls. Category adapters improved accuracy further while retaining large speed and cost gains.
A **15.5M-parameter** student uses two precomputed 768-dimensional product embeddings at inference. Reasoning distillation reached **0.924 AUC**, versus 0.912 for label-only training, on 8,352 annotated pairs.
For high-volume classification, use an LLM to produce structured supervision offline, then serve a compact non-generative model. Product-type adapters raised AUC to **0.941** without unfreezing the shared student.
A **15.5M-parameter** student uses two precomputed 768-dimensional product embeddings at inference. Reasoning distillation reached **0.924 AUC**, versus 0.912 for label-only training, on 8,352 annotated pairs. For high-volume classification, use an LLM to produce structured supervision offline, then serve a compact non-generative model. Product-type adapters raised AUC to **0.941** without unfreezing the shared student. On a 100K-pair proxy catalog, the student was about **5,000x faster and 10,000x cheaper** than direct LLM inference. Those estimates used one eight-GPU machine, and the evidence is specific to trade-up recommendations rather than coding-agent workloads.
This provides workload-specific evidence for moving LLM reasoning out of the serving path: generate richer supervision offline, deploy a compact embedding-based classifier, then adapt only small product-type components. It complements data-quality and distillation evidence but narrows the lesson to repetitive trade-up classification; the reported speed and cost advantages do not establish transfer to generative or coding-agent tasks.