Scaling Compute on Context — Jack Morris, Engram
Training directly on a private corpus can memorize it without producing useful behavior. Durable context learning likely needs synthetic study tasks and recursive self-improvement, not a single fine-tuning pass.
Morris frames the problem as teaching a pretrained model an unstructured private corpus. Direct next-token training on **10,000 financial reports** can drive loss to **0.00001** yet still produce collapsed generations, showing that memorization is not usable knowledge.
For private code, messages, or documents, distinguish retrieval from learning. Context compression, on-policy distillation, synthetic question generation, and continued pretraining offer different tradeoffs; the desired system should keep generating harder study material as the model improves.
Morris frames the problem as teaching a pretrained model an unstructured private corpus. Direct next-token training on **10,000 financial reports** can drive loss to **0.00001** yet still produce collapsed generations, showing that memorization is not usable knowledge. For private code, messages, or documents, distinguish retrieval from learning. Context compression, on-policy distillation, synthetic question generation, and continued pretraining offer different tradeoffs; the desired system should keep generating harder study material as the model improves. Every described method has limits: context methods require the data to fit, synthetic training can overwrite prior knowledge, and one-pass approaches plateau. Recursive self-improvement is presented as the research target, not a settled recipe.
This rules out low training loss as evidence that a private corpus has become usable knowledge and turns corpus adaptation into a choice among retrieval, compression, and parameter updates. It reinforces hybrid memory designs and workload-specific evaluation, while warning that consolidation can overwrite prior knowledge and that recursive improvement remains an open target.