# Scaling Compute on Context — Jack Morris, Engram

Source: [AI Engineer](https://www.youtube.com/watch?v=WiqDvX6isc4)  
Feed7 permalink: https://feed7.dev/p/scaling-compute-on-context-jack-morris-engram-1pxr9bt  
Published: 2026-08-12T15:30:14.000Z  
Trust: Source Linked (source_linked)

## Why Included

Training directly on a private corpus can memorize it without producing useful behavior. Durable context learning likely needs synthetic study tasks and recursive self-improvement, not a single fine-tuning pass.

## Source Summary

Morris frames the problem as teaching a pretrained model an unstructured private corpus. Direct next-token training on **10,000 financial reports** can drive loss to **0.00001** yet still produce collapsed generations, showing that memorization is not usable knowledge.

## Practical Implication

For private code, messages, or documents, distinguish retrieval from learning. Context compression, on-policy distillation, synthetic question generation, and continued pretraining offer different tradeoffs; the desired system should keep generating harder study material as the model improves.

## Agent-Ready Context

Morris frames the problem as teaching a pretrained model an unstructured private corpus. Direct next-token training on **10,000 financial reports** can drive loss to **0.00001** yet still produce collapsed generations, showing that memorization is not usable knowledge.

For private code, messages, or documents, distinguish retrieval from learning. Context compression, on-policy distillation, synthetic question generation, and continued pretraining offer different tradeoffs; the desired system should keep generating harder study material as the model improves.

Every described method has limits: context methods require the data to fit, synthetic training can overwrite prior knowledge, and one-pass approaches plateau. Recursive self-improvement is presented as the research target, not a settled recipe.

## Connected Context

Feed7 judgment across 468 accumulated Signals:

This rules out low training loss as evidence that a private corpus has become usable knowledge and turns corpus adaptation into a choice among retrieval, compression, and parameter updates. It reinforces hybrid memory designs and workload-specific evaluation, while warning that consolidation can overwrite prior knowledge and that recursive improvement remains an open target.

- [UniMem: Complementary Episodic-to-Parametric Memory for Boundary-Agnostic Task Streams](https://feed7.dev/p/2607-26017v1-1opv1da) — UniMem operationalizes the retrieval-versus-learning distinction by routing novel episodes to retrieval and recurring patterns to parameters, but does not settle overwrite risk.
- [Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains](https://feed7.dev/p/2608-05138v1-0bvu6le) — The Greek RAG study reinforces the need to evaluate corpus access methods on the actual domain rather than assume learned or dense representations will outperform lexical retrieval.
- [Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection](https://feed7.dev/p/2608-02560v1-1u8ls14) — PRECOG offers an architecture-specific way to reuse corpus context without retraining or repeated ingestion, covering the retrieval side of the tradeoff.
- [Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI](https://feed7.dev/p/citation-needed-provenance-for-llm-built-knowledge-graphs-daniel-chalef-1iob5t8) — Provenance becomes an implementation requirement when compressed, synthesized, or learned corpus knowledge can no longer be traced through simple source pointers.

## Context Map

- Layer: context
- Domains: data, research
- Topics: context-engineering, retrieval, agent-memory

## Uncertainty

- Every described method has limits: context methods require the data to fit, synthetic training can overwrite prior knowledge, and one-pass approaches plateau. Recursive self-improvement is presented as the research target, not a settled recipe.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.