Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease End
Lease End replaced a fine-tuned intent classifier with skills and runtime context, cutting production fixes from about a week to under an hour. Higher API spend was offset by lower maintenance cost.
Lease End's fine-tuned classifier contributed to **$12 million in revenue at 50× ROI**, yet production errors could trigger unwanted calls. Gathering examples, labeling, retraining, regression testing, and deployment took about **one week** per repair cycle.
The team rebuilt the workflow around model-agnostic skills, tools, and resources. Fixes became edits to prompts or Markdown skills, checked against a curated evaluation set and deployed through S3 in **under one hour**.
Lease End's fine-tuned classifier contributed to **$12 million in revenue at 50× ROI**, yet production errors could trigger unwanted calls. Gathering examples, labeling, retraining, regression testing, and deployment took about **one week** per repair cycle. The team rebuilt the workflow around model-agnostic skills, tools, and resources. Fixes became edits to prompts or Markdown skills, checked against a curated evaluation set and deployed through S3 in **under one hour**. This is one company's structured messaging task, not proof that fine-tuning is generally inferior. The replacement cost more per message, and the speaker still leaves room for fine-tuning where privacy, offline operation, or inability to call a frontier model governs the choice.
This provides a concrete maintenance argument for moving one structured workflow from model weights into editable, evaluated skills: repairs fell from a week-long retraining cycle to under an hour. It narrows any general anti-fine-tuning conclusion because the replacement cost more per message and the evidence comes from one task, with privacy and offline constraints still favoring fine-tuning in some cases.