Measure Before You Train: Typed Decision Models vs. a Classical Baseline vs. a Generative LLM for Routing a Personal Multilingual Inbox
A single-user inbox study finds that about 200 human labels let logistic regression outperform tested LLMs on classification accuracy, yet it missed more actionable emails. Measure action costs, not accuracy alone.
The study evaluates nine inbox-routing systems on **739 Italian and English emails** with thread-grouped cross-validation. With about **200 labels**, logistic regression led on classification accuracy, while missing 18% of emails needing action versus 4% for the best LLM; a cost-based noise threshold reduced its miss rate by half.
If you build a routing agent, create human-labeled ground truth and measure the outcomes of its composed actions. For sender relationships, the study found information from other messages by a known contact was more useful than prompting from one email.
The study evaluates nine inbox-routing systems on **739 Italian and English emails** with thread-grouped cross-validation. With about **200 labels**, logistic regression led on classification accuracy, while missing 18% of emails needing action versus 4% for the best LLM; a cost-based noise threshold reduced its miss rate by half. If you build a routing agent, create human-labeled ground truth and measure the outcomes of its composed actions. For sender relationships, the study found information from other messages by a known contact was more useful than prompting from one email. This is a **single-user case study**; the emails are private and not released. The classifier also never predicted the two rarest classes, so its accuracy advantage does not mean it handles every class well.
This single-user inbox study shows why routing systems should be judged on action-relevant errors as well as accuracy: a conventional classifier led on accuracy, while an LLM missed fewer actionable emails, and a cost-based threshold reduced those misses. The private dataset and absent predictions for rare classes limit how far to generalize the result.