Why Off-the-Shelf AI Doesn't Understand Money — Udi Menkes, Intuit
Intuit argues that domain context alone does not create experience. Its approach learns from verified state-action-outcome histories, using frontier models only to propose candidates.
In Intuit’s study of roughly **100,000 business situations and time frames**, frontier models reduced more than half of their advice to acquiring customers or increasing revenue. The talk shows recommendations that ignored cash-flow and supplier-concentration constraints despite receiving business context.
Build domain agents from verified outcomes, not documents alone. Intuit converts records into **millions of state-action-outcome vectors**, uses an RL model to rank candidate actions, and then trains an LLM to express the grounded recommendation.
In Intuit’s study of roughly **100,000 business situations and time frames**, frontier models reduced more than half of their advice to acquiring customers or increasing revenue. The talk shows recommendations that ignored cash-flow and supplier-concentration constraints despite receiving business context. Build domain agents from verified outcomes, not documents alone. Intuit converts records into **millions of state-action-outcome vectors**, uses an RL model to rank candidate actions, and then trains an LLM to express the grounded recommendation. Observed outcomes are not automatically causal: a business can improve after an action for unrelated reasons. Similar-company comparisons help, but the validity of the advice still depends on data quality, matching assumptions, and the chosen outcome measures.
This shifts domain-agent grounding from retrieving authoritative documents to learning from verified state-action-outcome records, while narrowing confidence in those recommendations through causal and data-quality caveats. It reinforces deterministic controls around financial reasoning but adds an upstream requirement: the action-ranking evidence itself must be valid, well matched, and tied to suitable outcomes.