Cost-Aware Abstention for LLM-Based IT Ticket Classification
Cost-aware abstention cut wrong ticket routes by deferring uncertain cases, while prompted LLM confidence often failed to distinguish correct answers from errors.
On **47,837 tickets across eight classes**, TF-IDF plus logistic regression with a calibration-selected 0.60 threshold reduced test cost from 0.763 to 0.472 and wrong routes from **15.3% to 3.5%** under the paper’s cost assumptions.
For agentic classifiers, define the cost of a wrong action versus human review, freeze the routing policy on calibration data, then test it once on held-out data. Track discrimination and risk coverage, not calibration error alone.
On **47,837 tickets across eight classes**, TF-IDF plus logistic regression with a calibration-selected 0.60 threshold reduced test cost from 0.763 to 0.472 and wrong routes from **15.3% to 3.5%** under the paper’s cost assumptions. For agentic classifiers, define the cost of a wrong action versus human review, freeze the routing policy on calibration data, then test it once on held-out data. Track discrimination and risk coverage, not calibration error alone. The strongest prompted setup reached **0.477 accuracy**, versus **0.879 for fine-tuned DistilBERT** on the same 1,000-ticket subset. Results depend on one dataset, prompt design, model endpoints, and an illustrative 5:1 wrong-route-to-triage cost.
This turns abstention into an operational cost decision rather than a confidence add-on: calibrate a fixed routing threshold from the relative harm of mistakes and review, then measure held-out risk and coverage. It also sharply narrows expectations for prompted LLM routing, since a conventional fine-tuned classifier substantially outperformed the strongest prompt setup on the tested subset.