# Cost-Aware Abstention for LLM-Based IT Ticket Classification

Source: [arXiv](https://arxiv.org/abs/2610.31657v1)  
Feed7 permalink: https://feed7.dev/p/2610-31657v1-0pyz12o  
Published: 2026-10-22T17:58:52.000Z  
Trust: Needs Review (needs_review)

## Why Included

Cost-aware abstention cut wrong ticket routes by deferring uncertain cases, while prompted LLM confidence often failed to distinguish correct answers from errors.

## Source Summary

On **47,837 tickets across eight classes**, TF-IDF plus logistic regression with a calibration-selected 0.60 threshold reduced test cost from 0.763 to 0.472 and wrong routes from **15.3% to 3.5%** under the paper’s cost assumptions.

## Practical Implication

For agentic classifiers, define the cost of a wrong action versus human review, freeze the routing policy on calibration data, then test it once on held-out data. Track discrimination and risk coverage, not calibration error alone.

## Agent-Ready Context

On **47,837 tickets across eight classes**, TF-IDF plus logistic regression with a calibration-selected 0.60 threshold reduced test cost from 0.763 to 0.472 and wrong routes from **15.3% to 3.5%** under the paper’s cost assumptions.

For agentic classifiers, define the cost of a wrong action versus human review, freeze the routing policy on calibration data, then test it once on held-out data. Track discrimination and risk coverage, not calibration error alone.

The strongest prompted setup reached **0.477 accuracy**, versus **0.879 for fine-tuned DistilBERT** on the same 1,000-ticket subset. Results depend on one dataset, prompt design, model endpoints, and an illustrative 5:1 wrong-route-to-triage cost.

## Connected Context

Feed7 judgment across 898 accumulated Signals:

This turns abstention into an operational cost decision rather than a confidence add-on: calibrate a fixed routing threshold from the relative harm of mistakes and review, then measure held-out risk and coverage. It also sharply narrows expectations for prompted LLM routing, since a conventional fine-tuned classifier substantially outperformed the strongest prompt setup on the tested subset.

- [Domain-Specific Hallucination Detection in Large Language Models](https://feed7.dev/p/2609-11878v1-1ltrudx) — Both show that reliability controls are dataset-dependent: a calibrated threshold or detector must be validated on the deployment domain rather than assumed portable from aggregate calibration results.
- [Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo](https://feed7.dev/p/inside-847-production-clinical-ai-notes-sebastian-fox-composo-0lrc8td) — Cost-aware human triage provides a concrete escalation policy for the broader finding that consequential model errors require expert oversight, while the clinical evidence warns that the review pathway must keep discovering failures beyond a frozen rubric.
- [Dutch Books for Language Models](https://feed7.dev/p/2609-02797v1-15ybio3) — The Dutch-book test complements this held-out cost and coverage evaluation by checking internal probability coherence without labels; coherence alone would not determine whether abstention reduces routing harm.

## Context Map

- Layer: benchmark
- Domains: data
- Topics: agent-evals, agent-reliability

## Uncertainty

- The strongest prompted setup reached **0.477 accuracy**, versus **0.879 for fine-tuned DistilBERT** on the same 1,000-ticket subset. Results depend on one dataset, prompt design, model endpoints, and an illustrative 5:1 wrong-route-to-triage cost.

## Agent Instruction

Use this item as source-backed context. Do not invent claims beyond the linked source. If this item conflicts with another source, call out the conflict.