{
  "schema_version": "1.1",
  "id": "s13:https://arxiv.org/abs/2610.31657v1",
  "slug": "2610-31657v1-0pyz12o",
  "url": "https://feed7.dev/p/2610-31657v1-0pyz12o",
  "title": "Cost-Aware Abstention for LLM-Based IT Ticket Classification",
  "why_included": "Cost-aware abstention cut wrong ticket routes by deferring uncertain cases, while prompted LLM confidence often failed to distinguish correct answers from errors.",
  "summary": "On **47,837 tickets across eight classes**, TF-IDF plus logistic regression with a calibration-selected 0.60 threshold reduced test cost from 0.763 to 0.472 and wrong routes from **15.3% to 3.5%** under the paper’s cost assumptions.",
  "practical_implication": "For agentic classifiers, define the cost of a wrong action versus human review, freeze the routing policy on calibration data, then test it once on held-out data. Track discrimination and risk coverage, not calibration error alone.",
  "agent_context": "On **47,837 tickets across eight classes**, TF-IDF plus logistic regression with a calibration-selected 0.60 threshold reduced test cost from 0.763 to 0.472 and wrong routes from **15.3% to 3.5%** under the paper’s cost assumptions.\n\nFor agentic classifiers, define the cost of a wrong action versus human review, freeze the routing policy on calibration data, then test it once on held-out data. Track discrimination and risk coverage, not calibration error alone.\n\nThe strongest prompted setup reached **0.477 accuracy**, versus **0.879 for fine-tuned DistilBERT** on the same 1,000-ticket subset. Results depend on one dataset, prompt design, model endpoints, and an illustrative 5:1 wrong-route-to-triage cost.",
  "source": {
    "name": "arXiv",
    "url": "https://arxiv.org/abs/2610.31657v1",
    "published_at": "2026-10-22T17:58:52.000Z"
  },
  "source_class": "blog_post",
  "content_type": "Paper",
  "layer": "benchmark",
  "domains": [
    "data"
  ],
  "topics": [
    "agent-evals",
    "agent-reliability"
  ],
  "verification": {
    "status": "needs_review",
    "label": "Needs Review",
    "method": "unverified",
    "verified_at": null
  },
  "uncertainty": [
    "The strongest prompted setup reached **0.477 accuracy**, versus **0.879 for fine-tuned DistilBERT** on the same 1,000-ticket subset. Results depend on one dataset, prompt design, model endpoints, and an illustrative 5:1 wrong-route-to-triage cost."
  ],
  "connected_context": {
    "meaning": "This turns abstention into an operational cost decision rather than a confidence add-on: calibrate a fixed routing threshold from the relative harm of mistakes and review, then measure held-out risk and coverage. It also sharply narrows expectations for prompted LLM routing, since a conventional fine-tuned classifier substantially outperformed the strongest prompt setup on the tested subset.",
    "corpus_size": 898,
    "generated_at": "2026-10-24T10:28:17.103Z",
    "connections": [
      {
        "title": "Domain-Specific Hallucination Detection in Large Language Models",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.11878v1",
        "feed7_url": "https://feed7.dev/p/2609-11878v1-1ltrudx",
        "reason": "Both show that reliability controls are dataset-dependent: a calibrated threshold or detector must be validated on the deployment domain rather than assumed portable from aggregate calibration results."
      },
      {
        "title": "Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo",
        "source_name": "AI Engineer",
        "source_url": "https://www.youtube.com/watch?v=yqF6XhzbWBk",
        "feed7_url": "https://feed7.dev/p/inside-847-production-clinical-ai-notes-sebastian-fox-composo-0lrc8td",
        "reason": "Cost-aware human triage provides a concrete escalation policy for the broader finding that consequential model errors require expert oversight, while the clinical evidence warns that the review pathway must keep discovering failures beyond a frozen rubric."
      },
      {
        "title": "Dutch Books for Language Models",
        "source_name": "arXiv",
        "source_url": "https://arxiv.org/abs/2609.02797v1",
        "feed7_url": "https://feed7.dev/p/2609-02797v1-15ybio3",
        "reason": "The Dutch-book test complements this held-out cost and coverage evaluation by checking internal probability coherence without labels; coherence alone would not determine whether abstention reduces routing harm."
      }
    ]
  },
  "lifecycle": "Current",
  "published_at": "2026-10-22T17:58:52.000Z",
  "modified_at": "2026-10-22T17:58:52.000Z",
  "supersedes": [],
  "expires_at": null,
  "formats": {
    "html": "https://feed7.dev/p/2610-31657v1-0pyz12o",
    "json": "https://feed7.dev/p/2610-31657v1-0pyz12o.json",
    "markdown": "https://feed7.dev/p/2610-31657v1-0pyz12o.md"
  }
}