arXivPaperNeeds Review
Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools
A taxonomy-based audit maps open-source LLM safety tools to enterprise risks, finding strong technical coverage but major governance, legal, regulatory, and financial gaps.
arXiv
Source Summary
The study maps **21 open-source tools** against **32 subcategories** in an extended MIT risk taxonomy. An LLM-assisted retrieval pipeline analyzes code and documentation, with three reviewers assessing the resulting capability mappings.
Practical Implication
Builders deploying agent systems can use the same structure to inventory which risks are covered by evaluations, adversarial tests, runtime guardrails, and observability. The central design implication is layered mitigation: technical controls should sit alongside organizational and regulatory processes.
Agent-Ready Context
The study maps **21 open-source tools** against **32 subcategories** in an extended MIT risk taxonomy. An LLM-assisted retrieval pipeline analyzes code and documentation, with three reviewers assessing the resulting capability mappings. Builders deploying agent systems can use the same structure to inventory which risks are covered by evaluations, adversarial tests, runtime guardrails, and observability. The central design implication is layered mitigation: technical controls should sit alongside organizational and regulatory processes. The mapping reached **75.5% F1** after majority voting, while reviewer agreement was only **Fleiss’ κ = 0.509**. Coverage also clustered around technical and operational controls, leaving governance, legal, regulatory, financial, and market risks largely outside the tools’ reach.
Context Map
infrasecurity#observability#enterpriseUncertainty
The mapping reached **75.5% F1** after majority voting, while reviewer agreement was only **Fleiss’ κ = 0.509**. Coverage also clustered around technical and operational controls, leaving governance, legal, regulatory, financial, and market risks largely outside the tools’ reach.