Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools
A taxonomy-based audit maps open-source LLM safety tools to enterprise risks, finding strong technical coverage but major governance, legal, regulatory, and financial gaps.
The study maps **21 open-source tools** against **32 subcategories** in an extended MIT risk taxonomy. An LLM-assisted retrieval pipeline analyzes code and documentation, with three reviewers assessing the resulting capability mappings.
Builders deploying agent systems can use the same structure to inventory which risks are covered by evaluations, adversarial tests, runtime guardrails, and observability. The central design implication is layered mitigation: technical controls should sit alongside organizational and regulatory processes.
The study maps **21 open-source tools** against **32 subcategories** in an extended MIT risk taxonomy. An LLM-assisted retrieval pipeline analyzes code and documentation, with three reviewers assessing the resulting capability mappings. Builders deploying agent systems can use the same structure to inventory which risks are covered by evaluations, adversarial tests, runtime guardrails, and observability. The central design implication is layered mitigation: technical controls should sit alongside organizational and regulatory processes. The mapping reached **75.5% F1** after majority voting, while reviewer agreement was only **Fleiss’ κ = 0.509**. Coverage also clustered around technical and operational controls, leaving governance, legal, regulatory, financial, and market risks largely outside the tools’ reach.
This turns layered agent-risk mitigation into an auditable coverage matrix, while warning that tool inventories are partly judgment-dependent and systematically underrepresent nontechnical risks. It confirms observability and runtime controls as useful layers, but narrows their role: even broad open-source coverage cannot substitute for governance, legal, regulatory, financial, or market processes.