We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank
Treat agent skills as supply-chain dependencies. Nubank scans them locally and in CI with deterministic rules plus LLM review, then gates marketplace distribution and feeds findings into vulnerability management.
Nubank’s internal pipeline scanned **more than 2,000 skills** and identified **over 1,500 risks**, with more than 1,000 subsequently remediated. Checks cover unsafe instructions, destructive shell commands, credential handling, excessive permissions, data exposure, and risky MCP use.
Put a canonical marketplace behind a security gate. Let authors scan locally, rerun the current scanner in CI, combine deterministic checks with contextual LLM review, report actionable findings in the pull request, and enforce approval at the tool layer rather than in prompts.
Nubank’s internal pipeline scanned **more than 2,000 skills** and identified **over 1,500 risks**, with more than 1,000 subsequently remediated. Checks cover unsafe instructions, destructive shell commands, credential handling, excessive permissions, data exposure, and risky MCP use. Put a canonical marketplace behind a security gate. Let authors scan locally, rerun the current scanner in CI, combine deterministic checks with contextual LLM review, report actionable findings in the pull request, and enforce approval at the tool layer rather than in prompts. Risk counts are findings, not the number of vulnerable skills, and one skill may produce several. The talk provides no false-positive rate or scanner effectiveness benchmark; it also notes that weak signals and context-dependent severity required refinement.
This turns skill governance from library hygiene into a supply-chain control: portable instructions and tool access need repeatable scanning, review, and tool-layer enforcement before distribution. It reinforces treating skills as governed product features and makes selective installation more consequential. The reported remediation volume demonstrates operational scale, while the missing false-positive and effectiveness measures limit claims about scanner quality.