skillsmith / scan verdict

Loading…

Markdown for your README

Heuristic static scan — not a security guarantee. Scan your own skill at skillsmith.ch.

Understanding Your Scan Results

Every Claude Agent Skill you publish, install, or point an AI agent at should be audited for security risks. skillsmith performs a multi-layered static analysis on each SKILL.md you submit, looking for known attack patterns documented in the AI-security research community. The scanner combines three detection engines:

Prompt-injection phrase matching — flags 700+ phrases that try to override Claude's system prompt, ignore prior instructions, exfiltrate gathered data, or chain tool calls toward attacker-controlled goals. The phrase list is built from real-world samples seen in the wild and from published academic threat catalogues.

Code-pattern detection — scans fenced code blocks (Python, JavaScript, shell, PowerShell, Lua) for dangerous primitives: eval, exec, pickle.loads, subprocess, yaml.load without SafeLoader, os.system, dynamic __import__, and dozens more. These are weighted by how directly they can be weaponized in a SKILL.md context.

Normalization + evasion analysis — decodes base64 payloads (including UTF-16), strips zero-width characters and bidi overrides, normalizes Cyrillic/Greek homoglyphs, and re-scans the result. Many real attacks hide their payload behind Unicode obfuscation; we collapse that and look underneath.

Risk Verdict Scale

Each scan produces a risk verdict from clean to critical. A clean verdict means the scanner found no high-confidence indicators of malicious intent. A critical verdict means at least one pattern matched with enough weight to suggest the skill is actively trying to abuse an AI agent. Mid-tier verdicts (low, medium, high) reflect increasing density of suspicious signals.

Important: a clean verdict is a static-analysis signal, not a guarantee. It means the skill didn't trip any of the 1,900+ patterns we currently track. New attack techniques appear regularly, and heuristics can miss novel evasion. Use skillsmith as one input among many — code review, sandbox testing, and source provenance still matter.

What Gets Flagged

Some examples of patterns that will fire a finding:

Direct attempts to override agent behavior: "ignore previous instructions", "disregard all prior rules", "you are now in developer mode".

Data exfiltration: skill asks the agent to read secrets, environment variables, or chat history and POST them to an attacker-controlled Discord webhook or Slack hook.

Tool abuse: skill instructs the agent to run curl, wget, or fetch() to non-allowlisted domains, or to spawn child processes via os.system / subprocess.Popen.

Obfuscation: base64-encoded payloads inside YAML frontmatter, zero-width characters splitting the word ignore across the line break, fullwidth Latin characters used to hide dangerous strings.

Dropper behaviour: skill tells the agent to download a second-stage payload from an external URL and execute it locally.

False Positives and How We Reduce Them

Security scanners are noisy. A skill that legitimately teaches the agent to call eval() on trusted input — for example, a sandboxed calculator skill — will trigger the code-pattern engine. That's by design: the scanner errs on the side of asking you to look. The risk verdict reflects aggregate signal, so a single low-weight match does not push a clean skill into "high" territory. You can always click the snippet link to see the exact line that matched and decide whether it's actually dangerous in context.

If you believe a scan result is wrong, you can file a community report from the API or via the MCP tool. Crowd reports help calibrate the rule weights over time.

How to Get the Most Out of skillsmith

Scan before you install. Even skills from trusted authors can be compromised through supply-chain attacks (typosquatted dependencies, repo-jacking, maintainer-account takeover). Scan every new skill, every time it updates.

Use the watch feature for skills in production. A rug-pull watch pings a Discord or Slack webhook when a tracked SKILL.md changes upstream. A sudden change is a strong signal to re-scan and review the diff.

Compare against the Safe Skills Database. Clean, lint-valid skills can be published to the public database. If you're building an agent that consumes skills, prefer those entries over unscanned ones.

Combine with the behavioral sandbox. Static analysis catches what the text says. The behavioral sandbox simulates what the agent would do if it followed the skill faithfully — catching instructions that don't trigger the static patterns but still feel off.

About This Scanner

skillsmith is built and maintained as an independent research tool. The pattern lists, weight calibrations, and explainer heuristics are all versioned in the public repository at github.com/Larslllllll/skillsmith-web. The scanner is open source, the API is free for low-volume use, and the underlying threat-intelligence work is funded by Pro subscriptions and a small grant from the Solana Foundation. If you find a pattern that should be added (or one that fires too aggressively), please open an issue.