Rug Pulls: When a Vetted Skill Turns Malicious Later

Guide · skillsmith.ch

You scanned a skill, it came back clean, you installed it. Three weeks later it exfiltrates credentials. Nothing about your review was wrong — the skill simply changed after you looked at it. That is a rug pull, and it is the most underrated attack against agent ecosystems.

Why vetting once is not enough

Packages mutate silently: a GitHub-hosted SKILL.md is a live file, not a snapshot. Whoever has push access can rewrite it any time — your "clean" verdict describes a version that no longer exists.
Social timing: the classic pattern is publish clean, accumulate installs and stars, then flip. The flip lands exactly when the trust around the skill peaks.
Indirect flips: the SKILL.md stays innocent but swaps the remote script it loads — same effect, harder to notice.

What a defense looks like

Pin, don't float: reference the exact commit hash, not a branch. A branch pointer follows every change; a hash cannot lie.
Baseline the content you vetted: record the SHA-256 of the exact text you approved. This is what skillsmith's watch does: POST /api/watch {url} stores the baseline hash of a GitHub-hosted skill.
Re-check on your schedule: a watch check re-fetches the URL and compares hashes — unchanged / changed / unreachable. With an optional Discord or Slack webhook you get pushed an alert automatically, no manual polling.
Treat "changed" as hostile until proven otherwise: a changed hash means everything you read during review is void. Re-run the full checklist, not just a diff skim — and consider that the change may be timed for a moment you are distracted.

Detection limits, honestly stated

Hash watching detects any change — including harmless typo fixes. Expect some noise; the point is that silence is never assumed.
A watcher cannot see commits that were flipped and flipped back between checks. Frequent checks shrink this window.
If the author rotates the hosted URL entirely, the old watch goes unreachable — treat unreachable as a signal too.

The full loop

Scan (skillsmith.ch) → behavioral sandbox (analysis) → watch with webhook → alert on change → re-audit. Related reading: anatomy of a malicious skill, threat modeling.