Results
Tripwire flags 44 of 50 attack cases and leaves 41 of 50 benign cases unflagged. Every case is versioned in the repository and every result is available in the machine-readable report.
| Predicted risk | Predicted clean | |
|---|---|---|
| Attack | 44 true positives | 6 false negatives |
| Benign | 9 false positives | 41 true negatives |
Balanced by design
The set contains 50 attack cases and 50 benign cases. Attack families cover instruction override, authority spoofing, role hijacking, exfiltration, secret solicitation, dangerous actions, control bypass, evidence removal and obfuscation.
Benign hard negatives include security guidance, developer documentation, creative writing, policy text, structured data and ordinary business requests. These deliberately expose where literal signatures confuse discussion of an attack with an attack itself.
Known misses are product requirements.
- 6 false negatives
- Semantic paraphrases, unsupported Spanish, unlabeled encoded text and some indirect secret requests remain outside deterministic coverage.
- 9 false positives
- Quoted attack examples, security policy, transcript syntax and destructive-command documentation can still require human review.
- Residual risk
- Images, audio, cross-origin frames, conversational attacks and tool-call context are not represented by this text-only set.
Methodology
Cases are original maintainer-authored examples informed by public security taxonomies. They are evaluated with the exact engine and ruleset shipped in Inspect, Core and Guard. Any non-clean verdict counts as a positive signal.
No benchmark examples were copied into the rules. The set is public, however, so future tuning can overfit it. Results should be read as transparent regression coverage, not a claim of generalization.
Reproduce locally
$ npm install
$ npm run benchmark
100 cases
Accuracy 85.0%
Precision 83.0%
Recall 88.0%
Specificity 82.0%
TP 44 / TN 41 / FP 9 / FN 6The release pipeline reruns the benchmark, unit tests, extension build, CLI build, MCP handshake and dependency audit.