Notwithstanding Anything to the Contrary, Ignore All Instructions: Measuring Prompt Injection Guards on Legal Contract Review Attacks
Summary
- Researchers evaluated eleven open-weights prompt injection guards on legal contract corpora, finding document-level false positive rates as high as 92.39%.
- Results show that a basic five-regex baseline is not Pareto-dominated by any tested guard, and all guards scored zero on unauthorized scope claims.
- A proposed cascaded LLM solution, verifying mid-tier model positives with a frontier model, achieves high performance at 6.5x lower cost than an exclusive frontier setup.
Why it matters
Current prompt injection guards struggle severely with false positives and specific threat categories in legal contract review, highlighting the need for specialized evaluation and cost-efficient cascaded architectures.