Swap, Replicate, Read the Mass: A Protocol for Auditing Demographic Bias in System One Decision Models
Summary
- Researchers proposed an auditing protocol to evaluate demographic bias in automated System One decision models using typed, probability-scored answers.
- The study evaluated twelve models and a no-thinking ablation across 1,638,000 decisions using adapted benchmarks including BBQ, CrowS-Pairs, and SilverSpoon.
- Results show widespread stereotype alignment, order-dependent answers where option swaps flip choices up to 72.6%, and confidence tracking instability rather than bias.
- Commercial models frequently abstained on ambiguous BBQ items, hiding aligned probability mass, while lower-SES labels were favored in several decision models.
Why it matters
System One decision models execute production tasks without human oversight, making rigorous demographic bias audits and order-invariant scoring essential procurement requirements for AI infrastructure and safety.