IRON-Meme: A Benchmark for Structured Understanding of Identities and Roles in Online Meme Narratives
Summary
- Researchers introduced IRON-Meme, a human-annotated benchmark containing 1,110 political memes designed to evaluate structured understandings of identities and roles.
- The framework incorporates social science theories to categorize social identities, viewpoints, and narrative roles like moral character, intelligence, status, and power.
- An analysis of IRON-Meme reveals that most memes assign derogatory roles and that identities, roles, and templates are systematically interconnected.
- Evaluating nine vision-language models showed that frontier-scale and reasoning models perform best, though accuracy drops for less frequent labels.
Why it matters
Understanding complex narrative roles and social nuances in online media is a key frontier for vision-language models, and benchmarks like IRON-Meme provide critical evaluation data for automated content analysis.