IRON-Meme: A Benchmark for Structured Understanding of Identities and Roles in Online Meme Narratives

Impact 6 arXiv cs.AI

Summary

  • Researchers introduced IRON-Meme, a human-annotated benchmark containing 1,110 political memes designed to evaluate structured understandings of identities and roles.
  • The framework incorporates social science theories to categorize social identities, viewpoints, and narrative roles like moral character, intelligence, status, and power.
  • An analysis of IRON-Meme reveals that most memes assign derogatory roles and that identities, roles, and templates are systematically interconnected.
  • Evaluating nine vision-language models showed that frontier-scale and reasoning models perform best, though accuracy drops for less frequent labels.

Why it matters

Understanding complex narrative roles and social nuances in online media is a key frontier for vision-language models, and benchmarks like IRON-Meme provide critical evaluation data for automated content analysis.

Who should care