PuzzleVQA: Diagnosing Multimodal Reasoning Skills of Language Models with Abstract Visual Patterns

1Singapore University of Technology and Design 2DAMO Academy, Alibaba Group

PuzzleVQA is a challenging benchmark for multimodal reasoning over abstract patterns. The puzzle components include diverse layouts and objects, requiring perception and reasoning to solve.

Abstract

Large multimodal models extend the impressive capabilities of large language models by integrating multimodal understanding abilities. However, it is not clear how they can emulate the general intelligence and reasoning ability of humans. As recognizing patterns and abstracting concepts are key to general intelligence, we introduce PuzzleVQA, a collection of puzzles based on abstract patterns. With this dataset, we evaluate large multimodal models with abstract patterns based on fundamental concepts, including colors, numbers, sizes, and shapes. Through our experiments on state-of-the-art large multimodal models, we find that they are not able to generalize well to simple abstract patterns. Notably, even GPT-4V cannot solve more than half of the puzzles.

To diagnose the reasoning challenges in large multimodal models, we progressively guide the models with our ground truth reasoning explanations for visual perception, inductive reasoning, and deductive reasoning. Our systematic analysis finds that the main bottlenecks of GPT-4V are weaker visual perception and inductive reasoning abilities. Through this work, we hope to shed light on the limitations of large multimodal models and how they can better emulate human cognitive processes in the future.

Puzzle Samples

Results

RPO Results

Our detailed analysis finds that the large gap between models like GPT-4V and human performance can be attributed to their weaker visual perception and inductive reasoning abilities.

BibTeX

@article{chia2024puzzlevqa,
  author    = {Yew Ken Chia and Vernon Toh Yan Han and Deepanway Ghosal and Lidong Bing and Soujanya Poria},
  title     = {PuzzleVQA: Diagnosing Multimodal Reasoning Skills of Language Models with Abstract Visual Patterns},
  journal   = {ACL},
  year      = {2024},
}