VisualFLIP Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?

Modern multimodal LLMs answer many visual-reasoning questions correctly. But single-image accuracy can hide answers that are not driven by the visual evidence — driven instead by language priors, memorized regularities, or whatever a confident reasoning chain happens to settle on. VisualFLIP turns this into a behavioral test. 现代多模态大模型能答对很多视觉推理题。但单图准确率会掩盖那些并非依赖视觉证据的答案——它们可能源自语言先验、记忆的规律，或一段听上去自洽的推理链恰好落到的位置。VisualFLIP 把这件事变成一个行为测试。

One pair = one question, two images. The second image makes a minimal task-critical edit that deterministically flips the gold answer. A prediction sensitive to the evidence should change between the two; a prediction that repeats the same answer despite the gold flipping collapses. The benchmark is behavioural — it measures whether predictions move with the evidence, not the internal mechanism behind any individual answer. 一对样本 = 一个问题，两张图。第二张图做一个对任务关键的最小化编辑，使标准答案确定性翻转。对证据敏感的预测应当随之改变；若两张图给出相同答案而标准答案已翻转，则发生坍塌。这是一个行为基准——它衡量预测是否随证据更新，并不主张任何关于内部机制的结论。

140 of the 687 pairs additionally carry an irrelevant_image control arm to test answer stability under non-task-critical edits. 其中 140 对样本额外带 irrelevant_image 对照臂，用于检验答案在非任务关键编辑下的稳定性。

Table 1 uses the independent protocol (each image queried separately); Table 2 uses the sequential two-turn protocol (original then edited in one conversation; SeqCR measures original-answer persistence). Within each block, rows are ranked by Acc_p; bold marks the best value in a column. 表 1 为独立协议(两图分别独立询问);表 2 为序列两轮协议(同一对话先原图后编辑图,SeqCR 衡量原答案的惯性保持)。每个分块内按 Acc_p 排名;加粗为该列最优。

Table 1 — Independent evaluation表 1 — 独立评测

Table 2 — Sequential evaluation表 2 — 序列评测

@article{zhu2026visualflip,
  title   = {VisualFLIP: Do Predictions Depend on Task-Critical
             Visual Evidence in Multimodal Reasoning?},
  author  = {Zhu, Didi and Chen, Changrui and Zafeiriou, Stefanos and Deng, Jiankang},
  year    = {2026},
  journal = {arXiv preprint}
}

If you use the real-image pairs (source == "real_mathvision"), please also cite the MathVision benchmark (Wang et al., 2024). 如果使用了真实图像样本（source == "real_mathvision"），请同时引用 MathVision 基准（Wang et al., 2024）。