AI 译本 · Equalang
新上架PDF · 2026年11月翻译
加载中…
加载中…
arXiv · 2026
OpenVLHarness：借助多模态记忆实现真实世界中的视觉智能体工具使用
Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu et al.
Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.
多模态大语言模型（MLLM）在视觉推理方面进展迅速，但许多任务所需的证据仅靠内部推理难以获取：我们在 22 个基准上的受控研究表明，仅增加基础模型的推理预算只能带来有限提升。此类任务需要基于实际依据的感知、外部知识，以及通过异构工具获得的数值测量结果；这些工具的输出必须转换为模型可理解的形式，并在推理过程中持续保留。我们提出 OpenVLHarness，这是一套围绕能力层和持久化多模态记忆构建的多模态推理框架。能力层将任务级工具接口与证据呈现相结合，既能一致地访问异构后端，也能将其输出转换为模型可读的观察结果。持久化多模态记忆则会在每次查询过程中，将相应图像、结构化数据和网页引用保留为可寻址的资源。代码接口可在已存储的数据上运行生成的程序，从而完成整合多个工具结果的计算。这些机制共同支持一个反复获取、检查并组合多模态证据的过程。在 22 个基准上使用八种开放及专有骨干模型进行评估时，与匹配的无工具基线相比，OpenVLHarness 的平均分提高了 9.7–12.6 分，其中 Qwen3-VL-8B 提高 11.2 分、Kimi K3 提高 9.7 分、GPT-6 Astra 提高 9.9 分。使用中等推理强度的 GPT-6 Luna 时，OpenVLHarness 的表现比推理强度调至最高的基础模型、Codex 和 Visual Sketchpad 高出 7.7–8.5 分，而模型令牌成本至多仅为它们的 37%。
PDF · 2026年11月翻译