OmniAlign-V-SFT/DPO Dataset & MM-AlignBench
OmniAlign-V-SFT/DPO Dataset & MM-AlignBench
Recent advancements in open-source multi-modal large language models (MLLMs) have primarily focused on enhancing foundational capabilities, leaving a significant gap in human preference alignment. This paper introduces OmniAlign-V, a comprehensive dataset of 200K high-quality training samples featuring diverse images, complex questions, and varied response formats to improve MLLMs' alignment with human preferences. We also present MM-AlignBench, a human-annotated benchmark specifically designed to evaluate MLLMs' alignment with human values. Experimental results show that finetuning MLLMs with OmniAlign-V, using Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO), significantly enhances human preference alignment while maintaining or enhancing performance on standard VQA benchmarks, preserving their fundamental capabilities.
The pipeline of OmniAlign-V. It contains semantic-rich natural images and infographic images(Chart/Diagram/Poster). We define several tasks and for each task, we propose different methods to construct high-quality questions and images. After this, we further design post-processing refinement to enhance the quality of the dataset.
Our OmniAlign-V SFT dataset not only significantly improves the alignment of MLLMs with human preference, but also boosts the performance of MLLMs on common downstream tasks, particularly on benchmarks like MMVet and MMMU.
Performance of existing MLLMs on MM-AlignBench. B+/B/T/W/W+ denotes MuchBetter/Better/Tie/Worse/MuchWorse. Our LLaVA-Next-OmniAlign(OA)-32B-DPO, trained with OmniAlign-V and applied DPO with OmniAlign-V-DPO, demonstrates outstanding performance, surpassing a wide range of strong MLLMs, even Qwen2VL-72B.
