Multimodal emotion reasoning requires models to recognize emotions and explain them with evidence from visual behavior, acoustic patterns, and language context. Although recent Omni-MLLMs have shown promising reasoning ability, their explanations are not always perceptually reliable: a model may ignore useful fine-grained cues, or mention visual and acoustic evidence that is not actually supported by the input.
We identify two key perception problems behind this failure: insufficient utilization of multimodal evidence and unfaithful cross-modal hallucination. To address them, we propose Omni-Perception Policy Optimization (OPPO), a perception-aware optimization framework for emotion Omni-MLLMs.
OPPO introduces an Omni-Perception Reward to encourage generated reasoning to recover annotated video, audio, and emotion cues, and an Omni-Perception Loss to test whether modality-specific claims remain valid when video or audio evidence is masked. These two objectives jointly push the model to use enough evidence while keeping its explanations grounded in the correct modality.
We further build MEP-Bench, a diagnostic benchmark for measuring evidence utilization and perception faithfulness. Experiments on MEP-Bench and MER-UniBench show that improving perceptual grounding leads to more reliable reasoning and stronger multimodal emotion recognition performance.
OPPO optimizes multimodal emotion reasoning from two complementary perspectives: whether the model uses enough perceptual evidence, and whether each modality-specific claim is faithful to the modality that actually supports it. As shown in the overview figure, the upper branch builds an evidence-aware reward, while the lower branch introduces a modality-masking loss to suppress hallucinated visual or acoustic cues.
Omni-Perception Reward: Given human-annotated video, audio, and emotion cues, OPPO first constructs a multimodal evidence set. The generated reasoning is then split into clauses and aligned with these cues through an evidence-routing matrix. This reward encourages the model to recover sufficient emotion-relevant evidence during reasoning, instead of only optimizing for the final emotion label or response format.
Omni-Perception Loss: To make the reasoning faithful to its perceptual source, OPPO masks one modality at a time and compares the token-level distributions of modality-specific evidence before and after masking. If a visual or acoustic claim remains confident even when its supporting modality is removed, the loss penalizes it. This discourages cross-modal hallucination and keeps the explanation grounded in visible facial behavior, acoustic cues, and valid emotion evidence.
Qualitative comparison: OPPO retrieves richer multimodal evidence during reasoning and avoids hallucinating modality-specific cues under masking. Compared with the baseline, the explanations are more tightly grounded in visible facial behavior, acoustic cues, and emotion-relevant evidence.
MEP-Bench is derived from OV-MERD and evaluates whether reasoning uses human-annotated multimodal cues and whether visual or acoustic claims remain faithful under unimodal masking. It separates underutilization from cross-modal hallucination through evidence recall and masked perception accuracy.
OPPO achieves the best mean score among compared Omni-MLLM baselines, with consistent improvements across sentiment analysis, basic emotion recognition, and fine-grained emotion recognition.
OPPO achieves the best overall performance on MME-Emotion, improving task-level emotion reasoning and the mean recognition, reasoning, and CoT scores.
@inproceedings{han2026omniperception,
title={Omni-Perception Policy Optimization for Multimodal Emotion Reasoning},
author={Zhiyuan Han and Beier Zhu and Wenwen Tong and Pengyang Shao and Peipei Song and Xinyi Wang and Jiangnan Chen and Lewei Lu and Xun Yang},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=4dwe2wlxx5}
}