AI 译本 · Equalang
新上架PDF · 2026年11月翻译
加载中…
加载中…
arXiv · 2026
VidDub：面向两步视频配音的替换感知蒸馏
Yexin Liu, Yunlong Lin, Zehan Wang, Guangxin He et al.
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.
视频配音旨在合成与目标语音相匹配的唇部动作，同时保留源视频中的说话人身份、头部姿态和背景。现有编辑器主要通过原始配对音频进行自重建训练，然而，当驱动语音被替换时，源视频中的嘴部动作就会成为与目标相冲突的监督信号。我们提出替换感知的跨音频监督：对于配对音频样本，采用全帧重建；对于非配对的替换音频样本，则屏蔽源视频嘴部的重建损失，仅通过音视频同步来监督嘴部，同时保留未屏蔽的面部区域和背景。鉴于离线配音可预先获得完整的源视频和目标语音，我们提出 VidDub：一种全上下文扩散编辑器，经蒸馏后仅需两步去噪（2 NFE）。我们的替换感知教师模型首先学习准确的唇音同步，避免回归到与目标冲突的源嘴部动作。随后，通过匹配轨迹回归，将教师模型经 CFG 引导的轨迹迁移到两步学生模型；分布匹配蒸馏则在压缩步数后恢复视觉质量。我们将发布经过整理的训练数据集、VidDub-Bench、固定的配对音频和跨音频评测清单、预处理结果以及统一的评测代码。在三个训练随机种子下，屏蔽冲突的源嘴部重建使跨音频 Sync-C 提升了 0.833；最终的两步模型在四个基准上兼具出色的唇音同步、视觉质量、身份保持能力和视觉语音识别表现。
PDF · 2026年11月翻译