Video summarization based on multi-scale feature fusion

Jing Bao, Shipeng Xu, Jing Zhang

Video summarization aims to identify the important segments of a video and form a concise representation, enabling users to quickly grasp the core information. Existing graph-based video summarization methods suffer from insufficient modeling of multi-scale feature interactions and difficulties in balancing local and global features. To address this issue, this paper proposes a multi-scale feature fusion video summarization model based on the Message Passing Neural Network (MPNN) framework. First, the model extracts features of representative frames from video shots, and constructs a graph structure where shot features serve as nodes and inter-shot semantic similarity as edges. Second, it decouples the multi-scale features in the graph via the MPNN, and utilizes the Graph Attention Network (GAT) and Graph Neural Network (GNN) to extract local correlation features and global features, respectively. Finally, it fuses the original shot features, local features and global features as the final features to calculate shot importance and generate the video summary. On SumMe (F1 = 50.0) and TVSum (F1 = 61.8) datasets, MSF-MPNN achieves competitive performance against mainstream RNN/GNN-based methods. MSF-MPNN provides a practical solution for efficient video content extraction, with potential in video surveillance and short-video platforms.