Extracting objects and events from MPEG sequences for video highlights indexing and retrieval

Automatic recognition of highlights from videos is a fundamental and challenging problem for content-based indexing and retrieval applications. In this paper, we propose techniques to solve this problem using knowledge supported extraction of semantics, and compressed-domain processing is employed for efficiency. Firstly, knowledgebased rules are utilized for shot detection on extracted DCimages, and statistical skin detection is applied for human object detection. Secondly, through filtering outliers in motion vectors, improved detection of camera motions like zooming, panning and tilting are achieved. Video highlight high-level semantics are then automatically extracted via low-level analysis in the detection of human objects and camera motion events, and finally these highlights are taken for shot-level annotation, indexing and retrieval. Results using a large test video data set have demonstrated the accuracy and robustness of the proposed techniques.