Annotated Free-Hand Sketches for Video Retrieval Using Object Semantics and Motion

Hu, Rui; James, Stuart; Collomosse, John

Abstract

We present a novel video retrieval system that accepts annotated free-hand sketches as queries. Existing sketch based video retrieval (SBVR) systems enable the appearance and movements of objects to be searched naturally through pictorial representations. Whilst visually expressive, such systems present an imprecise vehicle for conveying the semantics (e.g. object types) within a scene. Our contribution is to fuse the semantic richness of text with the expressivity of sketch, to create a hybrid ‘semantic sketch’ based video retrieval system. Trajectory extraction and clustering are applied to pre-process each clip into a video object representation that we augment with object classification and colour information. The result is a system capable of searching videos based on the desired colour, motion path, and semantic labels of the objects present. We evaluate the performance of our system over the TSF dataset of broadcast sports footage.

Citation

Hu, R., James, S., & Collomosse, J. (2012, January). Annotated Free-Hand Sketches for Video Retrieval Using Object Semantics and Motion. Presented at MMM 2012: Advances in Multimedia Modeling, Klagenfurt, Austria

• Presentation Conference Type: Conference Paper (published)
• Conference Name: MMM 2012: Advances in Multimedia Modeling
• Start Date: Jan 4, 2012
• End Date: Jan 6, 2012
• Publication Date: 2012
• Deposit Date: Dec 13, 2023
• Print ISSN: 0302-9743
• Publisher: Springer Berlin Heidelberg
• Pages: 473-484
• Series Title: Lecture Notes in Computer Science
• Series Number: 7131
• ISBN: 9783642273544
• DOI: https://doi.org/10.1007/978-3-642-27355-1_44
• Keywords: Advances in Multimedia Modeling 18th International Conference, MMM 2012, Klagenfurt, Austria, January 4-6, 2012, Proceedings
• Public URL: https://durham-repository.worktribe.com/output/1962764
• Publisher URL: https://link.springer.com/chapter/10.1007/978-3-642-27355-1_44