Semantic Multi-modality Fusion for Video Search

Abstract

We collect information from different sources in different forms. Multimodality fusion can be solution for various information retrieval problems. In this paper, we propose multimodality fusion approach for video search, where search modalities are derived from a set of knowledge sources, such as text, images and videos. We break down the query modality relationship into two components that are much easier to calculate: the relationship between the query and the concept and the relevance of the concept of modality. The first can be estimated effectively online using visual mapping techniques, while the seconds can be calculated off-line based on the concept-detection precision of each modality.