Knowledge graphs have become powerful sources for web search, but an equivalent source about things and their relations in pictures and videos does not exist yet. This project develops core techniques to learn image-centric knowledge graphs by connecting large collections of image/video and their descriptions to existing knowledge bases with encyclopedic, lexical, and commonsense knowledge. One compelling application for multimedia knowledge graphs is in the understanding of ongoing news and social events. We will design methods that construct high-quality knowledge graphs that are specifically relevant and adapted to each event, and propose new methods to automatically generate multimedia event summary documents.
Specific research questions within this project could include:
– Measuring the visual relevance of words and entities;
– Determining relevant social multi-media data sources and sampling method;
– Filtering visual relations and labeling relation types;
– Generating natural language summaries for media-rich collections.