Vision

A journalist starts editing a news article about the unrest in Egypt. The multimodal interpretation platform identifies during the editing process which multimedia items might be interesting for the journalist and offers a list of items categorized by content type (e.g. video, audio, text), relevance, up-to-dateness etc.

At the same time the platform considers the media and discussion agenda on the Social Web (e.g. Twitter trending topic). Through this additional analysis the journalist can consider more than one perspective in his article and contextualise the news article with the relevant multimodal information. Use case #1 addresses specific needs of today‟s multimedia journalists working in fully digitized newsrooms and publishing on multi-platform (web, mobile, Smart TV etc.) content distribution environments. By their very nature the workflows and tools involved need to incorporate multimedia items (mainly but not limited to text, images, audio, video).

The EUMSSI platform, thanks to its multimodal analysis and interpretation capabilities, facilitates real – time curation of the digital assets through automatic semantic annotation, augmenting, at the same time, the knowledge database with social media documents, conveniently interrelated and annotated.

In addition, a multimodal content-based recommendation system built on top of this platform empowers the journalist to monitor and gather up-to-date multimedia and social media documents related with his investigation without the need of reviewing an enormous amount of insufficiently annotated records.

With the help of the platform, he can perform his job in an optimized manner, thus increasing productivity and the quality of his work.

There is an increasing popularity of video on-demand. On the one hand, users want to choose the programs they like. On the other hand, they also like some guidance to find out ‘what’s on, what’s new, what’s cool, what’s for me’. Today’s TV guide and program should be personalized, based on user‟s preferences, user’s interests, daily routines (morning programs, evening programs) as well as importance or hype (such as breaking news or national events). Moreover, in order for the old TV medium to remain relevant, it should become more interactive. Many people already have an iPad or tablet at hand when watching television programs. This device can effectively function as a second screen, on which to exchange comments via Facebook or chat with a friend about the program, but also the two screens could be connected in such a way that the tablet provides background information on a given program, personalized news tickers, suggest what to watch next, and so on.

The EUMSSI platform makes it possible the implementation of a personalised multimodal content-based recommendation system able to make relevant suggestions of multimedia content to the end-user based on what she has watched, what other people have watched, or are watching, combined with multimedia contents related in some ways to what she is actually watching and what are the people saying about these contents in the social networks.

Both use-cases have a common technical base in that they exploit (multimodal) content-based recommendation algorithms but they differ in input/output aspects, in addition to the type of user, as mentioned above. On the one hand, in the case of the Contextualising tool, the input to the recommendation system is less ���controlled”, as it consists in incomplete textual input that is being created on the spot, but the output result needs not be as precise, since it is ���������raw material” for the journalist to process. On the other hand, in the case of the Second screen, the input is much more controlled since it is an already semantically annotated document from the internal database (DWs), but the output results, necessarily need to be more accurate for the end-user.