TUTCRIS - Tampereen teknillinen yliopisto


Multi-modal dense video captioning



OtsikkoProceedings - 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2020
ISBN (elektroninen)9781728193601
ISBN (painettu)978-1-7281-9361-8
DOI - pysyväislinkit
TilaJulkaistu - 2020
OKM-julkaisutyyppiA4 Artikkeli konferenssijulkaisussa
TapahtumaIEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops - Virtual, Online, Yhdysvallat
Kesto: 14 kesäkuuta 202019 kesäkuuta 2020


NimiIEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
ISSN (painettu)2160-7508
ISSN (elektroninen)2160-7516


ConferenceIEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops
KaupunkiVirtual, Online


Dense video captioning is a task of localizing interesting events from an untrimmed video and producing textual description (captions) for each localized event. Most of the previous works in dense video captioning are solely based on visual information and completely ignore the audio track. However, audio, and speech, in particular, are vital cues for a human observer in understanding an environment. In this paper, we present a new dense video captioning approach that is able to utilize any number of modalities for event description. Specifically, we show how audio and speech modalities may improve a dense video captioning model. We apply automatic speech recognition (ASR) system to obtain a temporally aligned textual description of the speech (similar to subtitles) and treat it as a separate input alongside video frames and the corresponding audio track. We formulate the captioning task as a machine translation problem and utilize recently proposed Transformer architecture to convert multi-modal input data into textual descriptions. We demonstrate the performance of our model on ActivityNet Captions dataset. The ablation studies indicate a considerable contribution from audio and speech components suggesting that these modalities contain substantial complementary information to video frames. Furthermore, we provide an in-depth analysis of the ActivityNet Caption results by leveraging the category tags obtained from original YouTube videos. Code is publicly available: github.com/v-iashin/MDVC.