The aim is to create an annotated dataset of human-to-human dialogue in Youtube cooking videos*, that can serve as a resource for training ML models to generate conversational explanations of the cooking process. This involves the identification of videos with multiple speakers, speaker diarization (partitioning audio and/or transcript according to speaker identity), identification of conversational interaction between the speakers, and investigating if these interactions qualify as ‘conversational explanations’ of the video content

Contact: Ray Kodali

Relevant literature:

Speaker diarization: https://arxiv.org/pdf/2101.09624.pdf
Potential videos: http://youcook2.eecs.umich.edu/explore
Background on ‘conversational explanations’ from an XAI perspective: https://arxiv.org/pdf/1706.07269.pdf (Sec. 5) Note that in this project, we focus on ‘explaining’ the video content rather than model predictions.

*We focus on the process of cooking as there is some related ongoing work at DFKI, but other instructional scenarios are possible.

Programming in Python, ideally experience with processing video and audio data