COMIQ

Abstract:Manga is a Japanese comic that has gained global recognition. It is a multimodal narrative format that integrates images and text in complex ways and presents challenges due to its complex multimodal nature. Over the years, numerous research efforts have been dedicated to the recognition and understanding of manga. In this talk, I will present the development journey of Manga109 -- the largest academic dataset for manga -- from its beginning to the present. Then, I will introduce our MangaUB, a benchmark dataset designed to evaluate the rapidly evolving capabilities of large multimodal models (LMMs) in comprehending manga. Furthermore, I will show our latest work, MangaLMM, a manga-specialized model finetuned from the open-source LMM. We evaluate LLMs reading manga in a double-page spread format, just as human readers do, using our carefully constructed visual question answering tasks MangaVQA. We show our assessments how well LMMs can understand manga content.

Abstract: With all the technologies that we have at our disposal nowadays, we can say that there is no comic book that is non-accessible on one side and accessible on the other. There is a progressive accessibility that needs to be implemented. Why? Because it's difficult. What needs to be done for the visually impaired, the blind, those with dyslexia, and those with motor disabilities are not the same kind of adaptations. Since we started partnership on accessible comics between academics, companies and adapting organisations, we always thought that we needed to work step by step, from short, easy comics to complex ones. We start by extracting the text from panels (for TTS) witch is still difficult in some very graphical pages, then we go into converting comic book images into text descriptions (textual adaptation) to allow screen readers to "read aloud and describe" the comics. Finally, we investigate how to automatically adapt them into multi-voice AI narrated audiobook by adding AI generated voices following character's traits and emotions, ambient sounds and playing sound effects 💬💥💨💢💤

Speaker Bio: Yael Vinker is a postdoctoral researcher at MIT CSAIL, working with Prof. Antonio Torralba. She completed her PhD at Tel Aviv University, advised by Prof. Daniel Cohen-Or and Prof. Ariel Shamir. Her research lies at the intersection of computer graphics, AI, and art. She develops computational systems that model and support visual thinking, visual communication, and creative expression. A core focus of her work is leveraging the rich priors of large vision-language models to generate visuals that are abstract, expressive, structured, and editable.

Abstract: Comics are a unique visual medium that construct meaning through the interplay of text, image, and time. Though they may appear simple, comics are rich in information, emotion, and subtle visual cues, making them a compelling domain for exploring the foundations of visual communication. In this talk, we will examine how principles from the world of comics illuminate key challenges in AI: How do visual symbols emerge and acquire meaning? What makes an abstract representation expressive? Why do schematic characters evoke stronger identification? Guided by insights from Scott McCloud's seminal book "Understanding Comics", we will explore how the language of comics can inspire new directions in AI research on visual abstraction and communication.