Let The Machine Translate The Manga!

3つの要点
✔️ マルチモーダルな文脈を意識した漫画翻訳
✔️ 最初の(多言語)漫画翻訳ベンチマーク&データセット
✔️ 初の画像から画像への完全自動化マンガ翻訳フレームワーク

Towards Fully Automated Manga Translation
written by Ryota Hinami, Shonosuke Ishiwatari, Kazuhiko Yasuda, Yusuke Matsui
(Submitted on 28 Dec 2020 (v1), last revised 9 Jan 2021 (this version, v3))
Comments: Accepted to AAAI2021
Subjects: Computation and Language (cs.CL)
   

Introduction 

Japanese cartoons (and webtoon: Korean, manhua: Chinese) are popular all over the world, and translating them manually line by line into other languages is a resource-intensive task. While there has been significant progress in the development of neural machine translation (NMT) models (Deepl & Google), their application to manga translation has been relatively limited. This is mainly due to the fact that text and images are mixed in cartoons. Also, the text boxes are not perfectly aligned and the reading order is not always consistent. Furthermore, some Japanese words can be translated as he, him, she, or her depending on the context. Therefore, without contextual information from the manga panels, an accurate translation may not be possible. In this paper, we present a novel context-aware multimodal translation system for manga translation to solve these problems. It also serves as the first benchmark new dataset for manga translation.

Thapa Samrat

I am a second year international student from Nepal who is currently studying at the Department of Electronic and Information Engineering at Osaka University. I am interested in machine learning and deep learning. So I write articles about them in my spare time.