Vision and Language

We, people, obtain a lot of information from vision and can associate what we see to natural language text in, e.g., Japanese and English, to describe and communicate the information. Vision and Language is a research area to implement this human ability with a machine, which lies in the intersection of Computer Vision, Pattern Recognition, Natural Language Processing, and Machine Learning. We are working on various tasks involving images, video, and natural language text, including image captioning and visual question answering using deep neural networks.

Deep Neural Networks and Bias

Currently, deep neural networks are an indispensable tool for many research fields, like Computer Vision, Pattern Recognition, and Natural Language Processing. On the other hand, they are prone to produce unfair outputs for, e.g., different races and genders. This bias in deep neural networks stems from spurious correlations and confounders (i.e., some irrelevant factors that affect to model outputs) in the dataset used for training models. Our group aims at finding and reducing such bias in neural networks.

Explainable AI

Deep neural networks learn what to see to solve the task from the dataset. This has boosted the performance of various tasks but at the same time causes a new problem, the “black box”-ness of neural networks, which means that the users cannot tell what a neural network sees. This problem is critical for some applications, like medicine. Explainable AI is a research field to solve the “black box” problem. We are exploring a new paradigm of Explainable AI that discovers concepts or a “language” to describe data through training.