TY - JOUR

T1 - Experts Collaboration Learning for Continual Multi-Modal Reasoning

AU - Xu, Li

AU - Liu, Jun

PY - 2023/12/31

Y1 - 2023/12/31

N2 - Multi-modal reasoning, which aims to capture logical and causal structures in visual content and associate them with cues from other modality inputs (e.g., texts) to perform various types of reasoning, is an important research topic in artificial intelligence (AI). Existing works for multi-modal reasoning mainly exploit offline learning, where the training samples of all types of reasoning tasks are assumed to be available at once. Here we focus on continual learning for multi-modal reasoning (i.e., continual multi-modal reasoning), where the model is required to continuously learn to solve novel types of multi-modal reasoning tasks in a lifelong fashion. Continual multi-modal reasoning is challenging since the model needs to be able to effectively learn various types of new reasoning tasks, meanwhile avoiding forgetting. Here we propose a novel brain-inspired exp erts co llaboration network (Expo), which incorporates multiple learning blocks (experts). When encountering a new task, our network dynamically assembles and updates a set of task-specific experts that are most relevant to learning the current task, by either utilizing learned experts or exploring new experts. This thus enables effective learning of new tasks, and meanwhile consolidates previously learned reasoning skills. Moreover, to automatically find optimal task-specific experts, an effective experts selection strategy is designed. Extensive experiments demonstrate the efficacy of our model for continual multi-modal reasoning.

AB - Multi-modal reasoning, which aims to capture logical and causal structures in visual content and associate them with cues from other modality inputs (e.g., texts) to perform various types of reasoning, is an important research topic in artificial intelligence (AI). Existing works for multi-modal reasoning mainly exploit offline learning, where the training samples of all types of reasoning tasks are assumed to be available at once. Here we focus on continual learning for multi-modal reasoning (i.e., continual multi-modal reasoning), where the model is required to continuously learn to solve novel types of multi-modal reasoning tasks in a lifelong fashion. Continual multi-modal reasoning is challenging since the model needs to be able to effectively learn various types of new reasoning tasks, meanwhile avoiding forgetting. Here we propose a novel brain-inspired exp erts co llaboration network (Expo), which incorporates multiple learning blocks (experts). When encountering a new task, our network dynamically assembles and updates a set of task-specific experts that are most relevant to learning the current task, by either utilizing learned experts or exploring new experts. This thus enables effective learning of new tasks, and meanwhile consolidates previously learned reasoning skills. Moreover, to automatically find optimal task-specific experts, an effective experts selection strategy is designed. Extensive experiments demonstrate the efficacy of our model for continual multi-modal reasoning.

U2 - 10.1109/TIP.2023.3310336

DO - 10.1109/TIP.2023.3310336

M3 - Journal article

VL - 32

SP - 5087

EP - 5098

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

ER -