Early action prediction aims to successfully predict the class
label of an action before it is completely performed. This is a challenging
task because the beginning stages of different actions can be very similar,
with only minor subtle differences for discrimination. In this paper,
we propose a novel Expert Retrieval and Assembly (ERA) module that
retrieves and assembles a set of experts most specialized at using discriminative
subtle differences, to distinguish an input sample from other
highly similar samples. To encourage our model to effectively use subtle
differences for early action prediction, we push experts to discriminate
exclusively between samples that are highly similar, forcing these experts
to learn to use subtle differences that exist between those samples.
Additionally, we design an effective Expert Learning Rate Optimization
method that balances the experts’ optimization and leads to better performance.
We evaluate our ERA module on four public action datasets
and achieve state-of-the-art performance.