Home > Research > Publications & Outputs > Generating feature spaces for linear algorithms...


Text available via DOI:

View graph of relations

Generating feature spaces for linear algorithms with regularized sparse kernel slow feature analysis

Research output: Contribution to Journal/MagazineJournal articlepeer-review

<mark>Journal publication date</mark>10/2012
<mark>Journal</mark>Machine Learning
Issue number1
Number of pages20
Pages (from-to)67-86
Publication StatusPublished
Early online date13/06/12
<mark>Original language</mark>English


Without non-linear basis functions many problems can not be solved by linear algorithms. This article proposes a method to automatically construct such basis functions with slow feature analysis (SFA). Non-linear optimization of this unsupervised learning method generates an orthogonal basis on the unknown latent space for a given time series. In contrast to methods like PCA, SFA is thus well suited for techniques that make direct use of the latent space. Real-world time series can be complex, and current SFA algorithms are either not powerful enough or tend to over-fit. We make use of the kernel trick in combination with sparsification to develop a kernelized SFA algorithm which provides a powerful function class for large data sets. Sparsity is achieved by a novel matching pursuit approach that can be applied to other tasks as well. For small data sets, however, the kernel SFA approach leads to over-fitting and numerical instabilities. To enforce a stable solution, we introduce regularization to the SFA objective. We hypothesize that our algorithm generates a feature space that resembles a Fourier basis in the unknown space of latent variables underlying a given real-world time series. We evaluate this hypothesis at the example of a vowel classification task in comparison to sparse kernel PCA. Our results show excellent classification accuracy and demonstrate the superiority of kernel SFA over kernel PCA in encoding latent variables.