In this paper, we describe a framework for learning-based image enhancement. At the core of our algorithm lies a generic regularization framework that comprises a prior on natural images, as well as an application-specific conditional model based on Gaussian processes. In contrast to prior learning-based approaches, our algorithm can instantly learn task-specific degradation models from sample images which enables users to easily adopt the algorithm to a specific problem and data set of interest. This is facilitated by our
efficient approximation scheme of large-scale Gaussian processes. We demonstrate the efficiency and effectiveness of our approach by applying it to two example enhancement applications: single-image super-resolution as well as artifact removal in JPEG-encoded images.