Although both widely used in the financial industry, there is quite often very little justification why GARCH or stochastic volatility is preferred over the other in practice. Most of the relevant literature focuses on the comparison of the fit of various volatility models to a particular data set, which sometimes may be inconclusive due to the statistical similarities of both processes. With an ever growing interest among the financial industry in the risk of extreme price movements, it is natural to consider the selection between both models from an extreme value perspective. By studying the dependence structure of the extreme values of a given series, we are able to clearly distinguish GARCH and stochastic volatility models and to test statistically which one better captures the observed tail behaviour. We illustrate the performance of the method using some stock market returns and find that different volatility models may give a better fit to the upper or lower tails.