The classic approach in robust optimization is to optimize the solution with respect to the worst case scenario. This pessimistic approach yields solutions that perform best if the worst scenario happens, but also usually perform bad for an average case scenario. On the other hand, a solution that optimizes the performance of this average case scenario may lack in the worst-case performance guarantee. In practice it is important to find a good compromise between these two solutions. We propose to deal with this problem by considering it from a bicriteria perspective. The Pareto curve of the bicriteria problem visualizes exactly how costly it is to ensure robustness and helps to choose the solution with the best balance between expected and guaranteed performance. In this paper we consider linear programming problems with uncertain cost functions. Building upon a theoretical observation on the structure of Pareto solutions for these problems, we present a column generation approach that requires no direct solution of the computationally expensive worst-case problem. In computational experiments we demonstrate the effectiveness of both the proposed algorithm, and the bicriteria perspective in general.