Informally, Occam’s razor states, “Given two hypotheses which
equally agree with the observed data, choose the simpler”, and
has become a central guiding heuristic in the empirical sciences
and in particular machine learning. We criticize previous
arguments for the validity of Occam’s razor.
The nature of hypotheses spaces is explored and we observe a
correlation between the complexity of a concept yielded by a
hypothesis and the frequency with which it is represented when
the hypothesis space is uniformly sampled. We argue that there is
not a single best hypothesis but a set of hypotheses which give
rise to the same predictions (i.e. the hypotheses are semantically
equivalent), whereas Occam’s razor suggests there is a single best
hypothesis. We prefer one set of hypotheses over another set
because it is the larger set (and therefore the most probable) and
the larger set happens to contain the simplest consistent
hypothesis. This gives the appearance that simpler hypotheses
generalize better. Thus, the contribution of this paper is the
justification of Occam’s razor by a simple counting argument.