TY - GEN

T1 - A Syntactic Justification for Occam's razor

AU - Woodward, John

AU - Evans, Andrew

AU - Dempster, Paul

PY - 2008/10/31

Y1 - 2008/10/31

N2 - Informally, Occam’s razor states, “Given two hypotheses whichequally agree with the observed data, choose the simpler”, andhas become a central guiding heuristic in the empirical sciencesand in particular machine learning. We criticize previousarguments for the validity of Occam’s razor.The nature of hypotheses spaces is explored and we observe acorrelation between the complexity of a concept yielded by ahypothesis and the frequency with which it is represented whenthe hypothesis space is uniformly sampled. We argue that there isnot a single best hypothesis but a set of hypotheses which giverise to the same predictions (i.e. the hypotheses are semanticallyequivalent), whereas Occam’s razor suggests there is a single besthypothesis. We prefer one set of hypotheses over another setbecause it is the larger set (and therefore the most probable) andthe larger set happens to contain the simplest consistenthypothesis. This gives the appearance that simpler hypothesesgeneralize better. Thus, the contribution of this paper is thejustification of Occam’s razor by a simple counting argument.

AB - Informally, Occam’s razor states, “Given two hypotheses whichequally agree with the observed data, choose the simpler”, andhas become a central guiding heuristic in the empirical sciencesand in particular machine learning. We criticize previousarguments for the validity of Occam’s razor.The nature of hypotheses spaces is explored and we observe acorrelation between the complexity of a concept yielded by ahypothesis and the frequency with which it is represented whenthe hypothesis space is uniformly sampled. We argue that there isnot a single best hypothesis but a set of hypotheses which giverise to the same predictions (i.e. the hypotheses are semanticallyequivalent), whereas Occam’s razor suggests there is a single besthypothesis. We prefer one set of hypotheses over another setbecause it is the larger set (and therefore the most probable) andthe larger set happens to contain the simplest consistenthypothesis. This gives the appearance that simpler hypothesesgeneralize better. Thus, the contribution of this paper is thejustification of Occam’s razor by a simple counting argument.

UR - http://scotek.org/pubs/occamNKS.pdf

M3 - Conference contribution/Paper

BT - 2008 Midwest, A New Kind of Science Conference

T2 - 2008 Midwest, A New Kind of Science Conference

Y2 - 31 October 2008 through 2 November 2008

ER -