Accepted author manuscript, 268 KB, PDF document
Available under license: CC BY-NC-ND: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License
Final published version
Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review
A Syntactic Justification for Occam's razor. / Woodward, John; Evans, Andrew; Dempster, Paul.
2008 Midwest, A New Kind of Science Conference. 2008.Research output: Contribution in Book/Report/Proceedings - With ISBN/ISSN › Conference contribution/Paper › peer-review
}
TY - GEN
T1 - A Syntactic Justification for Occam's razor
AU - Woodward, John
AU - Evans, Andrew
AU - Dempster, Paul
PY - 2008/10/31
Y1 - 2008/10/31
N2 - Informally, Occam’s razor states, “Given two hypotheses whichequally agree with the observed data, choose the simpler”, andhas become a central guiding heuristic in the empirical sciencesand in particular machine learning. We criticize previousarguments for the validity of Occam’s razor.The nature of hypotheses spaces is explored and we observe acorrelation between the complexity of a concept yielded by ahypothesis and the frequency with which it is represented whenthe hypothesis space is uniformly sampled. We argue that there isnot a single best hypothesis but a set of hypotheses which giverise to the same predictions (i.e. the hypotheses are semanticallyequivalent), whereas Occam’s razor suggests there is a single besthypothesis. We prefer one set of hypotheses over another setbecause it is the larger set (and therefore the most probable) andthe larger set happens to contain the simplest consistenthypothesis. This gives the appearance that simpler hypothesesgeneralize better. Thus, the contribution of this paper is thejustification of Occam’s razor by a simple counting argument.
AB - Informally, Occam’s razor states, “Given two hypotheses whichequally agree with the observed data, choose the simpler”, andhas become a central guiding heuristic in the empirical sciencesand in particular machine learning. We criticize previousarguments for the validity of Occam’s razor.The nature of hypotheses spaces is explored and we observe acorrelation between the complexity of a concept yielded by ahypothesis and the frequency with which it is represented whenthe hypothesis space is uniformly sampled. We argue that there isnot a single best hypothesis but a set of hypotheses which giverise to the same predictions (i.e. the hypotheses are semanticallyequivalent), whereas Occam’s razor suggests there is a single besthypothesis. We prefer one set of hypotheses over another setbecause it is the larger set (and therefore the most probable) andthe larger set happens to contain the simplest consistenthypothesis. This gives the appearance that simpler hypothesesgeneralize better. Thus, the contribution of this paper is thejustification of Occam’s razor by a simple counting argument.
UR - http://scotek.org/pubs/occamNKS.pdf
M3 - Conference contribution/Paper
BT - 2008 Midwest, A New Kind of Science Conference
T2 - 2008 Midwest, A New Kind of Science Conference
Y2 - 31 October 2008 through 2 November 2008
ER -