Transitivity is perhaps the most fundamental axiom in economic models of choice. The empirical literature has regularly documented violations of transitivity, but these violations pose little problem if they are simply a result of somewhat-noisy decision making and not a reflection of the deterministic part of individuals’ preferences. However, what if transitivity violations reflect genuinely nontransitive preferences? And how can we separate nontransitive preferences from noise-generated transitivity violations–a problem that so far appears unresolved? To tackle these fundamental questions, we develop a theoretical framework which allows for nontransitive choices and behavioral noise. We then derive a non-parametric method which uses response times and choice frequencies to distinguish genuine (and potentially nontransitive) preferences from noise. We apply this method to two different datasets, demonstrating that a substantial proportion of transitivity violations reflect genuinely nontransitive preferences. These violations cannot be accounted for by any model using transitive preferences and noisy choices.