Utilizing scattering theory, we quantify the consequences of physical constraints that limit the visibility of non-Hermitian effects in passive devices. The constraints arise from the fundamental requirement that the system obeys causality, and can be captured concisely in terms of an internal time-delay operator, which furthermore provides a direct quantitative measure of the visibility of specific non-Hermitian phenomena in the density of states. We illustrate the implications by contrasting two prominent non-Hermitian effects, exceptional points, and the non-Hermitian skin effect, whose underlying extreme mode nonorthogonality turns out to be undetectable in the density of states.