We study the impact of stochastic noise on the generation of primordial black hole (PBH) seeds in ultra-slow-roll (USR) inflation with numerical simulations. We consider the non-linearity of the system by consistently taking into account the noise dependence on the inflaton perturbations, while evolving the perturbations on the coarse-grained background affected by the noise. We capture in this way the non-Markovian nature of the dynamics, and demonstrate that non-Markovian effects are subleading. Using the ΔN formalism, we find the probability distribution P(R) of the comoving curvature perturbation R. We consider inflationary potentials that fit the CMB and lead to PBH dark matter with i) asteroid, ii) solar, or iii) Planck mass, as well as iv) PBHs that form the seeds of supermassive black holes. We find that stochastic effects enhance the PBH abundance by a factor of O(10)-o O(108), depending on the PBH mass. We also show that the usual approximation, where stochastic kicks depend only on the Hubble rate, either underestimates or overestimates the abundance by orders of magnitude, depending on the potential. We evaluate the gauge dependence of the results, discuss the quantum-to-classical transition, and highlight open issues of the application of the stochastic formalism to USR inflation.