In D-term inflation models, Affleck-Dine baryogenesis produces baryonic isocurvature fluctuations. In most cases the Affleck-Dine condensate is unstable with respect to collapse to B-balls, which can transform the baryon number perturbations into perturbations in the number of dark matter neutralinos. The requirement that the deviation of the adiabatic perturbations from scale invariance is not too large imposes a lower bound on the magnitude of the isocurvature fluctuations. In general this is larger than about 10(-4) times the adiabatic perturbation, and, for the particular case of late decaying B-balls, larger than about 10(-2) times the adiabatic perturbation, which should be observable by MAP and PLANCK.