Natural regeneration of forests has significantly led to increased native forest cover in some regions. Several studies have explored the spatial drivers of forest cover increase, yet little is known about their effects on forest structure and species richness and diversity. We quantified the effects of local (forest age, remnant Eucalyptus basal area, slope, soil fertility, and clay content) and landscape drivers (surrounding land use, distance from streams, and surrounding forest cover and its change over time) on the aboveground biomass, species density, and phylogenetic diversity of native trees in second-growth forests. We sampled 44 naturally regenerating forests established on former pastures and abandoned Eucalyptus plantations for 11–46 years in agricultural landscapes of the southeastern Atlantic Forest, Brazil. We used generalized linear mixed effect models to quantify the effect drivers on forest attributes. While only Eucalyptus basal area and proximity to sugarcane plantations had a consistent negative effect on forest biomass, other drivers were among the best models to explain forest attributes, but their effect was variable. Age increased tree biomass but the effect was not consistent. Similarly, species richness and phylogenetic diversity were mainly affected by landscape drivers such as surrounding forest cover. In tropical agricultural landscapes, effective forest recovery requires more than just time, as forest age can be less important in determining forest attributes than human land uses and surrounding native forest cover. Crucially, forest recovery can be improved through interventions and policies that promote less impactful human activities and enhance existing forest cover. Abstract in Portuguese is available with online material.