We report the influence of the matrix in InAs nanostructures grown on InP[001] substrates by solid source molecular beam epitaxy. Three buffers of InGaAs, InAlAs, and InP lattice matched on InP have been studied. Differences in nanostructure morphology have been evaluated by atomic force microscopy and by low-temperature photoluminescence measurements. We have observed quantum wire formation in the InAs/InGaAs and InAs/InAlAs system, while quantum dots with bimodal size distribution appeared in the InAs/InP system. The observed drastic modification of nanostructures is attributed to the anisotropic buffer layer surface, and the bimodal size distribution of InAs quantum dots on InP buffer is simply explained by the surface mass transfer.