Modelling irregularly shaped aggregates in concrete uses advanced algorithms to describe complex geometric profile of, e.g, crushed gravels. Consequently, discretization of the complex geometry and computational convergence of the resulting mesh scheme are the bottlenecks that prevent wider application of mesoscopic fracture simulation of concrete with more realistic aggregates. In this paper, a novel modelling framework with simple pre-processing and good computational convergence is proposed based on the diffuse meshing technique and the coupled elasto-viscoplastic damage model. The proposed model discretizes concrete using only regular elements that may have different material components such as aggregates, mortar matrix and aggregate/matrix interface. The accuracy of the proposed model is validated by comparing with the full three-phase model that meshes aggregate, mortar and their interfaces separately. The proposed model is applied then to investigate the failure mechanism of concrete under uniaxial compression and tension. Furthermore, the effect of using simplified aggregate meso-structure is studied by comparing the results of using the complex shaped aggregate models and the spherical shaped aggregate model. The results show that the morphology of aggregate has a no-negligible influence on post-peak mechanical behavior, the orientation of aggregate affects crack initiation and propagation, and self-restraint stresses reduce the strength of concrete.