Fiber reinforced concrete (FRC) has improved strength and ductility, making it suitable for a wider range of engineering structures. The development of a constitutive model is crucial for analyzing mechanical behavior of these structures, while this still remains challenging as the complex composition of FRC makes it difficult to formulate explicit relationships among a range of critical material parameters. With the latest development of data and digital technologies, data-driven approaches have emerged as a powerful alternative that are capable of solving advanced and complex engineering problems. Developing data-driven methods based on objective data analysis and decision-making to predict mechanical behavior of engineering structures is an important direction of such development. In this paper, a uniaxial constitutive model of fiber-reinforced concrete is developed by a physics-based data-driven framework. The framework consists of three important parts, including construction of experimental database, parameters calibration of physical models, and implementation of neural network. From the experimental data collected from published literature, an experimental database is built first. A physical model is proposed by modifying the uniaxial constitute model for normal concrete, so that it is more convenient and straightforward to consider the effect of fibers on fiber reinforced concrete subjected to compression, tension and repeat loading conditions. The parameters of the model are calibrated against experimental data by the swarm intelligence optimization algorithms. With the calibrated parameters of the physical model, a Fully Connected Neural Network (FCNN) is trained to be used to predict the physical parameters of fiber reinforced concrete. By comparing with independent experimental data, the proposed fiber-reinforced concrete uniaxial constitutive model constructed using the physics-based data-driven framework can accurately predict the stress–strain relationships of a range of FRC, which suggests that the FRC material model can be used in the numerical simulation and design of FRC components and structures. In addition, the proposed model is applicable to multiple loading conditions.