Deep artificial neural networks have been popular for time series forecasting literature in recent years. The recurrent neural networks present more suitable architectures for forecasting problems than other deep neural network types. The simplest deep recurrent neural network type is simple recurrent neural networks according to the number of employed parameters. These neural networks can be preferred to solve forecasting problems because of their simple structure if they are trained well. Unfortunately, the training of simple recurrent neural networks is problematic because of exploding or vanishing gradient problems. The contribution of this study is proposing a new training algorithm based on particle swarm optimization. The algorithm does not use gradients so it has not vanished or exploding gradient problem. The performance of the new training algorithm is compared with long short-term memory trained by the Adam algorithm and Pi-Sigma artificial neural network. In the applications, ten-time series are used to compare the performance of the methods. The ten-time series is consisting of daily observations of the Dow-Jones and Nikkei stock exchange opening prices between the years 2014 and 2018. At the end of the analysis processes, the proposed method produces more accurate forecast results than established benchmarks.