This paper proposes a method for identifying the optimal strategy for substituting pitchers in a baseball game played not under the Designated Hitter rule. Using a Markov Chain model we incorporate the effect of the defensive ability of pitchers using Earned Run Average (ERA) as a measure, and develop a dynamic programming formulation to evaluate the effect of pitcher substitution. The method is illustrated using a match based on the real line-ups of the Colorado Rockies and the San Francisco Giants in the National League of Major League Baseball. We show how this approach may help to determine when to make a substitution and how much the probability of winning increases if the optimal strategy is followed.