We address the problem of developing a well-performing and implementable scheduler of users with wireless connections to the central controller, which arise in areas such as mobile data networks, heterogeneous networks, or vehicular communications systems. The main feature of such systems is that the connection quality of each user is time-varying, resulting in time-varying transmission rate corresponding to available channel states. We assume that this evolution is Markovian, relaxing the common but unrealistic assumption of stationary channels. We first focus on the three-state channel and study the optimal policy, showing that threshold policies (of giving higher priority to users with higher transmission rate) are not necessarily optimal. For the general channel we design a scheduler which generalizes the recently proposed Potential Improvement (PI) scheduler, and propose its two practical approximations, whose performance is analyzed and compared to existing alternative schedulers in a variety of simulation scenarios. We suggest and give evidence that the variant of PI which only relies on the steady-state distribution of the channel, performs extremely well, and therefore should be used for practical implementation.