Device-free passive localization aims to localize or track targets without requiring them to carry any devices or to be actively involved with the localization process. This technique has received much attention recently in a wide range of applications including elderly people surveillance, intruder detection, and indoor navigation. In this paper, we propose a novel localization and tracking system based on the Received Signal Strength field formed by a set of cost-efficient passive RFID tags. We firstly formulate localization as a classification task, where we compare several state-of-theart learning-based classification methods including k Nearest Neighbor (kNN), Multivariate Gaussian Mixture Model (GMM) and Support Vector Machine (SVM). To track a moving subject, we propose two HiddenMarkovModel (HMM)- based methods, namely GMM-based HMM and kNNbased HMM. kNN-based HMM extends kNN into a probabilistic style to approximate the Emission Probability Matrix in HMM. The proposed methods can be easily applied into other fingerprint-based tracking systems regardless of their hardware platforms. We conduct extensive experiments and the results demonstrate the effectiveness and accuracy of our approaches with up to 98% localization accuracy and an average of 0.7m tracking error.