Device-free indoor localization aims to localize people without requiring them to carry any devices or being actively involved in the localizing process. It underpins a wide range of applications including older people surveillance, intruder detection and indoor navigation. However, in a cluttered environment such as a residential home, the Received Signal Strength Indicator (RSSI) is heavily obstructed by furniture or metallic appliances, thus reducing the localization accuracy. This environment is important to observe as human-object interaction (HOI) events, detected by pervasive sensors, can potentially reveal people's interleaved locations during daily living activities, such as watching TV, opening the fridge door. This paper aims to enhance the performance of commercial off-the-shelf (COTS) RFID-based localization system by leveraging HOI contexts in a furnished home. Specifically, we propose a general Bayesian probabilistic framework to integrate both RSSI signals and HOI events to infer the most likely location and trajectory. Experiments conducted in a residential house demonstrate the effectiveness of our proposed method, in which we can localize a resident with average 95% accuracy and track a moving subject with 0.58m mean error distance.