Unobtrusive indoor localization aims to localize people without requiring them to carry any devices or being actively involved with the localizing process. It underpins a wide range of applications including older people surveillance, intruder detection and indoor navigation. However, in a residential home, the Received Signal Strength Indicator (RSSI) is heavily obstructed by furniture or domestic appliances, reducing the localization accuracy. This environment is important to observe as human-object interaction (HOI) events, detected by pervasive sensors, can reveal people's interleaved locations during daily living activities. Thus, this paper aims to enhance the performance of the RFID-based localization system by fusing human-object interactions. Specifically, we propose a general Bayesian probabilistic multi-sensor fusion framework to integrate both RSSI signals and human-object interaction events to infer the most likely location and trajectory. Unlike other RFID-based unobtrusive localization systems, which are limited to deployment and testing in cleared spacial areas, our system can work in a furnished environment. The extensive experiments with this system have a localization accuracy up to 96.7%, and average 0.58m tracking error.