Children learning language in multilingual settings have to learn that objects take different labels within each different language to which they are exposed. Previous research has shown that adults can learn one-to-one and two-to-one word-object mappings via cross-situational statistical learning (CSSL), and that socio-pragmatic cues may differentially influence monolingual and bilingual adults’ learning of such mappings. However, the extent to which monolingual and bilingual learners can keep track of multiple labels from multiple speakers has not yet been investigated. We manipulated the number of speakers in a CSSL task that involved learning both mapping types. We successfully replicated previous studies that found that both monolinguals and bilinguals could learn both types of mappings via CSSL. In addition, we found that bilinguals showed a steeper learning rate for two-to-one mappings than monolinguals, and bilinguals were more likely to accept two words for the same object than monolinguals. These results show that the effect of speaker identity on tracking word-object mappings varies according to language experience.