With gesture-based interactions in mobile settings becoming more popular, there is a growing concern regarding the social acceptance of these interaction techniques. In this paper we begin by examining the various definitions of social acceptance that have been proposed in the literature to synthesize a definition that is based on how the user feels about performing a particular interaction as well as how the bystanders perceive the user during this interaction. We then present the main factors that influence gestures' social acceptance including culture, time, interaction type and the user's position on the innovation adoption curve. Through a user study we show that an important factor in determining social acceptance of gesture-based interaction techniques is the user's perception of others ability to interpret the potential effect of a manipulation.