Video has become an essential medium for learning. However, there are challenges when using traditional methods to measure how learners attend to lecture videos in video learning analytics, such as difficulty in capturing learners' attention at a fine-grained level. Therefore, in this paper, we propose a gaze-based metric - "with-me-ness direction"that can measure how learners' gaze-direction changes when they listen to the instructor's dialogues in a video-lecture. We analyze the gaze data of 45 participants as they watched a video lecture and measured both the sequences of with-me-ness direction and proportion of time a participant spent looking in each direction throughout the lecture at different levels. We found that although the majority of the time participants followed the instructor's dialogues, their behaviour of looking-ahead, looking-behind or looking-outside differed by their prior knowledge. These findings open the possibility of using eye-tracking to measure learners' video-watching attention patterns and examine factors that can influence their attention, thereby helping instructors to design effective learning materials.