Studying local variations in the Seebeck coefficient of materials is important for understanding and optimizing their thermoelectric properties, yet most thermoelectric measurements are global over a whole device or material, thus overlooking spatial divergences in the signal and the role of local variation and internal structure. Such variations can be caused by local defects, metallic contacts or interfaces that often substantially influence thermoelectric properties, especially in two dimensional materials. Here, we demonstrate scanning thermal gate microscopy, a non-destructive method to obtain high resolution 2-dimensional maps of the thermovoltage, to study graphene samples. We demonstrate the efficiency of this newly developed method by measuring local Seebeck coefficient in a graphene ribbon and in a junction between single-layer and bilayer graphene.