The buildout of private Wide Area Networks (WANs) by cloud providers allows providers to extend their network to more locations and establish direct connectivity with end user Internet Service Providers (ISPs). Tenants of the cloud providers benefit from this proximity to users, which is supposed to provide improved performance by bypassing the public Internet. However, the performance impact of private WANs is not widely understood. To isolate the impact of a private WAN, we measure from globally distributed vantage points to a large cloud provider, comparing performance when using its worldwide WAN and when forcing traffic to instead use the public Internet. The benefits are not universal. While 40% of our vantage points saw improved performance when using the WAN, half of our vantage points did not see significant performance improvement, and 10% had better performance over the public Internet. We find that the benefits of the private WAN tend to improve with client-to-server distance, but that the benefits (or drawbacks) to a particular vantage point depend on specifics of its geographic and network connectivity.