The HTTP response header “Vary: User-Agent” plays an important role in how your website cache works. When set correctly in Varnish Cache, it will ensure that your visitors see the correct content, regardless of whether caching is applied or not. If set up incorrectly, however, it can have a negative impact on the caching hit rate. If you want to improve your cache hit rate and user experience, it is worth considering, but be sure to use it with great care.
The Role of Vary: User-Agent
If your Vary header has User-Agent, Cookie, Referer or *(wildcard) as a value, this in turn can reduce the effect of your caching system. The most commonly used value is User-Agent. It tells Dynamic caching which version of your page should be served depending on the browser type, which can prevent a cached desktop version being served to a mobile visitor. However, it has become a bit outdated since most sites today don’t serve different HTML for their mobile sites; instead CSS typically performs this function. Unless you are using a specific plugin such as WP Touch or you definitely have a difference in the HTML output of your site, you are unlikely to need to send the Vary: User-Agent header.
How Vary: User-Agent Can Negatively Impact Cache Hits
This is important as the use of Vary: User-Agent can negatively impact your site because it uses up resources. The User-Agent HTTP head has an extremely wide range of values, which effectively splits the cache into many different buckets and can significantly lower the chance for a cache hit.
Given that there are thousands of User-Agent strings, any site with a reasonable amount of traffic will see thousands of different User-Agent strings per hour. Platform-wide, using the Section HTTP logs, we see approximately 25,000 variations of User-Agent every hour. These requests can create a resource-intensive strain and the information contained in these user-agent requests, while widely ranging, typically has no or little impact on the asset the origin generates. The User-Agent header, for example, will indicate which OS or browser version is issuing the request and different values of the header will lead to different buckets in the cache.
If the Vary: User-Agent header is present, for example, the following four different User-Agent values will result in four different buckets of cache:
- Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
- Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36
When requesting an asset from the web server using this set of User-Agents, the same URL would be identical irrespective of the User-Agent header value. Thus, it isn’t necessary to use the User-Agent parameter as a Vary header.
How to Improve the Cache Hit Rate
If the content of an asset does differ based on User-Agent, to minimize cache buckets, you can bucket into similar groups; for example, a common set of buckets we see at Section is Desktop, Mobile Large Screen, Mobile Small Screen, and Other. If you were to use these buckets, the four User-Agents in the example above can all be served from the “Desktop” bucket. By decreasing the number of requests on the cache, you can increase the cache hit rate. When the Vary: User-Agent header is used correctly, we often see cache hit rate increases of 50% or more.
While Vary: User-Agent can play an important role when necessary, ensure it is set up correctly to not create an overuse of your resources. By implementing it properly in the header, you can increase your overall cache hit rate, which in turn leads to a reduced load/data on origin web application servers and will improve user experience.
