What is a 404 error, how does Varnish Cache handle them by default and how can it handle them? We also explore the cost of 404 Errors in this article: Cost of 404 Not Found.
What is a 404 Error?
The 404 error in Varnish Cache can be a frustrating one as the status-line 404 means ‘File Not Found’. Cached data takes up space on your device so the 404 response being cached on your computer, smartphone or tablet can be an irritation as it essentially represents a waste of space.
A 404 error occurs when data hasn’t been updated in a timely manner. By default, Varnish Cache holds onto 404 responses along with other cached content. This is problematic since 404 responses are firstly trivial in relation to their file size and secondly, typically occur because of a mistake, which will usually be quickly resolved.
While the systems in place are updating, for instance, there may be two backend servers both serving the same data. However, one server may be slower than the other. When a request comes in to the server that is slow, it will throw a 404 Not Found.
Default Varnish Cache Behaviour
The following status codes will be cached by default in Varnish Cache: 200: OK 203: Non-Authoritative Information 300: Multiple Choices 301: Moved Permanently 302: Moved Temporarily 304: Not modified 307: Temporary Redirect 410: Gone 404: Not Found
Source: http://book.varnish-software.com/4.0/chapters/VCL_Basics.html
What Should Happen when Varnish Cache sees a 404 Error?
As seen above, Varnish Cache holds onto 404 responses along with other cached content by default. However, you can modify the settings to prevent 404 responses being cached at all, or change them so that they are only cached for a short window of time. If you stop Varnish Cache from caching 404 errors, this means that all 404 requests then get handled by the backend where they can still be logged and analyzed, if needed.
What do you need to do to ensure Varnish Cache does not Cache a 404 Error?
To prevent Varnish Cache from caching 404 responses, the following code can be applied (to version 4 or later) at, or near, the top of your vcl_backend_response:
sub vcl_backend_response {
# Don't cache 404 responses
if ( beresp.status == 404 ) {
set beresp.ttl = 120s;
set beresp.uncacheable = true;
return (deliver);
}
}
Thanks to Nigel Peck.
This code tells Varnish Cache not to cache any backend response with a 404 status and to store a “hit-for-pass” object instead in the cache, so that requests across the next two minutes (120s) get sent straight through to the backend.
Similarly to other cached objects, hit-for-pass objects have a TTL (time-to-live). After the object’s TTL has expired, the object is removed from the cache.
The vcl_backend_response built-in subroutine is designed precisely to avoid caching in conditions that are likely undesired, such as this one.
Peck also offers an option for caching 404 responses for a short time:
sub vcl_backend_response {
# Don't cache 404 responses
if ( beresp.status == 404 ) {
set beresp.ttl = 30s;
}
}
The 30 seconds TTL amount can obviously be altered to whatever duration you require the 404 responses to be cached for.
Learn More
To learn more about how to get started with Varnish Cache, including writing Varnish Cache Configuration Language to cache content for your application, please download the full Varnish Cache Guide. If you have specific questions about Varnish Cache and VCL, check out our community forum or contact us at our community Slack, and one of our Varnish Cache experts will be happy to help you.
