Achieving a high hitrate

Now that Varnish is up and running you can access your web application through Varnish. Unless your application is specifically written to work behind a web accelerator you’ll probably need to do some changes to either the configuration or the application in order to get a high hitrate in Varnish.

Varnish will not cache your data unless it’s absolutely sure it is safe to do so. So, for you to understand how Varnish decides if and how to cache a page, we’ll guide you through a couple of tools that you should find useful to understand what is happening in your Varnish setup.

Note that you need a tool to see the HTTP headers that fly between Varnish and the backend. On the Varnish server, the easiest way to do this is to use varnishlog and varnishtop but sometimes a client-side tool makes sense. Here are the ones we commonly use.

Tool: varnishtop

You can use varnishtop to identify what URLs are hitting the backend the most. varnishtop -i BereqURL is an essential command, showing you the top requests Varnish is sending to the backend. You can see some other examples of varnishtop usage in Statistics.

Tool: varnishlog

When you have identified an URL which is frequently sent to the backend you can use varnishlog to have a look at the request. varnishlog -q 'ReqURL ~ "^/foo/bar"' will show you the requests coming from the client matching /foo/bar.

For more information on how varnishlog works please see Logging in Varnish or then man page.

Tool: lwp-request

lwp-request is tool that is a part of The World-Wide Web library for Perl. It’s a couple of really basic programs that can execute an HTTP request and show you the result. We mostly use the two programs, GET and HEAD.

vg.no was the first site to use Varnish and the people running Varnish there are quite clueful. So it’s interesting to look at their HTTP Headers. Let’s send a GET request for their home page:

$ GET -H 'Host: www.vg.no' -Used http://vg.no/
GET http://vg.no/
Host: www.vg.no
User-Agent: lwp-request/5.834 libwww-perl/5.834

200 OK
Cache-Control: must-revalidate
Refresh: 600
Title: VG Nett - Forsiden - VG Nett
X-Age: 463
X-Cache: HIT
X-Rick-Would-Never: Let you down
X-VG-Jobb: http://www.finn.no/finn/job/fulltime/result?keyword=vg+multimedia Merk:HeaderNinja
X-VG-Korken: http://www.youtube.com/watch?v=Fcj8CnD5188
X-VG-WebCache: joanie
X-VG-WebServer: leon

OK. Lets look at what GET does. GET usually sends off HTTP 0.9 requests, which lack the ‘Host’ header. So we add a ‘Host’ header with the ‘-H’ option. ‘-U’ print request headers, ‘-s’ prints response status, ‘-e’ prints response headers and ‘-d’ discards the actual content. We don’t really care about the content, only the headers.

As you can see, VG adds quite a bit of information in their headers. Some of the headers, like the ‘X-Rick-Would-Never’ are specific to vg.no and their somewhat odd sense of humour. Others, like the ‘X-VG-Webcache’ are for debugging purposes.

So, to check whether a site sets cookies for a specific URL, just do:

GET -Used http://example.com/ |grep ^Set-Cookie

Tool: Live HTTP Headers

There is also a plugin for Firefox called Live HTTP Headers. This plugin can show you what headers are being sent and received. Live HTTP Headers can be found at https://addons.mozilla.org/en-US/firefox/addon/3829/ or by googling “Live HTTP Headers”.