The problem Varnish solves
By the late 2000s web traffic grows exponentially and the typical stack — Apache HTTP Server + PHP + MySQL — shows its scalability limits. Caching reverse proxies (primarily Squid) were designed for forward proxy and adapt with friction to the inverse case. Poul-Henning Kamp, longstanding FreeBSD committer, launched Varnish Cache in 2006 for Norwegian newspaper Verdens Gang (VG), with a specific goal: an HTTP reverse proxy designed natively for modern web caching.
Varnish 2.0 was released in November 2008; the 2.0.x series, culminating in 2.0.5 in October 2009, consolidates the project’s maturity. BSD 2-Clause licence.
Technical ideas
Varnish introduces distinctive architectural choices:
- Shared memory and mmap — the whole cache is mapped in shared memory, delegating paging to the kernel. Design is explicitly “let the kernel do it”: Varnish does not explicitly manage RAM vs. disk cache, but relies on the virtual memory subsystem
- Worker thread pool — each connection handled by a dedicated thread; a different choice from single-threaded event-loops (Nginx) but effective on modern Linux/FreeBSD kernels with efficient schedulers
- VCL — Varnish Configuration Language — DSL compiled to C by varnishd at load time; rule set deciding what to cache, how, for how long, with which variations
- Sticky backends and directors — balancing to multiple backends with policies (round-robin, random, client hash, DNS)
- HTTP/1.1 compliance — compliant handling of Vary, Cache-Control, Age, If-Modified-Since, conditional GET, ESI (Edge Side Includes)
A VCL example
sub vcl_recv {
if (req.http.host ~ "^(www\.)?example\.com$") {
set req.http.host = "example.com";
}
if (req.url ~ "\.(png|jpg|css|js)$") {
unset req.http.cookie;
}
}
sub vcl_backend_response {
if (bereq.url ~ "\.(png|jpg|css|js)$") {
set beresp.ttl = 1d;
}
}Expressive power is enough to handle complex scenarios: authentication, A/B testing, rewrite, compression, ESI. C compilation guarantees performance comparable to hand-written code.
Performance
Varnish 2.x on 2009 commodity hardware handles tens of thousands of requests per second with sub-millisecond latency for HITs. Throughput is typically limited by network or kernel, not the Varnish process.
The varnishlog command exposes a detailed circular log in shared memory, consumable by tools like varnishstat, varnishtop, varnishhist for real-time monitoring. This model — circular log in shared memory — is another technical innovation.
Adoption
By late 2009 Varnish is in production at:
- Verdens Gang (Norwegian newspaper that commissioned the project)
- The New York Times, The Guardian, The Times — international publishing
- Wikipedia (WMF) — cache layer in front of Squid and MediaWiki
- Twitter, Flickr, LinkedIn — high social traffic
- Drupal, WordPress VIP — CMS ecosystems
Italian adoption follows in 2010-2012 in high-traffic portals, online newspapers, e-commerce platforms.
Varnish Plus / Enterprise
Varnish Software (a company founded in 2010 to support the project) will progressively introduce an Enterprise version with additional features (native SSL support — public 2.x has no TLS, handled externally; PSU persistent storage; commercial support). The core remains open source BSD.
The emerging cloud stack
The typical architectural pattern at end-2009 for a high-scalability website includes Varnish:
[Client] → [Load Balancer] → [Varnish] → [Web Server] → [Application] → [Database]
↓
[Static assets]With Varnish in front, the web server and application are relieved of most requests: typical HIT rates of 80-95% for content-heavy sites. This pattern will naturally move in the next decade toward distributed CDNs (Akamai, Cloudflare, Fastly — the latter Varnish-based itself) and eventually toward serverless architectures with integrated caching.
References: Varnish Cache 2.0.5 (October 2009). Poul-Henning Kamp (FreeBSD committer, original author). Verdens Gang as commissioner. BSD 2-Clause licence. VCL — Varnish Configuration Language. HTTP/1.1 RFC 2616.