Varnish4.1.2代理缓存

简介：
Varnish是一款高性能、开源的反向代理服务器和缓存服务器，官方说是squid的四倍，实际应用测试中虽然达不到四倍的性能，那也能达到1-2倍的效果。
Varnish和Squid的对比：
Squid 也是一种开源的代理缓存软件，下面对比 Varnish 和 Squid 的不同点。
Varnish的稳定性很好。两者在完成相同负载的工作时，Squid服务器发生故障的几率要高于Varnish，因此Squid需要经常重启。Varnish访问速度更快。Varnish采用了 Visual Page Cache技术，所有缓存的数据都直接从内存读取，而Squid从硬盘读取缓存的数据，所以Varnish在访问速度方面会更快一些。Varnish可以支持更多的并发连接。因为Varnish的TCP连接与释放比Squid快，所以在高并发连接情况下可以支持更多的TCP连接。Varnish可以通过管理端口来管理缓存，使用正则表达式就可以批量清除部分缓存，而Squid做不到这一点。
Varnish缺点：
Varnish在高并发状态下，CPU、I/O和内存等资源的开销高于Squid。Varnish的进程一旦挂起、崩溃或者重启，缓存的数据都会从内存中释放出来。此时的所有请求都会被发送到后端应用服务器上，在高并发的情况下，就会给后端服务器造成很大压力。

一、安装
■1.下载：https://repo.varnish-cache.org/source/varnish-4.1.2.tar.gz
___________________________________________________________

yum install -y autoconf automake jemalloc-devel libedit-devel libtool ncurses-devel pcre-devel pkconfig python-docutils python-sphinx gcc gcc-c++  #安装依赖包
tar xf varnish-4.1.2.tar.gz
cd varnish-4.1.2
./autogen.sh
./configure --prefix=/usr/local/varnish --enable-debugging-symbols --enable-developer-warnings --enable-dependency-tracking
make && make install

■2.创建用户、组和日志文件

groupadd varnish
useradd -g varnish -s /sbin/nologin varnish
touch /usr/local/varnish/access.log                   #创建访问日志文件
chown -R varnish.varnish /usr/local/varnish

[b]二、修改配置varnish
[/b]
前端varnish缓存是以轮询的方式分担到后端web服务器，以下vcl不对php进行缓存
vim /usr/local/varnish/default.vcl

#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/ # and https://www.varnish-cache.org/trac/wiki/VCLExamples for more examples.
# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;
import std;
import directors;
# Default backend definition. Set this to point to your content server.
#backend default {
#    .host = "127.0.0.1";
#    .port = "8080";
#}
#probe healthcheck {
#   .url = "/";
#   .interval = 6s;
#   .timeout = 1s;
#   .window = 5;
#   .threshold = 3;
#}
backend fdfs_g1_s1 {
.host = "192.168.1.202";
.port = "80";
#     .probe = healthcheck;     #健康状态检测
}
backend fdfs_g1_s2 {
.host = "192.168.1.203";
.port = "80";
#    .probe = healthcheck;     #健康状态检测
}
acl purgers {
"localhost";
"127.0.0.1";
}
sub vcl_init {
new vdir = directors.round_robin();
vdir.add_backend(fdfs_g1_s1);
vdir.add_backend(fdfs_g1_s2);
return (ok);
}
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
set req.backend_hint = vdir.backend();
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
if (req.method == "PURGE") {    # PURGE请求的处理
if (!client.ip ~ purgers) {
return(synth(405,"Method not allowed"));
}
#清理缓存
return (purge);
}
# 禁止缓存的文件
if (req.url ~ "\.(php|jsp|do)($|\?|#)") {
return (pass);
}
if (req.method == "PRI") {
#/* We do not support SPDY or HTTP/2.0 */
return (synth(405));
}
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
#/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.method != "GET" && req.method != "HEAD") {
#/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
#/* Not cacheable by default */
return (pass);
}
return (hash);
}
sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
unset beresp.http.set-cookie;
}
# Large static files are delivered directly to the end-user without
# waiting for Varnish to fully read the file first.
# Varnish 4 fully supports Streaming, so use streaming here to avoid locking.
if (bereq.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
unset beresp.http.set-cookie;
set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
set beresp.do_gzip   = false;   # Don't try to compress it for storage
}
# Set 2min cache if unset for static files
if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") {
set beresp.ttl = 120s; # Important, you shouldn't rely on this, SET YOUR HEADERS in the backend
set beresp.uncacheable = true;
return (deliver);
}
# Allow stale content, in case the backend goes down.
# make Varnish keep all objects for 6 hours beyond their TTL
set beresp.grace = 6h;
return (deliver);
}
sub vcl_backend_fetch {
return (fetch);
}
sub vcl_pipe {
# By default Connection: close is set on all piped requests, to stop
# connection reuse from sending future requests directly to the
# (potentially) wrong backend. If you do want this to happen, you can undo
# it here.
# unset bereq.http.connection;
return (pipe);
}
sub vcl_pass {
return (fetch);
}
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}
sub vcl_purge {
return (synth(200, "Purged"));
}
sub vcl_hit {
if (obj.ttl >= 0s) {
#// A pure unadultered hit, deliver it
return (deliver);
}
if (obj.ttl + obj.grace > 0s) {
#// Object is in grace, deliver it
#// Automatically triggers a background fetch
return (deliver);
}
#// fetch & deliver once we get the result
return (miss);
}
sub vcl_miss {
return (fetch);
}
sub vcl_deliver {
set resp.http.x-hits = obj.hits;
if (obj.hits > 0) {
set resp.http.X-Cache = "Hit varnish cache";
}else {
set resp.http.X-Cache = "Miss varnish cache";
}
return (deliver);
}
sub vcl_synth {
set resp.http.Content-Type = "text/html; charset=utf-8";
set resp.http.Retry-After = "5";
synthetic( {"<!DOCTYPE html>
<html>
<head>
<title>"} + resp.status + " " + resp.reason + {"</title>
</head>
<body>
<h1>Error "} + resp.status + " " + resp.reason + {"</h1>
<p>"} + resp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>XID: "} + req.xid + {"</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
"} );
return (deliver);
}
sub vcl_backend_error {
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
synthetic( {"<!DOCTYPE html>
<html>
<head>
<title>"} + beresp.status + " " + beresp.reason + {"</title>
</head>
<body>
<h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>
<p>"} + beresp.reason + {"</p>
<h3>Guru Meditation:</h3>
<p>XID: "} + bereq.xid + {"</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
"} );
return (deliver);
}
sub vcl_fini {
return (ok);
}

语句解析：
Varnish处理请求的主要处理方法
1. vcl_recv
首先接收请求，判断是否要进一步处理，还是直接转发给后端（pass）等。 此过程中可以使用和请求相关的变量，例如客户端请求的url，ip，user-agent，cookie等，此过程中可以把不需缓存的地址，通过判断（相等、不相等、正则匹配等方法）转给后端，例如gif/png/jpg/css/js等静态文件；
2. vcl_fetch
当从后端服务器获取内容后会进入此阶段，除了可以使用客户端的请求变量，还可以使用从后端获取的信息（bersp），如后端返回的头信息，设置此信息的缓存时间TTL等；
3. vcl_miss
缓存未命中时中要做的处理
4. vcl_hit
缓存命中后做的处理
5. vcl_deliver
发送给客户端前的处理
6. vcl_pass
交给后端服务器
7. vcl_hash
设置缓存的键值key
Varnish处理流程
首次请求时过程如下：
recv->hash->miss->fetch->deliver
缓存后再次请求：
recv->hash->hit->deliver（fetch的过程没了，这就是我们要做的，把要缓存的页面保存下来）
直接交给后端pass的情况：
recv->hash->pass->fetch->deliver（直接从后端获取数据后发送给客户端，此时Varnish相当于一个中转站，只负责转发）

三、测试
■1.启动
/usr/local/varnish/sbin/varnishd -f /usr/local/varnish/default.vcl -s malloc,512M -T 127.0.0.1:2000 -a 0.0.0.0:80
-f：选项用于指定Varnishd使用的配置文件的路径。
-s malloc,2G:–s选项用来确定Varnish使用的存储类型和存储容量，这里使用的是malloc类型（malloc是一个C函数，用于分配内存空间），2G 定义多少内存被malloced。
-T：127.0.0.1:2000是Varnish基于文本方式的一个管理接口，启动后可以在不停止Varnish的情况下来管理Varnish。管理端口2000可以指定。因为不是任何人都可以访问Varnish管理端口，所以这里推荐只监听本机端口。
-a：0.0.0.0:80中-a选项表示Varnish监听所有IP发给80端口的HTTP请求。
■2.设置varnish访问日志

/usr/local/varnish/bin/varnishncsa -n /usr/local/varnish/var/varnish/test.com -a -w /usr/local/varnish/access.log & #将varnish访问日志写入到access.log里
netstat -tuplan |grep varnish #查看是否启动2000和80端口
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 28551/varnishd
tcp 0 0 127.0.0.1:2000 0.0.0.0:* LISTEN 28550/varnishd
tcp 0 0 :::80 :::* LISTEN 28551/varnishd
curl -I http://ip #使用curl命令查看是否被缓存
X-Cache: Hit varnish cache
killall -9 varnishd #关闭varnish
■3、设置开机启动:
echo '/usr/local/varnish/sbin/varnishd -f /usr/local/varnish/etc/varnish/default.vcl -s malloc,512M -T 127.0.0.1:2000 -a 0.0.0.0:80' >> /etc/rc.local