Resolve configuration optimization of Varnish cache under Linux
- 2020-06-15 07:54:44
- OfStack
Varnish is a high-performance open source HTTP accelerator. Norway's largest online newspaper, Verdens Gang, USES three Varnish instead of 12 Squid.
However, compared with the old squid, each of them has its own advantages and disadvantages. A large number of online comparisons are just for the best use of the application they are familiar with. Maybe squid is enough to play its most powerful role only when it is in the hands of competent people
Varnish adopts the "Visual Page Cache" technology. In terms of memory utilization, Varnish has advantages over Squid. It avoids frequent file exchange in memory and disk and has higher performance than Squid.
With the Varnish administration port, you can use regular expressions to clear some caches quickly and in bulk, which is one thing Squid doesn't have.
I will give a brief introduction and note on some views and configuration methods of varnish 1
Experimental environment: Red Hat Enterprise Linux Server release 5.4 (Tikanga)
Kernels 2.6.18-164. el5
yum install ES39en-ES40en ## # preinstall 1 package, otherwise error will be prompted
tar zxvf varnish-2.1.3.tar.gz
cd varnish-2.1.3
./configure --prefix=/usr/local/varnish-2.1.3
make & & make install
Edit configuration file, there are templates, but too many comments, it is better to create a new one
vim /usr/local/varnish-2.1.3/etc/varnish/varnish.conf
The following is the content of the configuration file and the comments ###################
#http request processing
#1,receive request entry status, pass or lookup local query according to vcl
#lookup, find data in hash table, enter hit state if found, otherwise enter fetch state
#pass, select background and enter fetch
#fetch, get the request back end, send the request, get the data, and store it locally
#deliver, send the data to the client, enter done
#done, end of processing
Configure the backend server #############
# # # # # # # # # # # # # # to configure the back-end server group, detect 6 seconds for health, use random set weight # # # # # # # #
# # # # # # # # # 1 other way round - robin the default polling mechanism # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # define access lists, to allow the following address to remove varnish cache # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # from behind the url judgment on what kind of server and cache configuration # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# # # # # # # # # # # # # # # # # # # # # # # # # for above varnish configuration file # # # # # # # # # # # # # # # # # # # # # # # # # #
Create user:
groupadd www
useradd www -g www
Create the cache location for varnish_cache
mkdir /data/varnish_cache
Start the varnish
ulimit-SHn 8192 #### # setup file descriptor, because my machine is not good, I can set it according to my own configuration
/usr/local/varnish-2.1.3/sbin/varnishd -u www -g www -f /usr/local/varnish-2.1.3/etc/varnish/varnish.conf -a 0.0.0.0:80 -s file,/data/varnish_cache/varnish_cache.data,100M -w 1024,8192,10 -t 3600 -T 127.0.0.1:3500
#### ## -ES116en for what -g for what group -f varnish configuration file - IP binds IP and port -s varnish cache file location and size -w minimum, maximum thread and timeout -T varnish management port, primarily for clearing caches
# End the varnishd process
pkill varnishd
Start varnishncsa to write the Varnish access log to the log file:
/usr/local/varnish-2.1.3/bin/varnishncsa -w /data/logs/varnish.log &
Zero run each day, according to cutting Varnish log, generated a compressed file, delete the old log last month at the same time the script (/ var logs/cutlog sh) :
vim /usr/local/varnish-2.1.3/etc/varnish/cut_varnish_log.sh
Write the following script:
#!/bin/sh
# This file run at 00:00
date=$(date -d "yesterday" +"%Y-%m-%d")
pkill -9 varnishncsa
mv /data/logs/varnish.log /data/logs/${date}.log
/usr/local/varnish-2.1.3/bin/varnishncsa -w /data/logs/varnish.log &
mkdir -p /data/logs/varnish/
gzip -c /data/logs/${date}.log > /data/logs/varnish/${date}.log.gz
rm -f /data/logs/${date}.log
rm -f /data/logs/varnish/$(date -d "-1 month" +"%Y-%m*").log.gz
Timed tasks:
crontab -e
00 00 * * * /usr/local/varnish-2.1.3/etc/varnish/cut_varnish_log.sh
Optimize Linux kernel parameters
vi /etc/sysctl.conf
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 5000 65000
Enable configuration
/sbin/sysctl -p
Manage the port via Varnish and clear the cache in bulk using regular expressions
Clear all caches
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 url.purge *$
Clear all caches in the image directory
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 url.purge /image/
127.0.0.1:3500 to be clear the cache server address www linuxidc. com for removal of domain name/static image/tt jsp url address list to be cleared
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 purge "req.http.host ~ www.linuxidc.com$ & & req.url ~ /static/image/tt.jsp"
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 PHP function to clear the Squid cache
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Configuration Boot Varnish automatically
vim /etc/rc.d/rc.local
Write the following on the last line:
ulimit -SHn 8192
/usr/local/varnish-2.1.3/sbin/varnishd -u www -g www -f /usr/local/varnish-2.1.3/etc/varnish/varnish.conf -a 0.0.0.0:80 -s file,/data/varnish_cache/varnish_cache.data,100M -w 1024,8192,10 -t 3600 -T 127.0.0.1:3500
/usr/local/varnish-2.1.3/bin/varnishncsa -w /data/logs/varnish.log &
View Varnish server connections and hit ratio:
/usr/local/varnish-2.1.3/bin/varnishstat
The above is the state of varnish,
Client requests received refers to the number of client requests received by the server
Cache hits refers to hit cache. The number of times data is obtained from the cache and returned to the client, namely hit ratio
Cache misses refers to the number of times the data is returned to the user from the back-end service application by skipping the pass cache
Use help to see which Varnish commands you can use:
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 help
However, compared with the old squid, each of them has its own advantages and disadvantages. A large number of online comparisons are just for the best use of the application they are familiar with. Maybe squid is enough to play its most powerful role only when it is in the hands of competent people
Varnish adopts the "Visual Page Cache" technology. In terms of memory utilization, Varnish has advantages over Squid. It avoids frequent file exchange in memory and disk and has higher performance than Squid.
With the Varnish administration port, you can use regular expressions to clear some caches quickly and in bulk, which is one thing Squid doesn't have.
I will give a brief introduction and note on some views and configuration methods of varnish 1
Experimental environment: Red Hat Enterprise Linux Server release 5.4 (Tikanga)
Kernels 2.6.18-164. el5
yum install ES39en-ES40en ## # preinstall 1 package, otherwise error will be prompted
tar zxvf varnish-2.1.3.tar.gz
cd varnish-2.1.3
./configure --prefix=/usr/local/varnish-2.1.3
make & & make install
Edit configuration file, there are templates, but too many comments, it is better to create a new one
vim /usr/local/varnish-2.1.3/etc/varnish/varnish.conf
The following is the content of the configuration file and the comments ###################
#http request processing
#1,receive request entry status, pass or lookup local query according to vcl
#lookup, find data in hash table, enter hit state if found, otherwise enter fetch state
#pass, select background and enter fetch
#fetch, get the request back end, send the request, get the data, and store it locally
#deliver, send the data to the client, enter done
#done, end of processing
Configure the backend server #############
backend linuxidc01 {
.host = "192.168.1.142";
.port = "7070";
.probe = {
.timeout = 5s;
.interval = 2s;
.window = 10;
.threshold = 8;
}
}
backend linuxidc02 {
.host = "192.168.1.141";
.port = "7070";
.probe = {
.timeout = 5s;
.interval = 2s;
.window = 10;
.threshold = 8;
}
}
# # # # # # # # # # # # # # to configure the back-end server group, detect 6 seconds for health, use random set weight # # # # # # # #
# # # # # # # # # 1 other way round - robin the default polling mechanism # # # # # # # # # # # # # # # # # # # #
director linuxidc15474 random
{ .retries = 6;
{ .backend = linuxidc02;
.weight = 2;
}
{ .backend = linuxidc01;
.weight = 2;
}
}
# # # # # # # # # # define access lists, to allow the following address to remove varnish cache # # # # # # # # # # # # # # # # # # # # # # #
acl local {
"localhost";
"127.0.0.1";
}
# # # # # # # # from behind the url judgment on what kind of server and cache configuration # # # # # # # # # # # # # # # # # # # # # # # # # # # #
sub vcl_recv
{
if (req.http.host ~ "^linuxidc15474.vicp.net") # Match the domain name to the backend server
{ set req.backend = linuxidc15474; }
else { error 404 "Unknown HostName!"; }
if (req.request == "PURGE") # Non - access control list is not allowed IP remove varnish The cache
{ if (!client.ip ~ local)
{
error 405 "Not Allowed.";
return (lookup);
}
}
# remove url There are jpg Of the file such as the cookie
if (req.request == "GET" && req.url ~ "\.(jpg|png|gif|swf|jpeg|ico)$")
{
unset req.http.cookie;
}
# judge req.http.X-Forwarded-For If you have multiple reverse agents on the front end, you can get the client IP Address.
if (req.http.x-forwarded-for)
{
set req.http.X-Forwarded-For = req.http.X-Forwarded-For ", " client.ip;
}
else { set req.http.X-Forwarded-For = client.ip; }
##varnish Realize the anti-hotlinking picture
# if (req.http.referer ~ "http://.*)
# {
# if ( !(req.http.referer ~ "http://.*vicp\.net" ||
# req.http.referer ~ "http://.*linuxidc15474\.net" ) )
# {
# set req.http.host = "linuxidc15474.vicp.net";
# set req.url = "/referer.jpg";
# }
# return(lookup);
# }
# else {return(pass);}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE")
{ return (pipe); }
# For the GET|HEAD The request is forwarded directly to the back-end server
if (req.request != "GET" && req.request != "HEAD")
{ return (pass); }
## right GET The request, and url in .php and .php? At the end, it is forwarded directly to the back-end server
if (req.request == "GET" && req.url ~ "\.(php)($|\?)")
{ return (pass); }
## There is validation and on the request cookie , directly forward to the back-end server
if (req.http.Authorization || req.http.Cookie)
{ return (pass);}
{
## In addition to the above access requests, look in the cache
return (lookup);
}
## The specified font Directories are not cached
if (req.url ~ "^/fonts/")
{ return (pass); }
}
sub vcl_pipe
{ return (pipe); }
## Enter the pass In mode, the request is sent to the back end, which returns data to the client but not to the cache
sub vcl_pass
{ return (pass); }
sub vcl_hash
{
set req.hash += req.url;
if (req.http.host)
{ set req.hash += req.http.host; }
else { set req.hash += server.ip; }
return (hash);
}
## in lookup If after cache Find the cache of the request, 1 Generally end with the following keywords
sub vcl_hit
{
if (!obj.cacheable)
{ return (pass); }
return (deliver);
}
##lookup Call after no cache found, end with the following keywords, and call fetch Parameter to retest whether to join the cache
sub vcl_miss
{ return (fetch); }
# let varnish Type of server cache that is called after retrieving data from the back end
sub vcl_fetch
{ if (!beresp.cacheable)
{ return (pass); }
if (beresp.http.Set-Cookie)
{ return (pass); }
##WEB The server specifies that the content is not cached, varnish Server does not cache
if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~ "no-cache" || beresp.http.Cache-Control ~ "private")
{ return (pass); }
## To visit get Have included jpg,png And the format of the file for caching, caching time is 7 Day, s For the second
if (req.request == "GET" && req.url ~ "\.(js|css|mp3|jpg|png|gif|swf|jpeg|ico)$")
{ set beresp.ttl = 7d; }
## To access get Contained in the htm Such as static pages, caching 300 seconds
if (req.request == "GET" && req.url ~ "\/[0-9]\.htm$")
{ set beresp.ttl = 300s; }
return (deliver);
}
#### Add to page head View cache hits in header information ########
sub vcl_deliver
{
set resp.http.x-hits = obj.hits ;
if (obj.hits > 0)
{ set resp.http.X-Cache = "HIT cqtel-bbs"; }
else { set resp.http.X-Cache = "MISS cqtel-bbs"; }
}
# # # # # # # # # # # # # # # # # # # # # # # # # for above varnish configuration file # # # # # # # # # # # # # # # # # # # # # # # # # #
Create user:
groupadd www
useradd www -g www
Create the cache location for varnish_cache
mkdir /data/varnish_cache
Start the varnish
ulimit-SHn 8192 #### # setup file descriptor, because my machine is not good, I can set it according to my own configuration
/usr/local/varnish-2.1.3/sbin/varnishd -u www -g www -f /usr/local/varnish-2.1.3/etc/varnish/varnish.conf -a 0.0.0.0:80 -s file,/data/varnish_cache/varnish_cache.data,100M -w 1024,8192,10 -t 3600 -T 127.0.0.1:3500
#### ## -ES116en for what -g for what group -f varnish configuration file - IP binds IP and port -s varnish cache file location and size -w minimum, maximum thread and timeout -T varnish management port, primarily for clearing caches
# End the varnishd process
pkill varnishd
Start varnishncsa to write the Varnish access log to the log file:
/usr/local/varnish-2.1.3/bin/varnishncsa -w /data/logs/varnish.log &
Zero run each day, according to cutting Varnish log, generated a compressed file, delete the old log last month at the same time the script (/ var logs/cutlog sh) :
vim /usr/local/varnish-2.1.3/etc/varnish/cut_varnish_log.sh
Write the following script:
#!/bin/sh
# This file run at 00:00
date=$(date -d "yesterday" +"%Y-%m-%d")
pkill -9 varnishncsa
mv /data/logs/varnish.log /data/logs/${date}.log
/usr/local/varnish-2.1.3/bin/varnishncsa -w /data/logs/varnish.log &
mkdir -p /data/logs/varnish/
gzip -c /data/logs/${date}.log > /data/logs/varnish/${date}.log.gz
rm -f /data/logs/${date}.log
rm -f /data/logs/varnish/$(date -d "-1 month" +"%Y-%m*").log.gz
Timed tasks:
crontab -e
00 00 * * * /usr/local/varnish-2.1.3/etc/varnish/cut_varnish_log.sh
Optimize Linux kernel parameters
vi /etc/sysctl.conf
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 5000 65000
Enable configuration
/sbin/sysctl -p
Manage the port via Varnish and clear the cache in bulk using regular expressions
Clear all caches
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 url.purge *$
Clear all caches in the image directory
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 url.purge /image/
127.0.0.1:3500 to be clear the cache server address www linuxidc. com for removal of domain name/static image/tt jsp url address list to be cleared
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 purge "req.http.host ~ www.linuxidc.com$ & & req.url ~ /static/image/tt.jsp"
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 PHP function to clear the Squid cache
<?php
function purge($ip, $url)
{
$errstr = '';
$errno = '';
$fp = fsockopen ($ip, 80, $errno, $errstr, 2);
if (!$fp)
{
return false;
}
else
{
$out = "PURGE $url HTTP/1.1\r\n";
$out .= "Host:blog.s135.com\r\n";
$out .= "Connection: close\r\n\r\n";
fputs ($fp, $out);
$out = fgets($fp , 4096);
fclose ($fp);
return true;
}
}
purge("192.168.0.4", "/index.php");
?>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Configuration Boot Varnish automatically
vim /etc/rc.d/rc.local
Write the following on the last line:
ulimit -SHn 8192
/usr/local/varnish-2.1.3/sbin/varnishd -u www -g www -f /usr/local/varnish-2.1.3/etc/varnish/varnish.conf -a 0.0.0.0:80 -s file,/data/varnish_cache/varnish_cache.data,100M -w 1024,8192,10 -t 3600 -T 127.0.0.1:3500
/usr/local/varnish-2.1.3/bin/varnishncsa -w /data/logs/varnish.log &
View Varnish server connections and hit ratio:
/usr/local/varnish-2.1.3/bin/varnishstat
The above is the state of varnish,
Client requests received refers to the number of client requests received by the server
Cache hits refers to hit cache. The number of times data is obtained from the cache and returned to the client, namely hit ratio
Cache misses refers to the number of times the data is returned to the user from the back-end service application by skipping the pass cache
Use help to see which Varnish commands you can use:
/usr/local/varnish-2.1.3/bin/varnishadm -T 127.0.0.1:3500 help