Of Stack

Detailed method of analyzing nginx log using goaccess

  • 2021-11-01 05:45:49
  • OfStack

Recently, I want to use goaccess to analyze the nginx log, but suffer from the fact that the configuration format of nginx log is not in accordance with the normal format, which is completely written according to our own needs, so goaccess can't be analyzed, and I need to redefine the format myself; However, although there are many introductions of goaccess on the Internet, most of them are heavy and avoid light, ignoring the customization of format, so I will talk about customization.

GoAccess is an open source, real-time web log analysis tool running under the command line terminal. This tool provides fast and diverse HTTP state statistics, which can make administrators no longer struggle to count all kinds of data, complicated instructions and a lot of pipes/regular expressions to say byebye.

Analyze nginx log

Various display modes of GoAccess
goaccess has many ways to visualize data, namely:

Command line output formatted data
Using access. log to generate static visual data
Generate real-time visual data
Note that the configuration file needs to be edited if enable-geoip=mmdb is compiled and installed, and the parameters config-file=/usr/local/etc/goaccess/goaccess. conf are taken when using the command, which is not required if the package manager is installed

Command line output GoAccess
goaccess/var/log/nginx/access. log-c, I will ask you the format of the data first, and I use the first kind of log here.

Parsing accesslog to generate static html
GoAccess can also parse access. log to generate static html to present data in a more intuitive way.

goaccess/var/log/nginx/access. log-o report. html-log-format=COMBINED, and then visit report.html with a browser to view the report. All kinds of data should be available.

Parsing access logs in real time
GoAccess can generate static html files, but also can generate real-time website access data!

goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED --real-time-html --config-file=/usr/local/etc/goaccess/goaccess.conf

Add Chinese support
Goaccess install language-pack-zh-hans installs the Chinese package on the command line, then modifies the environment variables using export LANG=zh_CN. UTF-8, and then uses goaccess/var/log/nginx/access. log-o/var/www/html/report. html-log-format=COMBINED-real-time-ES114 EN-config-file=/usr/local/etc/goaccess/goaccess. conf Start GoAccess and you will find that it is already a Chinese interface.

For real-time mode, you can see demo https://rt.goaccess.io/ in official website. 20200209201008

Abnormal exit
If real-time mode does not exit normally, it may not start normally again. GoAccess uses port 7890 websocket by default, so use lsof-i: 7890 to view the process number occupying this port and kill.

ssl support
If you need to output real-time data on an encrypted connection, You need to use --ssl-cert= and--ssl-key=, and I visited report. html after setup and found that the data was still static, and suddenly remembered that I used cloudflare cdn and port 7890 was not in the list of supported ports for cloudflare, so I changed the port to 8443 with the parameter --ws-url=wss://server domain name (our browser will try to connect to port 8443 of this domain name and see ws): 8443-port=8443. Unexpectedly, report. html can be connected when using proxy link, and can view real-time information, while it is still static data when connecting directly, tcping1.

Go to cloudflare's official website and find the following contents

Only ports 80 and 443 are compatible with the following services:

For HTTP/HTTPS traffic of data centers in China with the domain name of China network enabled,
That is to say, there is no way to connect non-80/443 ports through cloudflare in China …

Reverse proxy
But it's not that there is no way to connect. Finally, I thought of the reverse proxy scheme.

Change the startup parameter to -ws-url=wss://Your domain name. com/goaccess-port=7890

Modify the nginx site profil/etc/nginx/site-available/default to add the following 


location /goaccess {
    proxy_redirect off;
    proxy_pass https://127.0.0.1:7890;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
}

Note that if url rewriting is enabled in your site configuration file, in order to avoid/goaccess being affected, we need to exclude rewriting of this path.

Put all the rewrite rules in location/ 


location / { 
    if (-f $request_filename/index.html){
    rewrite (.*) $1/index.html break;
    }
    if (-f $request_filename/index.php){
    rewrite (.*) $1/index.php;
    }
    if (!-f $request_filename){
    rewrite (.*) /index.php;
    }
}

Nothing needs to be done below 


location /goaccess/ {
}

After that, restart nginx, then visit report. html, and find that connect is finally displayed at the left gear.

If you just look at it yourself or don't care about the exposure of ip, it won't be so troublesome to use ip directly without cdn.

Related articles: