Updating LogFormat for better stats collection

Blog Posts
1

As mentioned here collecting stats for Cyberpanel accesslog is a real pita.

I submitted fix to cyberpanel for this in 1.9.4 but this only fixes it for new domains created after the server is running it.

Please note the server should be upgraded to this version before doing any of the 2 or optional 3rd step.

This will upgrade to 1.9.4
sh <(curl -s https://raw.githubusercontent.com/usmannasir/cyberpanel/v1.9.4/preUpgrade.sh || wget -qO - https://raw.githubusercontent.com/usmannasir/cyberpanel/v1.9.4/preUpgrade.sh)

The information below is for how to update existing vhost and ols/LS LogFormat to the same universal standard cPanel uses which bring a lot of cool possibilities to getting stats.

Steps

  1. Updating httpd.conf for new log format for stats.

grep LogFormat /usr/local/lsws/conf/httpd.conf

Fix httpd.conf logformat inline
sed -i “s|^LogFormat.*|LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined|g” /usr/local/lsws/conf/httpd.conf

Before:
[root@wcloud:~]# grep LogFormat /usr/local/lsws/conf/httpd.conf
LogFormat ‘“%v %h %l %u %t "%r" %>s %b”’ combined
[root@wcloud:~]#

After:
[root@wcloud:~]# grep LogFormat /usr/local/lsws/conf/httpd.conf
LogFormat ‘%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”’ combined
[root@wcloud:~]#

Example:
[root@wcloud:~]# sed -i “s|^LogFormat.*|LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined|g” /usr/local/lsws/conf/httpd.conf
[root@wcloud:~]#

  1. Fixing vhost log format for a single vhost.

What this does is insert the line “LogFormat ‘%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”’ combined” above the lines with “CustomLog” in vhost conf file

Before:
CustomLog /home/example.com/logs/example.com.access_log combined

After:
LogFormat ‘%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”’ combined
CustomLog /home/example.com/logs/example.com.access_log combined

Fixing a single vhost full example:

Commands:
Test without editing
sed “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/g” /usr/local/lsws/conf/vhosts/example.com/vhost.conf

Edit inline
sed -i “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/g” /usr/local/lsws/conf/vhosts/example.com/vhost.conf

Before:
[root@wcloud:~]# cat /usr/local/lsws/conf/vhosts/example.com/vhost.conf
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
[root@wcloud:~]#

After:
[root@wcloud:~]# cat /usr/local/lsws/conf/vhosts/example.com/vhost.conf
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
[root@wcloud:~]#

Testing Example:
[root@wcloud:~]# sed “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/” /usr/local/lsws/conf/vhosts/example.com/vhost.conf
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
[root@wcloud:~]#
  1. Bulk editing all hosts to have new log format. This is great if you have alot of domains on cyberpanel server and do not want to have to do this one time per site.

Test on all vhosts without editing
sudo find /usr/local/lsws/conf/vhosts/ -type f -name ‘vhost.conf’ -exec sed “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/g” {} \;

Apply to all vhosts via inline editing
sudo find /usr/local/lsws/conf/vhosts/ -type f -name ‘vhost.conf’ -exec sed -i “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/g” {} \;

Testing Example:
[root@wcloud:~]# sudo find /usr/local/lsws/conf/vhosts/ -type f -name ‘vhost.conf’ -exec sed “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/g” {} \;
<VirtualHost *:80>

ServerName dev.example.com
ServerAlias www.dev.example.com
ServerAdmin [email protected]
SuexecUserGroup devuser devuser
DocumentRoot /home/dev.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/dev.example.com/logs/dev.example.com.access_log combined
AddHandler application/x-httpd-php73 .php .php7 .phtml
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

<VirtualHost *:443>

ServerName dev.example.com
ServerAlias www.dev.example.com
ServerAdmin [email protected]
SuexecUserGroup devuser devuser
DocumentRoot /home/dev.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/dev.example.com/logs/dev.example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/dev.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/dev.example.com/privkey.pem
AddHandler application/x-httpd-php73 .php .php7 .phtml

ServerName wcloud.example.com
ServerAlias www.wcloud.example.com
ServerAdmin [email protected]
SuexecUserGroup wcloudw wcloudw
DocumentRoot /home/wcloud.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/wcloud.example.com/logs/wcloud.example.com.access_log combined
AddHandler application/x-httpd-php73 .php .php7 .phtml
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

<VirtualHost *:443>

ServerName wcloud.example.com
ServerAlias www.wcloud.example.com
ServerAdmin [email protected]
SuexecUserGroup wcloudw wcloudw
DocumentRoot /home/wcloud.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/wcloud.example.com/logs/wcloud.example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/wcloud.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/wcloud.example.com/privkey.pem
AddHandler application/x-httpd-php73 .php .php7 .phtml

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php

ServerName my.example.com
ServerAlias www.my.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html/my.example.com
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/my.example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>

ServerName my.example.com
ServerAlias www.my.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html/my.example.com
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/my.example.com.access_log combined
<IfModule LiteSpeed>
    CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/my.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my.example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
[root@wcloud:~]#

Applying changes to all vhost.confs by editing inline.
[root@wcloud:~]# sudo find /usr/local/lsws/conf/vhosts/ -type f -name ‘vhost.conf’ -exec sed -i “s/.CustomLog./ LogFormat ‘%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"’ combined
&/g” {} \;
[root@wcloud:~]#

If all goes well you will want to restart the lsws webservice to activate changes
[root@wcloud:~]# service lsws restart
Redirecting to /bin/systemctl restart lsws.service
[root@wcloud:~]# service lsws status
Redirecting to /bin/systemctl status lsws.service
● lshttpd.service - LiteSpeed HTTP Server
Loaded: loaded (/usr/lib/systemd/system/lshttpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-03 09:42:22 EST; 4s ago
Process: 27822 ExecStop=/usr/local/lsws/bin/lswsctrl delay-stop (code=exited, status=0/SUCCESS)
Process: 27834 ExecStart=/usr/local/lsws/bin/lswsctrl start (code=exited, status=0/SUCCESS)
Main PID: 27844 (litespeed)

I hope you enjoyed this post and it helps get your access_logs in a usable format.

My next post will show how you can get cool stats once you have log format changed into the new format.

2

I have a few servers running Cyberpanel v2.0. The format of the vhost access logs is different that what you show here. Here is an example extract from the vhost.conf:

accesslog $VH_ROOT/logs/$VH_NAME.access_log {
useServer 0
logFormat “%v %h %l %u %t “%r” %>s %b”
logHeaders 5
rollingSize 10M
keepDays 10 compressArchive 1
}

I can certainly add a line for LogFormat. The question is this: Should log format look like this:

LogFormat ‘%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”’ combined

or like this (without combined):

LogFormat ‘%h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”’