High CPU and Server Crash everyday

The new Ubuntu KVM server running WordPress sites crashes every day. During further debug, I’ve found that ‘lsphp’ process consumes almost high CPU all the time that might cause this crash but don’t see anything relevant in the error logs.

I am able to access CyberPanel always but can’t access OpenLiteSpeed WebAdmin or any sites in the browser so this looks like web server or database server issue? Restarting LSWS, LSCPD, MySQL, MariaDB and PowerDNS does nothing. Only server reboot get the sites working again.

Here are some useful command results:

systemctl status lsws

lshttpd.service - OpenLiteSpeed HTTP Server
Loaded: loaded (/etc/systemd/system/lshttpd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-10-27 13:26:11 BST; 1h 25min ago
Process: 27405 ExecStart=/usr/local/lsws/bin/lswsctrl start (code=exited, status=0/SUCCESS)
Main PID: 27420 (litespeed)
CGroup: /system.slice/lshttpd.service
├─ 1073 lsphp
├─27420 openlitespeed (lshttpd - main)
├─27421 openlitespeed (lscgid)
├─27423 openlitespeed (lshttpd - #01)
├─27424 openlitespeed (lshttpd - #02)
├─27425 openlitespeed (lshttpd - #03)
└─27428 openlitespeed (lshttpd - #04)

Oct 27 13:26:08 ubuntu20 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 27 13:26:08 ubuntu20 systemd[1]: lshttpd.service: Found left-over process 27369 (lsphp) in control group while starting unit. Ignoring.
Oct 27 13:26:08 ubuntu20 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 27 13:26:08 ubuntu20 systemd[1]: lshttpd.service: Found left-over process 27378 (lsphp) in control group while starting unit. Ignoring.
Oct 27 13:26:08 ubuntu20 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 27 13:26:08 ubuntu20 systemd[1]: lshttpd.service: Found left-over process 27379 (lsphp) in control group while starting unit. Ignoring.
Oct 27 13:26:08 ubuntu20 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Oct 27 13:26:08 ubuntu20 systemd[1]: Starting OpenLiteSpeed HTTP Server…
Oct 27 13:26:08 ubuntu20 lswsctrl[27405]: [OK] Send SIGUSR1 to 8012
Oct 27 13:26:11 ubuntu20 systemd[1]: Started OpenLiteSpeed HTTP Server.

top

top - 14:58:07 up 11:16, 1 user, load average: 0.03, 0.09, 0.15
Tasks: 134 total, 1 running, 133 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 0.8 sy, 3.5 ni, 94.6 id, 0.4 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 3935.6 total, 206.1 free, 716.0 used, 3013.6 buff/cache
MiB Swap: 2560.0 total, 2332.0 free, 228.0 used. 2809.8 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31099 chilt23+ 21 1 234744 74792 63532 S 9.0 1.9 0:00.27 lsphp
785 mysql 20 0 2765528 362216 9584 S 1.3 9.0 40:26.55 mysqld
27423 nobody 20 0 109308 71068 14688 S 0.7 1.8 0:23.69 litespeed
27425 nobody 20 0 105296 66772 14168 S 0.7 1.7 0:53.61 litespeed
1 root 20 0 168992 7848 5844 S 0.3 0.2 0:05.97 systemd
10 root 20 0 0 0 0 I 0.3 0.0 0:39.99 rcu_sched
704 lscpd 20 0 11680 5488 4656 S 0.3 0.1 0:24.08 lscpd
29167 root 20 0 0 0 0 I 0.3 0.0 0:00.27 kworker/1:2-mm_percpu_wq
29841 root 20 0 0 0 0 I 0.3 0.0 0:00.43 kworker/u8:1-events_power_efficient
31097 root 20 0 12024 3852 3176 R 0.3 0.1 0:00.02 top
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par_gp
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H-kblockd
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
9 root 20 0 0 0 0 S 0.0 0.0 0:01.47 ksoftirqd/0
11 root rt 0 0 0 0 S 0.0 0.0 0:00.20 migration/0
12 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_inject/0
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0

Thanks in advance.

What about your server specs?

@usmannasir Server specs are as below:

4 GB RAM
4 CPU Cores
60 GB NVMe Disk

Let me know if you need any more specific info.

This is what I’ve done so far on site level:

  • Disabled the Cron (WordPress)
  • Disabled open_basedir
  • Confirmed that there are no malicious code inserted on any site files

Are you getting cache hit for your website?

Yes. Let me share the headers:

pragma: no-cache
server: LiteSpeed
strict-transport-security: max-age=31536000; includeSubDomains;preload
vary: Accept-Encoding
x-litespeed-cache: hit,private
x-ua-compatible: IE=edge

Enable cloudflare under attack mode.

I don’t use Cloudflare yet but happy to try it. Should I?

What do you think the possible causes for the above described issue?

I can’t tell much with this limited information, but, if your site is having usual visitors and this started happening randomly then you can be under attack.

Fair enough. Let me try Cloudflare and add my comment later.

@usmannasir I’ve configured CloudFlare with Attack mode enabled. However, it didn’t fix the issue and all the sites and OpenLiteSpeed WebAdmin went down after few hours as usual.

Now before rebooting the server, this time I’ve decided to disable ModSecurity (as this was the option I didn’t try). So under CyberPanel → Security →ModSecurity Conf, I’ve turned OFF ModSecurity Status

…and it did a magic and all sites started to run normally without restarting any services. So this is something definitely has to do with the ModSecurity.

Here are some additional info. in case it helps.

ModSecurity Rules Pack


owasp-master.conf: No such file or directory : Is this the issue?

ModSecurity Rules

CSF is not installed

I understand it is not a wise solution to keep ModSecurity disabled. Do you think are there any other settings I need to look into to fix this? How about reinstalling or reconfiguring ModSecurity under CyberPanel?

@usmannasir This is what I see in the ModSecurity Audit logs:

ModSecurity: Warning. Matched "Operator ‘PmFromFile’ with parameter ‘userdata_wl_content_type’ against variable ‘TX:0’ (Value: ‘application/x-www-form-urlencoded’ ) [file “/usr/local/lsws/conf/modsec/comodo/09_HTTP_HTTP.conf”] [line “16”] [id “210710”] [rev “5”] [msg “COMODO WAF: Request content type is not allowed by policy. Please update file userdata_wl_content_type.||www.my-domain.co.uk|F|2”] [data “TX:0=application/x-www-form-urlencoded”] [severity “2”] [ver “”] [maturity “0”] [accuracy “0”] [tag “CWAF”] [tag “HTTP”] [hostname “www.my-domain.co.uk”] [uri “/”] [unique_id “1635406677”] [ref “v0,4o0,33o0,33v87,33”]

I’ve tried to add the following ModSecurity Rule but no joy.

SecRuleRemoveById 210710

@shoaibkk can you check this, I think this issue got resolved with OpenLiteSpeed?

@usmannasir Still some servers are facing Mode security and OpenLiteSpeed issue. @dipakcg what is the current version of OLS on your server

@shoaibkk I am using OpenLiteSpeed 1.7.14 on CyberPanel 2.1 Build 2.

I am happy to run any commands for you, or let me know if you need any login access.

OLS is running as user(nobody) : group(nogroup). Is that OK?

Just turn off ModSecurity for now from the cyberpanel and then restart lsws

As stated earlier, sites and OLS WebAdmin started to work again since I turned off the ModSecurity under CyberPanel → Security →ModSecurity Conf. It didn’t require me to restart LSWS even after turning it off.

Should I keep ModSecurity disabled for now until you (or We) come up with a fix/patch?

This is what I see under the ModSecurity Rules Pack:

Screenshot URL :link:

I don’t see any Comodo Rules Pack or anything similar. However, the info. I’ve found under ModSecurity Audit logs were related to Comodo.

Just thought to share this with you in case it helps.

This is what I see under OLS WebAdmin → Server Configuration → Modules → mod_security:


modsecurity off
modsecurity_rules ’
SecDebugLogLevel 9
SecDebugLog /usr/local/lsws/logs/modsec.log
SecAuditEngine on
SecAuditLogRelevantStatus “^(?:5|4(?!04))”
SecAuditLogParts ABCDEFGHI
SecAuditLogType Serial
SecAuditLog /usr/local/lsws/logs/auditmodsec.log
SecRuleEngine On’
modsecurity_rules_file /usr/local/lsws/conf/modsec/rules.conf

modsecurity_rules_file /usr/local/lsws/conf/modsec/owasp-modsecurity-crs-3.0-master/owasp-master.conf

modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/modsecurity.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/00_Init_Initialization.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/01_Init_AppsInitialization.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/02_Global_Generic.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/03_Global_Agents.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/04_Global_Domains.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/05_Global_Backdoor.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/06_XSS_XSS.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/07_Global_Other.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/08_Bruteforce_Bruteforce.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/09_HTTP_HTTP.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/10_HTTP_HTTPDoS.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/11_HTTP_Protocol.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/12_HTTP_Request.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/13_Outgoing_FilterGen.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/14_Outgoing_FilterASP.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/15_Outgoing_FilterPHP.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/16_Outgoing_FilterSQL.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/17_Outgoing_FilterOther.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/18_Outgoing_FilterInFrame.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/19_Outgoing_FiltersEnd.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/20_PHP_PHPGen.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/21_SQL_SQLi.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/22_Apps_Joomla.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/23_Apps_JComponent.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/24_Apps_WordPress.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/25_Apps_WPPlugin.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/26_Apps_WHMCS.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/27_Apps_Drupal.conf
modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/28_Apps_OtherApps.conf


Should I remove modsecurity_rules_file /usr/local/lsws/conf/modsec/comodo/09_HTTP_HTTP.conf line as that’s what I see under ModSecurity Audit Logs?