Possible dead lock [LSAPI application]

& the usage of the server is still normal no cpu usage ,

Which kind of websites are you running?

Are they built froma framework like Laravel, WordPress or ?

Are they running latest versions ?

There seems to be a php process choking the system

WordPress Only

Yes the all sites are running latest 6.1.1 versions ,

and all plugins are updated

How to check which process is choking it ?

After Applying those changes , now i am facing new issue ,

Before It was accessed like this on ServerUrl:8080


c

But now i am receiving Deceptive Site Ahead:

Turn on debugging for OLS How to turn on Extensive Debug logs in OpenLiteSpeed!

Then run

$ ps aux | grep lsphp80
## run lighweight debugger for "hanging" processes
$ strace -p <PID_PROCESS>

Post your /usr/local/lsws/logs/error.log in the meantime


I don’t find any cyberpanel in it

Run in ssh terminal or vnc tool

image


the first command worked but second one did not ? it gives the error given above

Here are the error logs

2023-03-18IP.4____7 [NOTICE] [2____4] [AutoRestarter] child process with pid=236046 exited with status=0!

2023-03-18IP.5___4 [DEBUG] [NUMBER] NtwkIOLink::flushSslWpending()…

2023-03-18IP.5____7 [DEBUG] [NUMBER] SSL wpending: 0

2023-03-18IP.6___3 [DEBUG] [NUMBER] NtwkIOLink::flushSslWpending()…

2023-03-18IP.6____1 [DEBUG] [NUMBER] SSL wpending: 0

2023-03-18IP.6____7 [DEBUG] [NUMBER] [FDBIO] free rbuf 4096

2023-03-18IP.7____2 [DEBUG] [NUMBER2] onTimer_T(), CR, wantRead: 1, wantWrite: 0, allowWrite: 1, allowRead: 1, m_ssl.wantRead: 0, m_ssl.wantWrite: 0, m_ssl.lastRead: 0, m_ssl.lastWrite: 0

2023-03-18IP.NUMBER8’s19 [DEBUG] [NUMBER2] Shutdown time out!

2023-03-18IP.NUMBER8’s97 [DEBUG] [NUMBER2] Close socket …

2023-03-18IP.8’s212 [DEBUG] [NUMBER2] Available Connections: 9999, concurrent conn: 0

2023-03-18IP.8’s245 [DEBUG] [NUMBER2] [.com:lsapi] calling removeSessionCb on this 0x167dc90

2023-03-18IP.8’s251 [DEBUG] [NUMBER2] removeSessionCb(), session=0x167dc98

2023-03-18IP.8’s264 [DEBUG] [NUMBER2] [T0 virtual void HttpSession::recycle()] sess seq now 23

2023-03-18IP.8’s268 [DEBUG] [NUMBER2] [.com:lsapi] HttpSession::recycle() 0x167dc90

2023-03-18IP.8’s272 [DEBUG] [NUMBER2] [.com:lsapi] [Tm] failed trylockMtRace.

2023-03-18IP.8’s310 [DEBUG] [NUMBER2] Recycle NtwkIoLink

2023-03-18 IP2.16__1 [DEBUG] [2___2] NtwkIOLink::flushSslWpending()…

2023-03-18 IP2.16__1 [DEBUG] [2____2] SSL wpending: 0

2023-03-18 IP2.3___9 [DEBUG] [NUMBER3] DNS reverse lookup: [1]: 33.bl.bot.semrush.comoglebot.com.comD

2023-03-18 IP2.3___7 [DEBUG] [NUMBER3] New client info. Check if AC_CAPTCHA: 0, Check if CIF_PENDING_CAPTCHA: 0

2023-03-18 IP2.340622 [DEBUG] [NUMBER3] [*:443] New connection from IpAddress:Port_NO.

2023-03-18 IP2.340681 [DEBUG] [NUMBER3] NtwkIOLink::setLink called pInfo is m_pClientInfo 0x161e700, m_pCrypto 0x167d168, m_pServerAddrInfo 0x15c53f0, m_remotePort 59828

2023-03-18 IP2.NUMBER01 [DEBUG] [NUMBER3] [SSL: 0x1679c98] setfd: 18

2023-03-18 IP2.NUMBER07 [DEBUG] [NUMBER3] [BIO] ls_fdbio_create bio: 0x1651168

2023-03-18 IP2.NUMBER10 [DEBUG] [NUMBER3] [BIO] setup_writes

2023-03-18 IP2.NUMBER13 [DEBUG] [NUMBER3] [SSL: 0x1679c98] toAccept

2023-03-18 IP2.NUMBER17 [DEBUG] [NUMBER3] concurrent conn: 1

2023-03-18 IP2.NUMBER21 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] NtwkIOLink::handleEvents() fd: 18, mask=25, events=1!

2023-03-18 IP2.NUMBER25 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [SSL] SSLAgain()!

2023-03-18 IP2.NUMBER28 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [SSL: 0x1679c98] accept SSL: 0x167d168

2023-03-18 IP2.NUMBER32 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [SSL: 0x1679c98] Call SSL_do_handshake, ssl: 0x167d168, ctx: 0x1594098

2023-03-18 IP2.NUMBER42 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [BIO] bio_fd_read((0x1651168:18), 0x160b126, 5)

2023-03-18 IP2.NUMBER45 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [FDBIO] alloc read buf 4096

2023-03-18 IP2.NUMBER74 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [BIO] bio_fd_read: read into buffer (0x167e130, 4096) = 307

2023-03-18 IP2.NUMBER79 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [BIO] bio_fd_read: Preserve read: 307

2023-03-18 IP2.NUMBER83 [DEBUG] [NUMBER3] [185.IADDRSS:PORT28] [BIO] bio_fd_read: Use existing buffer 5/307

2023-03-18 IP2.NUMBER87 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read(0x160b126, 5) return 5

2023-03-18 IP2.NUMBER92 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read((0x1651168:18), 0x167dfd8, 302)

2023-03-18 IP2.NUMBER95 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read: Use existing buffer 302/302

2023-03-18 IP2.NUMBER98 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read(0x167dfd8, 302) return 302

2023-03-18 IP2.3___4 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [FDBIO] bio_fd_write: 0x1651168, 4139 bytes on 18

2023-03-18 IP2.3____0 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read((0x1651168:18), 0x160b126, 5)

2023-03-18 IP2.3____4 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read: read into buffer (0x167e130, 4096) = -1

2023-03-18 IP2.3____1 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read: Read error: (11) Resource temporarily unavailable

2023-03-18 IP2.3____4 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [BIO] bio_fd_read: set retry read errno: 11

2023-03-18 IP2.3___7 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [SSL: 0x1679c98] SSL_accept SSL_do_handshake rc: -1

2023-03-18 IP2.3____5 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] [SSL: 0x1679c98] checkError returned 2, first error: error:00000000:invalid library (0):OPENSSL_internal:invalid library (0), last error: error:00000000:invalid library (0):OPENSSL_internal:invalid library (0)

2023-03-18 IP2.3____9 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] setSSLAgain(), CR, wantRead: 0, wantWrite: 0, allowWrite: 1, allowRead: 1, m_ssl.wantRead: 1, m_ssl.wantWrite: 0, m_ssl.lastRead: 0, m_ssl.lastWrite: 0

2023-03-18 IP2.3___3 [DEBUG] [NUMBER3] [185.IADDRSS:PORT2] setSSLAgain(), SW, wantRead: 0, wantWrite: 0, allowWrite: 1, allowRead: 1, m_ssl.wantRead: 1, m_ssl.wantWrite: 0, m_ssl.lastRead: 0, m_ssl.lastWrite: 0

2023-03-18 IP2.3____6 [DEBUG] [NUMBER3] [*:443] 1 connections accepted!

That is just an example. Click on the website that you think has an issue

i have done those changes mentioned and now i don’t find any logs here as

I am using Putty to access the terminal

where do i find <PID_PROCESS> ? is it id or number ?

How long it should be run the command of strace ?


Process and Their Consuming CPU is also 0

Go to ssh terminal and run this

$ nano /home/processes.py

add the folllowing content (copy paste)

#!/usr/bin/env python3

import os
import subprocess

def createlist_appcommands():
    dtfile_dir = "/usr/share/applications"
    dtfile_list = [item for item in os.listdir(dtfile_dir) if item.endswith(".desktop")]
    commands = []
    for item in dtfile_list:
        try:
            with open(dtfile_dir+"/"+item) as data:
                searchlines = data.readlines()
                command = [line for line in searchlines if line.startswith("Exec=")
                           and not "NoDisplay=true\n" in searchlines
                            ][0].replace("Exec=", "").replace("\n", "").split("/")[-1].split(" ")[0]
            commands.append(command)
        except Exception:
            pass
    return commands + [trace_symlinks(item) for item in commands if not trace_symlinks(item)== None]

def trace_symlinks(command):
    target = subprocess.Popen(["which", command], stdout=subprocess.PIPE)
    location = (target.communicate()[0].decode("utf-8")).split("\n")[0]                                                          
    check_filetype = subprocess.Popen(["file", location], stdout=subprocess.PIPE)
    filetype = (check_filetype.communicate()[0].decode("utf-8")).split("\n")[0]
    if "symbolic link" in filetype:
        return filetype.split("/")[-1].replace("' ", "")
    else:
        pass

def createlist_runningprocs():
    processesb = subprocess.Popen(["ps", "-e"], stdout=subprocess.PIPE)
    process_listb = (processesb.communicate()[0].decode("utf-8")).split("\n")
    linked_commands = [(item, item[24:]) for item in process_listb][1:]
    applist = createlist_appcommands()
    print("Processes, related to applications:\n  PID TTY"+" "*10+"TIME CMD")
    matches = []
    for item in applist:
        for i in range(0, len(linked_commands)):
            if item[:15] in linked_commands[i][1] and len(item[:15])/len(linked_commands[i][1]) > 0.5:
                matches.append(i)
    matches = sorted(matches)
    for i in range(0, len(linked_commands)):
        if i in matches:
            print(linked_commands[i][0])
    print("\nOther processes:\n  PID TTY"+" "*10+"TIME CMD")
    for i in range(0, len(linked_commands)):
        if not i in matches:
            print(linked_commands[i][0])

createlist_runningprocs()

Run this

$ python3 /home/processes.py

Post the output here (copy paste or screenshot)

OR run this command

$ ps -axjf

post results here

root@vmi123:~# nano /home/processes.py
root@vmi12
3:~# python3 /home/processes.py
Processes, related to applications:
PID TTY TIME CMD

Other processes:
PID TTY TIME CMD
1 ? 00:02:12 systemd
2 ? 00:00:00 kthreadd
3 ? 00:00:00 rcu_gp
4 ? 00:00:00 rcu_par_gp
6 ? 00:00:00 kworker/0:0H-kblockd
8 ? 00:00:00 mm_percpu_wq
9 ? 00:00:44 ksoftirqd/0
10 ? 00:22:12 rcu_sched
11 ? 00:00:03 migration/0
12 ? 00:00:00 idle_inject/0
14 ? 00:00:00 cpuhp/0
15 ? 00:00:00 cpuhp/1
16 ? 00:00:00 idle_inject/1
17 ? 00:00:04 migration/1
18 ? 00:00:31 ksoftirqd/1
20 ? 00:00:00 kworker/1:0H-kblockd
21 ? 00:00:00 cpuhp/2
22 ? 00:00:00 idle_inject/2
23 ? 00:00:04 migration/2
24 ? 00:00:34 ksoftirqd/2
26 ? 00:00:00 kworker/2:0H-kblockd
27 ? 00:00:00 cpuhp/3
28 ? 00:00:00 idle_inject/3
29 ? 00:00:04 migration/3
30 ? 00:01:18 ksoftirqd/3
32 ? 00:00:00 kworker/3:0H-kblockd
33 ? 00:00:00 cpuhp/4
34 ? 00:00:00 idle_inject/4
35 ? 00:00:04 migration/4
36 ? 00:01:23 ksoftirqd/4
38 ? 00:00:00 kworker/4:0H-kblockd
39 ? 00:00:00 cpuhp/5
40 ? 00:00:00 idle_inject/5
41 ? 00:00:04 migration/5
42 ? 00:01:28 ksoftirqd/5
44 ? 00:00:00 kworker/5:0H-kblockd
45 ? 00:00:00 kdevtmpfs
46 ? 00:00:00 netns
47 ? 00:00:00 rcu_tasks_kthre
48 ? 00:00:00 kauditd
50 ? 00:00:04 khungtaskd
51 ? 00:00:00 oom_reaper
52 ? 00:00:00 writeback
53 ? 00:00:00 kcompactd0
54 ? 00:00:00 ksmd
55 ? 00:00:00 khugepaged
102 ? 00:00:00 kintegrityd
103 ? 00:00:00 kblockd
104 ? 00:00:00 blkcg_punt_bio
105 ? 00:00:00 tpm_dev_wq
106 ? 00:00:00 ata_sff
107 ? 00:00:00 md
108 ? 00:00:00 edac-poller
109 ? 00:00:00 devfreq_wq
110 ? 00:00:00 watchdogd
114 ? 00:00:00 kswapd0
115 ? 00:00:00 ecryptfs-kthrea
117 ? 00:00:00 kthrotld
118 ? 00:00:00 acpi_thermal_pm
121 ? 00:00:00 scsi_eh_0
122 ? 00:00:00 scsi_tmf_0
123 ? 00:00:00 scsi_eh_1
124 ? 00:00:00 scsi_tmf_1
126 ? 00:00:00 vfio-irqfd-clea
128 ? 00:00:00 ipv6_addrconf
137 ? 00:00:00 kstrp
140 ? 00:00:00 kworker/u13:0
153 ? 00:00:00 charger_manager
206 ? 00:00:00 scsi_eh_2
207 ? 00:00:00 scsi_tmf_2
228 ? 00:01:28 kworker/0:1H-kblockd
229 ? 00:01:23 kworker/4:1H-kblockd
231 ? 00:18:24 jbd2/sda3-8
232 ? 00:00:00 ext4-rsv-conver
270 ? 00:03:12 systemd-journal
294 ? 00:01:30 kworker/3:1H-kblockd
300 ? 00:01:22 kworker/5:1H-kblockd
301 ? 00:01:26 kworker/1:1H-kblockd
306 ? 00:00:18 systemd-udevd
307 ? 00:07:11 loop0
309 ? 00:01:26 kworker/2:1H-kblockd
327 ? 00:00:36 jbd2/loop0-8
328 ? 00:00:00 ext4-rsv-conver
346 ? 00:00:00 cryptd
380 ? 00:00:00 jbd2/sda2-8
383 ? 00:00:00 ext4-rsv-conver
407 ? 00:00:04 systemd-timesyn
411 ? 00:00:53 accounts-daemon
412 ? 00:00:10 cron
413 ? 00:00:02 dbus-daemon
418 ? 00:00:01 firewalld
423 ? 00:00:37 irqbalance
425 ? 00:00:00 networkd-dispat
428 ? 00:00:45 rsyslogd
430 ? 00:00:04 systemd-logind
450 ? 00:00:00 polkitd
462 ? 00:00:04 systemd-network
464 ? 00:00:00 none
482 ? 00:00:01 nghttpx
504 ? 00:00:02 nghttpx
509 ? 00:00:18 lsmcd
517 ? 00:01:14 sshd
519 ? 00:00:15 opendkim
521 ? 00:01:37 lscpd
523 ? 00:00:23 lscpd
524 ? 00:05:52 lscpd
537 ? 00:16:52 redis-server
584 ? 13-02:45:25 mysqld
589 ? 00:00:27 lsmcd
590 ? 00:00:28 lsmcd
591 ? 00:00:28 lsmcd
592 ? 00:00:27 lsmcd
593 ? 00:00:27 lsmcd
660 ? 00:00:53 inetd
685 tty1 00:00:00 agetty
821 ? 00:00:31 pdns_server
886 ? 00:00:02 pure-ftpd-mysql
235145 ? 00:00:07 lsphp
235153 ? 00:00:27 lsphp
240828 ? 00:00:02 kworker/3:1-events
242798 ? 00:00:00 kworker/2:1-events
244882 ? 00:00:00 kworker/4:2-mm_percpu_wq
248408 ? 00:00:00 kworker/2:0-events
248429 ? 00:00:00 kworker/4:1-mm_percpu_wq
248435 ? 00:00:00 kworker/0:0-mm_percpu_wq
248673 ? 00:00:00 kworker/5:0-mm_percpu_wq
250722 ? 00:00:00 kworker/1:0-mm_percpu_wq
252776 ? 00:00:00 master
252777 ? 00:00:00 pickup
252778 ? 00:00:00 qmgr
252784 ? 00:00:00 dovecot
252785 ? 00:00:00 anvil
252786 ? 00:00:00 log
252787 ? 00:00:00 config
253102 ? 00:04:35 lsphp
253118 ? 00:00:02 sshd
253124 ? 00:04:20 lsphp
253125 ? 00:04:07 lsphp
253142 ? 00:00:00 systemd
253143 ? 00:00:00 (sd-pam)
253172 pts/0 00:00:00 bash
253405 ? 00:00:00 kworker/5:2
253452 ? 00:00:00 kworker/1:1-events
255952 ? 00:00:01 lswsgi
255954 ? 00:00:13 lswsgi
255955 ? 00:00:08 lswsgi
257373 ? 00:00:00 tlsmgr
257376 ? 00:00:00 stats
257442 ? 00:00:00 kworker/u12:3-events_power_efficient
257702 ? 00:00:00 kworker/0:2
257711 ? 00:00:00 kworker/3:2
258700 ? 00:00:00 litespeed
258701 ? 00:00:00 litespeed
258702 ? 00:00:01 litespeed
258703 ? 00:00:01 litespeed
258704 ? 00:00:01 litespeed
258705 ? 00:00:01 litespeed
258706 ? 00:00:01 litespeed
258707 ? 00:00:01 litespeed
259153 ? 00:00:00 kworker/u12:0-events_unbound
259836 ? 00:00:12 lsphp
260459 ? 00:00:06 lsphp
260489 ? 00:00:03 lsphp
260536 ? 00:00:07 lsphp
260578 ? 00:00:09 lsphp
260677 ? 00:00:00 kworker/5:1
260733 ? 00:00:00 kworker/2:2-mm_percpu_wq
260740 ? 00:00:00 kworker/2:3
261029 ? 00:00:00 kworker/u12:1-events_power_efficient
261124 ? 00:00:05 lsphp
261125 ? 00:00:00 sshd
261248 pts/1 00:00:00 bash
261277 ? 00:00:01 lsphp
261310 ? 00:00:07 lsphp
261431 ? 00:00:00 sshd
261448 ? 00:00:00 sshd
261449 ? 00:00:01 lsphp
261450 ? 00:00:01 lsphp
261574 ? 00:00:02 lsphp
261648 ? 00:00:01 lsphp
261649 ? 00:00:01 lsphp
261650 ? 00:00:01 lsphp
261665 ? 00:00:00 sshd
261666 ? 00:00:00 sshd
261683 pts/1 00:00:00 python3
261684 pts/1 00:00:00 ps

root@vmi1206583:~# PID TTY TIME CMD
103 ? 00:00:00 kblockd
104 ? 00:00:00 blkcg_punt_bio
105 ? 00:00:00 tpm_dev_wq
106 ? 00:00:00 ata_sff
107 ? 00:00:00 md
108 ? 00:00:00 edac-poller
109 ? 00:00:00 devfreq_wq
110 ? 00:00:00 watchdogd
114 ? 00:00:00 kswapd0
115 ? 00:00:00 ecryptfs-kthrea
117 ? 00:00:00 kthrotld
118 ? 00:00:00 acpi_thermal_pm
121 ? 00:00:00 scsi_eh_0
122 ? 00:00:00 scsi_tmf_0
123 ? 00:00:00 scsi_eh_1
124 ? 00:00:00 scsi_tmf_1
126 ? 00:00:00 vfio-irqfd-clea
128 ? 00:00:00 ipv6_addrconf
137 ? 00:00:00 kstrp
140 ? 00:00:00 kworker/u13:0
153 ? 00:00:00 charger_manager
206 ? 00:00:00 scsi_eh_2
207 ? 00:00:00 scsi_tmf_2
228 ? 00:01:28 kworker/0:1H-kblockd
229 ? 00:01:23 kworker/4:1H-kblockd
231 ? 00:18:24 jbd2/sda3-8
232 ? 00:00:00 ext4-rsv-conver
270 ? 00:03:12 systemd-journal
294 ? 00:01:30 kworker/3:1H-kblockd
300 ? 00:01:22 kworker/5:1H-kblockd
301 ? 00:01:26 kworker/1:1H-kblockd
306 ? 00:00:18 systemd-udevd
307 ? 00:07:11 loop0
309 ? 00:01:26 kworker/2:1H-kblockd
327 ? 00:00:36 jbd2/loop0-8
328 ? 00:00:00 ext4-rsv-conver
346 ? 00:00:00 cryptd
380 ? 00:00:00 jbd2/sda2-8
383 ? 00:00:00 ext4-rsv-conver
407 ? 00:00:04 systemd-timesyn
411 ? 00:00:53 accounts-daemon
412 ? 00:00:10 cron
413 ? 00:00:02 dbus-daemon
418 ? 00:00:01 firewalld
423 ? 00:00:37 irqbalance
425 ? 00:00:00 networkd-dispat
428 ? 00:00:45 rsyslogd
430 ? 00:00:04 systemd-logind
450 ? 00:00:00 polkitd
462 ? 00:00:04 systemd-network
464 ? 00:00:00 none
482 ? 00:00:01 nghttpx
504 ? 00:00:02 nghttpx
509 ? 00:00:18 lsmcd
517 ? 00:01:14 sshd
519 ? 00:00:15 opendkim
521 ? 00:01:37 lscpd
523 ? 00:00:23 lscpd
524 ? 00:05:52 lscpd
537 ? 00:16:52 redis-server
584 ? 13-02:45:25 mysqld
589 ? 00:00:27 lsmcd
590 ? 00:00:28 lsmcd
591 ? 00:00:28 lsmcd

You have 19 processes running php. Run this to kill them

killall lsphp

Disable all websites and leave one. Now test if yous ee same error