How fix it?
Error_Log
2019-10-29 08:11:57.018876 [INFO] [westmai.990]: locked pid file [/tmp/lshttpd/westmai.sock.pid].
2019-10-29 08:11:57.018884 [INFO] [westmai.990] remove unix socket for detached process: /tmp/lshttpd/westmai.sock
2019-10-29 08:11:57.018896 [ERROR] [westmai.990]: Failed to listen socket [/tmp/lshttpd/westmai.sock]: Too many open files
2019-10-29 08:11:57.018906 [WARN] [UDS://tmp/lshttpd/westmai.sock] Can not start this external application.
2019-10-29 08:11:57.018915 [INFO] [UDS://tmp/lshttpd/wpeasyt.sock] Error Timeout, restart and try again!
You can try this and see if you are having a file limit problem with LSWS.
at ssh commandline:
lsof | awk ‘{ print $2; }’ | sort -rn | uniq -c | sort -rn | head
In my case once the cache got “warmed up” by the crawlers, it was over the limit and I would get the “Too many open files” error.
@Redmound what does that command do? I see the following when running it:
adoot@server ~]# lsof | awk ‘{ print $2; }’ | sort -rn | uniq -c | sort -rn | he
48074 22427
690 8638
486 19169
455 993
371 603
238 1020
197 30828
174 22495
171 22577
155 31887
You can try this and see if you are having a file limit problem with LSWS.at ssh commandline:
lsof | awk ‘{ print $2; }’ | sort -rn | uniq -c | sort -rn | head
In my case once the cache got “warmed up” by the crawlers, it was over the limit and I would get the “Too many open files” error.
It just shows the number of open file handles by PID number…I think memory handles are counted.
I still have crashing issues…I don’t know why no one else seems to.
Specifically, the cache files from the wordpress sites I have create a large number of open files. This leads to the “too many open files error”. The wordpress site does have about 14k posts, but not a lot of traffic. With OpenLiteSpeed the cache crawler is always on.
Also the cache does not really delete items it just marks them expired. So I get about a week.
this shows the number of cache items:
lsof -n -l +D ‘/usr/local/lsws/cachedata’ | wc -l
I know that Cpanel sets the cache file number very high…but I have yet to get something working.
The newer OLS also has a system that leaves some workers open after a restart of LSWS…for a more seemless restart. But this has caused the file limit even when running a cron to restart LSWS.
The “too many open files” issue causes PID errors, Certificate errors, 503 errors, 500 errors…pretty much making all users unhappy.
@Redmound not sure if this will help [FIXED] Too many open files error on LiteSpeed Web Server | BaseZap
Yeah…I have been through those and more.
I am on Centos 7 and systemd ignores the limits.conf…
some of those settings on the first link actually crash sysctl after the latest update…
I thought I had it beat…but it crashed this week again and I was restarting LSWS from the airport with my phone.
Obviously, the MariaDB install setup does not have the same problem the “48074 22427” from the above example is most likely the mysql PID.
I have not been able to figure out how to get OLS to do the same with the cache files. The file handles seem to grow 2k a day so it does not take long before I hit the built in Centos limits.
please try go to webadmin console , set max connection and max SSL connection to higher number like 20k and 10k , run /usr/local/lsws/bin/lswsctrl restart
then run
cat /proc/PID_OF_OLS/limits | grep files
to see current open file limits
I did that a long time ago…but the server never gets that kind of connection rate.
The too many files error happens to cache files and pid files.
the limits for lshttpd - main, lshttpd - #01 and any of the lsphps are:
Max open files 655350 655350 files
503 and other errors and a std error log full of this with some pids mentioned
------------------>
/usr/local/lsws/cachedata/priv/c/3/9/c3974995c0994ce4: open() failed: Too many open files
/usr/local/lsws/cachedata/priv/d/3/5/d35fc04963467f7b: open() failed: Too many open files
/usr/local/lsws/cachedata/priv/b/f/1/bf1457db16fd0cd5: open() failed: Too many open files
/usr/local/lsws/cachedata/priv/6/d/2/6d2977c5a9a4b681: open() failed: Too many open files
/usr/local/lsws/cachedata/priv/b/a/d/bad2aa1785d4e08c: open() failed: Too many open files
/usr/local/lsws/cachedata/priv/e/7/4/e74eeec3967ecc0d: open() failed: Too many open files
/usr/local/lsws/cachedata/priv/3/0/1/301fe4c9bbf1a310: open() failed: Too many open files
try this , may help ?
sed -i ‘s|#DefaultLimitNOFILE=|DefaultLimitNOFILE=655350|g’ /etc/systemd/system.conf
this should set systemd limits to 655350
I had bumped up lsws.service , lscpd.service and the user limits (all websites and nobody)…
I did as you said…I may have done that before but there was a recent systemd update.
It crashed again today after a bot hit a bunch of links and filled the cache after an hour or so.
how many traffic you serve ? 655k that’s a lot of files
There is not a lot of traffic (10k a day) and I don’t consider 19k post on wordpress that big either. The 655k apparently is what cloudlinux / cpanel /litespeed is set at (which is overkill for me).
I am just trying to stop the “crashing” without giving up LScache for WP.
I really think the Cyberpanel install of litespeed is not addressing the low centos default nofile limits. Perhaps OLS does not have the same limit adjustments that ENT does. It is the only reason I posted this HERE since the problem has to do with OLS and SYSTEM file limits.
At first crashes traced back to indexing bots. I could run a link checker on the big WP site and cause a crash in a few hours. After adjustments i could get through without “crashing” and the file handles for lshttpd reached 28K.
But I still have “crashes”…even with daily LSWS restarts (cron).
When I say “crashes” I mean very odd server issues (some 503s, some 404s, some certificate errors, some email timeouts(cert issue) and rspamd not working. Without an OLS automatic restart that actually fixes things.
The fix is to stop LSWS and then start LSWS (systemctl restart lsws does not fix the issue).
I am pretty certain it is the cache files…but I have clients on the system now that use webmail 24/7 and every crash / restart turns into a complaint. So forcing a “crash” has become harder.
RDL
well , sorry I don’t have any better idea , but if you wish , please submit ticket to litespeedtech , I guess OLS dev could have take look