CyberPanel Community

ClosedHow to increase Inodes limit

AK
Abhishant Kumar #1

I’m facing a low storage issue in cyber panel form for many days and after searching many blogs, and videos and talking to many people. I found that the low storage problem is because my Inodes are 100% used.

In some situations my phpmyadmin is not opening then, i delete some cache and files then after some Inodes were free and then, phpmyadmin start working normally. But, I know this is not the solution.

I want to increase the Inode limit. Anyone, please help me to increase the limit of Inodes so, I didn’t face any problem of low storage in your device error in the future? I have still 52% storage available. But, inodes is 100% used.

54 replies
jo
josephgodwinke #2

Welcome @abhishant Happy you are here

  1. This maybe an indication of too many log files:
  • Run this command in your terminal to remove all logs generated rm -rf /usr/local/lsws/logs/*
  • Go to https://SERVER_URL:7080/index.php#view/confMgr.php?m=serv&p=log and setup Keep Days which specifies how many days the access log file will be kept on disk, Compress Archive allow compressing rotated log files in order to save disk space. Also reduce the rolling size of Access log
  1. This maybe an indication of a process opening too many files than the web server allows. Follow this guide

Additional context: Maybe you need to apply cgroups for your clients

AK
Abhishant Kumar #3

I run this command for log file removal but still server Inodes is full. I have to host a
client website which is in WordPress, but, I’m not able to do it because of this low space error. My ec2 volume space is free but, Inodes is full so, sometimes my PHPMyAdmin not starting, and facing many issues because of this issue. You can see this in the image. I deleted one website then some Inodes are free and now I"m able to open phpmyadmin. But, this is not the solution. I have to host a website.
Screenshot 2022-10-24 124004

jo
josephgodwinke #4

Solutions 1:

Set system-level file descriptor limits:

# check the current process-level file descriptor limit
$ more /proc/sys/fs/file-max  # or ulimit -n
# if its lower than 65000
$ echo 65000 > /proc/sys/fs/file-max
# adjust no. of processing units available
$ echo "10000" > /proc/sys/fs/nr_open
# assuming you are running linux kernel 2.2 adjust no. of maximum files that can be stored in file system
$ echo 65535 > /proc/sys/fs/inode-max
# reboot server
$ reboot
jo
josephgodwinke #6

This confirms that you have increases the number of open files allowed by processes on your system. Kindly follow the instructions I posted earlier or you can run this command to maximise the limit (not recommeded but should solve this):

$ ulimit -u unlimited

Note: For the purpose of helping the community always copy-paste the error you get alongside the screenshot to help others search the issue and find your question

AK
Abhishant Kumar #7

Thanks for your help and support Sir, please provide me with the method by which I will increase the limit of Inodes. Because now I have to delete the old website to host a new one because Inodes is 100% used. Because of this problem, I’m not able to upload any plugin, image or file through cyber panel. This is a very big issue for me. So, please help me to increase the Inodes limit.

jo
josephgodwinke #8

@abhishant we have already i idnetified the issue together here.

With this said, kindly follow the steps i gave here if all fails run the command here

Just open your terminal and run them as they are. I have added comments for you to understand why you have to run each and every command.

AK
Abhishant Kumar #9

I run all commands and this will fix the WordPress installation issue but, the main issue is Inodes. I have to delete my old website for the installation of new website. Do we google meet or anydesk to shwo what the issue is.

jo
josephgodwinke #10

Use control groups aka cgroups:

  1. verify that you are running cgroups v2:
$ ls /sys/fs/cgroup/cgroup.controllers
  1. Check available features to control:
$ cat /sys/fs/cgroup/cgroup.subtree_control
  1. Adjust TasksMax - the number of tasks that can be started by a user. This ensures that the number of tasks accounted for the user stays below a specific limit. A reasonable value might be 2500.
# make a drop-in file
$ mkdir -p /etc/systemd/system/user@.service.d
$ cat > /etc/systemd/system/user@.service.d/delegate.conf << EOF 
[Service] Delegate=tasks cpu io memory pids cpuset 
EOF
$ systemctl daemon-reload
  1. Configure the user you want to control by using linger, enter the following:

loginctl enable-linger admin

You can see if a user is lingered by entering:

loginctl user-status admin

  1. To enable cgroups at the server level, in OpenLiteSpeed WebAdmin, navigate to Server Configuration > Security , edit the CGI Settings group and enable cgroups by setting it to on

Note: Avoid systemctl configuration if you see it in any tutorial in the web

jo
josephgodwinke #14

You do not have cgroups v2. Setup cgroups:

1.Using drop-In files method

For each user you wish to configure, you will need to create a directory in mkdir /etc/systemd/system/user-UID.slice.d. e.g.mkdir /etc/systemd/system/wordpressuser-1001.slice.d

$ cd /etc/systemd/system/wordpressuser-1001.slice.d
echo "[Slice]" >> 50-TasksMax.conf
echo "TasksMax=10%" >> 50-TasksMax.conf

Validate the file changes by running cat 50-TasksMax.conf

To use cgroups with OpenLiteSpeed you need to install glib as root.

RedHat/Centos: sudo yum install glib2
Ubuntu: sudo apt-get install glib2.0

In OLS WebAdmin, configure OpenLiteSpeed to have CGI apps use the user ID of the owner of the file. Navigate to Configuration > Virtual Hosts > and select the virtual host you wish to manage. In the Basic tab, modify the Security entry and set the ExtApp Set UID Mode to CGI File UID

To enable cgroups at the server level, in OpenLiteSpeed WebAdmin, navigate to Server Configuration > Security , edit the CGI Settings group and enable cgroups by setting it to on

Note: Avoid systemctl configuration if you see it in any tutorial in the web

Ak
Abhishant kumar #15

Please explain were is the option is - In OLS WebAdmin, configure OpenLiteSpeed to have CGI apps use the user ID of the owner of the file. Navigate to Configuration > Virtual Hosts > and select the virtual host you wish to manage. In the Basic tab, modify the Security entry and set the ExtApp Set UID Mode to CGI File UID

jo
josephgodwinke #18

Click on Virtual Hosts from left sidebar > example.net In the Basic tab, modify the Security entry and set the ExtApp Set UID Mode to CGI File UID

Ak
Abhishant kumar #20

I have to do this for all domains? Because, I have more than 49 domains hosted on this panel. So, I have to do this process for all domains.

Question: Does this process will increase Inodes or I didn’t face Inodes exhaust problem in future after doing this process. Or if i was wrong then please tell me about this.

jo
josephgodwinke #21

cgroups isloates resources by accounts. So this has to be done for all users and virtual hosts if required.

If you set a CPUQuota of 10% for a user, all accesses by the user must not exceed 10%. This includes those in OLS and user login sessions. All will be shared fairly by the operating system. If you start more processes, the total percentage for the user does not change, but each process runs a bit slower so as to avoid monopolizing the CPU.

OLS setup starts with User Configuration via Drop-in files (recomended) and finally an OLS setup including the VHost Configuration

Ak
Abhishant kumar #22

Screenshot 2022-10-25 163353

Again Inodes uses increased by 3% without hosting any domain. When Inodes was 100% uses, I’m not able to do any thing then i faced server error 505. After that I increased volume stroage by 10GB then Inodes changed to 100% uses to 86% uses. But, again it was increasing automatically. I’m afraid now because how much storage i have to increase. Because when i check storage uses then its shows only 37% uses, 63% is waist and i haved to pay for 63% storage.

I increased 30Gb one by one because Inodes was always full in few days after incresing the volume. This is very serious issue. I will provide you the screenshot of list of files which is using most Inodes. The in left side of file name is Inodes uses.

You can also check my storage uses in below screenshot.

Screenshot 2022-10-25 164313

Please provide me method form which i find out the useless file which is eating more Inodes. And any method to modify the Inodes.

Ak
Abhishant kumar #23

Now Inodes uses increase by 3% more. Means in few hours Inodes uses increased by 6% and after some time Inodes was full and again I’m in big trouble because, when Inodes was full I’m not able to do any things. After that it will again gives me server error 500 while opening 8090 port. This is big issue, I’m very afraid about this issue. I already increased 30 to 40 Gb in few weeks because of Inodes issue. But, still I’m facing this issue. Do you please check why this problem was happening.

Screenshot 2022-10-25 221116

I hope you will understand my situation. After few hours Inodes uses become full and I’m not able to do anything. I’m deleted many wordperss sites but still it Inodes uses was increasing.

jo
josephgodwinke #24

Hello, would you rollback the server to a later date i.e when you first installed cyberpanel?

tb
tbaldur #25

Have you checked your sessions folder, if it’s just filled with thousands of small files?

Ak
Abhishant kumar #26

Do you please help me how to find the session folder and delete all unnecessary session files.

Ak
Abhishant kumar #27

How to do that. Please guide me to rollback the server later date.

tb
tbaldur #29

Inside those folders:

/var/lib/lsphp/session/lsphp71
/var/lib/lsphp/session/lsphp72
/var/lib/lsphp/session/lsphp73
/var/lib/lsphp/session/lsphp74
/var/lib/lsphp/session/lsphp80
/var/lib/lsphp/session/lsphp81

It should have many of those files, because devs never correctly set up the clean of sessions (last time I checked).
If that’s the case, I will post the fix here (but you will need to re-apply every time you update cyberpanel)

Ak
Abhishant kumar #30

After running below command it shows me unlimited number of files. Please help me to delete these files

find /var/lib/lsphp/session/lsphp80

Still it was showing me files. How to delete these files

jo
josephgodwinke #32

@usmannasir fixed this. Just run this if its PHP sessions

bash <(curl https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/CPScripts/setup_php_sessions.sh)

Finally run

# Reload systemd manager configuration. This will rerun all generators (see systemd.generator(7)), reload all unit files, and recreate the entire dependency tree. While the daemon is being reloaded, all sockets systemd listens on behalf of user configuration will stay accessible.
systemctl daemon-reload
systemctl restart lsws
jo
josephgodwinke #36

The script I posted Installs a PHP Session cleaning cronjob. Try and reboot the server

Ak
Abhishant kumar #37

I checked the Inodes uses after running the PHP Session cleaning cronjob script and restarting the server. Its still same.

Screenshot 2022-10-26 003402

jo
josephgodwinke #38

Do this manually. Run this command to delete all files:

rm -f /var/lib/lsphp/session/lsphp71/* && rm -f /var/lib/lsphp/session/lsphp72/* && rm -f /var/lib/lsphp/session/lsphp73/* 

rm -f /var/lib/lsphp/session/lsphp74/* && rm -f /var/lib/lsphp/session/lsphp80/* && rm -f /var/lib/lsphp/session/lsphp81/*

Run these commands separately

jo
josephgodwinke #40
  1. How many websites are you running?
  2. What technologies are these websites running?
Ak
Abhishant kumar #41
  1. How many websites are you running?
    Ans. 45 websites hosted in this cyberpanel.
  2. What technologies are these websites running?
    php, html & wordpress
tb
tbaldur #42

He didn’t fix it sadly as I see there.

imagem
imagem

He ignored the setting from php.ini file by hardcoding a value of 120.

So the way he “fixed it” will only delete files that were modified exactly 120 minutes ago when the script runs.
Won’t delete files that were changed 119, 118, 121, 150, and such minutes ago.

The php.ini settings use values in seconds, ctime in minutes.

To properly fix it, edit this file:
/usr/local/CyberCP/bin/cleansessions

With this content:

#!/bin/bash
for version in $(ls /usr/local/lsws|grep lsphp); do echo ""; echo "PHP $version"; session_time=$(/usr/local/lsws/${version}/bin/php -i |grep -Ei 'session.gc_maxlifetime'| grep -Eo "[[:digit:]]+"|sort -u); find -O3 "/var/lib/lsphp/session/${version}" -ignore_readdir_race -depth -mindepth 1 -name 'sess_*' -type f -cmin +$((session_time / 60)) -delete; done

This will properly read the session_time that is set up in your php.ini (default 14400 seconds = 240 minutes = 4 hours) and will use that value to delete sessions older than that time.

You can then also execute it directly by running:
bash /usr/local/CyberCP/bin/cleansessions

You can edit the setup_php_sessions.sh to reflect those changes.
But keep in mind that every time you run the cyberpanel update script, this changes will be replaced (unless you use that code to make a script outside of cyberpanel logic and crontab it, which I recommend)

Ak
Abhishant kumar #43

Getting this after running the command.

tb
tbaldur #44

Looks like you have more files then the shell can process.

You can bypass it by running a find and delete, but that will be slower as you have a lot of files.

Perhaps try simply deleting the folder all together and recreating it, not sure but that should do.

rm -rf /var/lib/lsphp/session/lsphp73; mkdir -p /var/lib/lsphp/session/lsphp73; chmod 1733 /var/lib/lsphp/session/lsphp73
tb
tbaldur #45

Sorry, this double-pasted on the forum.
Fixed it above, but also adding it here:

for version in $(ls /usr/local/lsws|grep lsphp); do echo ""; echo "PHP $version"; session_time=$(/usr/local/lsws/${version}/bin/php -i |grep -Ei 'session.gc_maxlifetime'| grep -Eo "[[:digit:]]+"|sort -u); find -O3 "/var/lib/lsphp/session/${version}" -ignore_readdir_race -depth -mindepth 1 -name 'sess_*' -type f -cmin +$((session_time / 60)) -delete; done
jo
josephgodwinke #46

Hello @tmoore

Just checked. good catch. You should propose this change as pull request to the CyberPanel repo.

tb
tbaldur #47

Me and others already pointed it out, a lot of times, it was ignored so, won’t lose my time again with it

Edit: I actually only resumed coming back to forum because I see that you are actively helping people, something that in my opinion, should relay to the guys who make the software.

So I decided to also give my help when I can, you’re going a great job

jo
josephgodwinke #49

@tmoore Thanks. I migrated most of my clients over to CyberP based servers so now am heavily invested in making this panel work, understanding every single issue with it and getting involved solving them

@stay How is the load now?

Ak
Abhishant kumar #52

Still running. How much its time its take to delete folder.
Screenshot 2022-10-26 011059

jo
josephgodwinke #53

Removing a file or directory is an atomic operation requiring a single system call, so if the rm command hangs, it’s because the kernel is stuck.

Restart the server and repeat that task. Its recommended to stop some services to reduce load if its offpeak for your clientele.For example systemctl stop lscpd once you are done with running the commands start LSCPD systemctl start lscpd

Ak
Abhishant kumar #54

Screenshot 2022-10-26 014631

Now process is completely done and I very satisfied by the result because, I get what i want. Very Very Thanks for your time and support.

Screenshot 2022-10-26 020724

I have one question that, How i stop the making of session file or how i set the cron job for deletation of session file of php80. Because session file is only created in php80. Now in other version of php

tb
tbaldur #55

Editing the cleansessions above is enough (runs twice per hour), but, just in case to be sure, you can add another cronjob outside of cyberpanel logic. That way you can safely update cyberpanel in the future:

sh <(curl https://raw.githubusercontent.com/tbaldur/cyberpanel-mods/main/cyberpanel_sessions_cronjob.sh || wget -O - https://raw.githubusercontent.com/tbaldur/cyberpanel-mods/main/cyberpanel_sessions_cronjob.sh)

This will create the script to delete the sessions and set up crontab to run the said script every hour at the 17th and 47th minute.

Ak
Abhishant kumar #56

Thank you very much Tmoore, I’m facing this issue form many weeks and i know what was the problem but, I have not any knowledge of Linux. So, I’m not able to fix the issue and i’m increasing the storage day by day. You solved this issue in few hours.

Thank you very much tamoore. I appreciate your knowledge and helpful nature.

I’m also want to learn Linux and server management. Do you please guide me from were i can start learning.

Sign in to reply