MariaDB errors on Ubuntu 20.04 ONLY when using CyberPanel

Hi

Been playing around for two days as my Plesk license runs out shortly and thought i would see what the alternatives where (have used many different OS Panels before)

Anyways i have come across a weird issue in that MariaDB seems to be not working correctly.

Now i will state i first tried yesterday and did not face this issue at all, however after wiping VPS and starting afresh today for a final setup test the issue is apparent on 2 fresh installs i did today.

Hetzner Cloud Vps (test environment)
Ubuntu 20.0.4
Fresh install of Script
Added CSF with my normal config (tested on and off)
Modsecurity rules active (tested on and off)

I create a domain and upload 2 files

Kickstart.php & The backup of my joomla site in .jpa format (this is from Akeeba Backup)

When i goto the Akeeba restore as i have done many times on Plesk as well as yesterday on Cyber Panel, It shows an error:

No Database server available

Now normally this will show MySql as an option but it does not see anything.

So i tried:

Systemctl Status mariadb and it shows these errors

Looking for ‘mysql’ as: /usr/bin/mysql
Looking for ‘mysqlcheck’ as: /usr/bin/mysqlcheck
Version check failed. Got the following error when calling the ‘mysql’ command line client
ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: NO)
FATAL ERROR: Upgrade failed
ERROR 1045 (28000): Access denied for user ‘root’@‘localhost’ (using password: NO)

So i am not sure if something has been changed on the install script since yesterday but i cannot connect to any local database server

Yesterday using the exact same setup steps and options this was working perfectly. But today after a fresh install this no longer works so has something gone wrong on the install script?

As an update Using the exact same steps and settings as above but using Centos 8, this error does not appear and the restore script works fine, so not sure whats happened to the Ubuntu 20 install script since yesterday

When I run systemctl status mariadb on numerous fresh installs of CP with Ubuntu 20.04, i get this error message.

root@intel-high-freq:~# systemctl status mariadb
● mariadb.service - MariaDB 10.3.34 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-17 15:47:55 UTC; 1h 55min ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 959 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 960 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 962 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
    Process: 1191 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 1198 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
   Main PID: 1037 (mysqld)
     Status: "Taking your SQL requests now..."
      Tasks: 37 (limit: 2274)
     Memory: 103.6M
     CGroup: /system.slice/mariadb.service
             └─1037 /usr/sbin/mysqld

Mar 17 15:47:53 intel-high-freq systemd[1]: Starting MariaDB 10.3.34 database server...
Mar 17 15:47:54 intel-high-freq mysqld[1037]: 2022-03-17 15:47:54 0 [Note] /usr/sbin/mysqld (mysqld 10.3.34-MariaDB-0ubuntu0.20.04.1) starting as process 1037 ...
Mar 17 15:47:55 intel-high-freq systemd[1]: Started MariaDB 10.3.34 database server.
Mar 17 15:47:55 intel-high-freq /etc/mysql/debian-start[1213]: Upgrading MySQL tables if necessary.
Mar 17 15:47:55 intel-high-freq /etc/mysql/debian-start[1221]: Looking for 'mysql' as: /usr/bin/mysql
Mar 17 15:47:55 intel-high-freq /etc/mysql/debian-start[1221]: Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
Mar 17 15:47:55 intel-high-freq /etc/mysql/debian-start[1221]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
Mar 17 15:47:55 intel-high-freq /etc/mysql/debian-start[1221]: FATAL ERROR: Upgrade failed
Mar 17 15:47:55 intel-high-freq /etc/mysql/debian-start[1263]: Checking for insecure root accounts.
Mar 17 15:47:55 intel-high-freq debian-start[1268]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

Conversely, on a fresh install of AlmaLinux i get this:

mariadb.service - MariaDB 10.3 database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-03-17 16:52:37 UTC; 34min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
 Main PID: 55866 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 38 (limit: 12431)
   Memory: 99.0M
   CGroup: /system.slice/mariadb.service
           └─55866 /usr/libexec/mysqld --basedir=/usr

Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: See the MariaDB Knowledgebase at http://mariadb.com/kb or the
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: MySQL manual for more instructions.
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: Please report any problems at http://mariadb.org/jira
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: The latest information about MariaDB is available at http://mariadb.org/.
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: You can find additional information about the MySQL part at:
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: http://dev.mysql.com
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: Consider joining MariaDB's strong and vibrant community:
Mar 17 16:52:37 vultr.guest mysql-prepare-db-dir[55534]: https://mariadb.org/get-involved/
Mar 17 16:52:37 vultr.guest mysqld[55866]: 2022-03-17 16:52:37 0 [Note] /usr/libexec/mysqld (mysqld 10.3.28-MariaDB) starting as process 55866 ...
Mar 17 16:52:37 vultr.guest systemd[1]: Started MariaDB 10.3 database server.

And if I run apt install mariadb-server on a completely fresh Ubuntu 20.04 install, without CyberPanel, I get the following:

root@guest:~# systemctl status mariadb
● mariadb.service - MariaDB 10.3.34 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-17 17:41:13 UTC; 12s ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
   Main PID: 45595 (mysqld)
     Status: "Taking your SQL requests now..."
      Tasks: 31 (limit: 2274)
     Memory: 64.0M
     CGroup: /system.slice/mariadb.service
             └─45595 /usr/sbin/mysqld

Mar 17 17:41:13 guest mysqld[45595]: 2022-03-17 17:41:13 0 [Note] /usr/sbin/mysqld (mysqld 10.3.34-MariaDB-0ubuntu0.20.04.1) starting as process 45595 ...
Mar 17 17:41:13 guest systemd[1]: Started MariaDB 10.3.34 database server.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45645]: Upgrading MySQL tables if necessary.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: Looking for 'mysql' as: /usr/bin/mysql
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: This installation of MariaDB is already upgraded to 10.3.34-MariaDB.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: There is no need to run mysql_upgrade again for 10.3.34-MariaDB.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: You can use --force if you still want to run mysql_upgrade
Mar 17 17:41:13 guest /etc/mysql/debian-start[45689]: Checking for insecure root accounts.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45694]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables

What is wrong with CyberPanel on Ubuntu?

It seems you have changed the mysql root password for the ubuntu server

I haven’t done anything - as I said, it is a completely fresh install of Ubuntu 20.04 and CyberPanel. Nothing else was done. So, if that’s the case, CyberPanel is doing something wrong with passwords when installing on Ubuntu (18.04 and 20.04)

This has also been reported in many other threads over the years - just search for ERROR 1045 (28000): Access denied for user 'root'@'localhost'

• Problem installing cyberpanel on 18.04 (mysql error) - Support and Discussion / General Discussion - CyberPanel Community
• CyberPanel Ubuntu Bug Reports - CyberPanel Project / Bug Report - CyberPanel Community
• Some Dont’s After Installing CyberPanel! - Docs / Unfiled - CyberPanel Community
• FATAL ERROR: Upgrade failed - Support and Discussion / Database Server - CyberPanel Community
• Access denied for user - Support and Discussion / Database Server - CyberPanel Community

Websites still work, so its not catastrophic, but it is clearly an issue that should be resolved.

Just try to restart the whole server and then see if still those notices are shown.

I’ve done that and many other things. It happens on fresh installs, old installs that have been upgraded etc…There is quite clearly a problem with CP.

I’m installing CP on a fresh 18.04 server now and will report back what happens. I expect the same issue.

please try it on 20.4 version and not older.

The problem seems to be twofold:

1. Something inherent to MariaDB on Ubuntu.

The output from systemctl status mariadb on Ubuntu without CyberPanel says

Mar 17 17:41:13 guest /etc/mysql/debian-start[45645]: Upgrading MySQL tables if necessary.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: Looking for 'mysql' as: /usr/bin/mysql
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: This installation of MariaDB is already upgraded to 10.3.34-MariaDB.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: There is no need to run mysql_upgrade again for 10.3.34-MariaDB.
Mar 17 17:41:13 guest /etc/mysql/debian-start[45649]: You can use --force if you still want to run mysql_upgrade

which makes it seems like it is trying to run mysql_upgrade when it shouldn’t be.

2. CyberPanel makes it try to run mysql_upgrade without a password

When I run cat /etc/cyberpanel/mysqlPassword on AlmaLinux and Ubuntu, both have a password set by default. But the error message says 'root'@'localhost' (using password: NO)

This leads me to believe that it has configured something to try running mysql_upgrade without a password, but needs one.

Seems relevant.

Just try to connect SSH and connect mysql sudo mysql -uroot -p

Sometimes, connecting back and forth solves such notice warnings.

I appreciate the attention, but PLEASE read my comments more carefully.

This has been happening on 20.04 - fresh installs, old installs that have been upgraded, etc… I also just installed CyberPanel on 18.04 to see if it is a 20.04 issue or Ubuntu issue.

I have confirmed that the issue happens with both versions of Ubuntu when CyberPanel is installed. It is clearly messing with something.

Yes, I can connect just fine with the password. So the question is whether this is just an issue with running

systemctl status mariadb

whereby it is trying to run mysql_upgrade, and then CyberPanel makes it try that without a password.

Or perhaps it is a deeper issue. I dont know how to investigate either though.

Also, as shown in the various other topics that I linked to above, the error presents during CyberPanel install as well.

Yes, I am getting what you are trying to explain. Its just that I am myself checking the same at my part and I am not able to reproduce this same issue

Which hosting you are using for this?

That’s odd. I have tried it on Vultr and another smaller VPS provider. No issue with AlmaLinux. Half issue with 20.04 and 18.04 without CP, full issue with CP installed.

I can’t see how the provider would matter though - Ubuntu is Ubuntu, especially after updating everything.

Which provider are you using where you dont get the issue with a new install?

You statement “Ubuntu is Ubuntu” is not valid when you get ubuntu installed using cloud hosting provider’s image. Images by hosting companies are modified in many cases. To point, Alibaba cloud hosting makes a heavy modifications to all their linux OS images.

Anyways, I am using Amazon AWS and there it isn’t showing such issue. I will try in Digital Ocean and see

Interesting, I didn’t know that. Thanks.

I’ll continue investigating as well. I will also try to figure out how to file some report with MariaDB.

Currently having same issue on 20.04 (linode)

Is there a resolution?