Sites went down after updating cyberpanel - can still access 8090, but sites all down 500 error

I just noticed sites being slow today and found out about the hack, but I may have missed the solution for my current problem.

Sites were working, but I could not SSH. I got to cyberpanel dashboard and saw high cpu. I rebooted the server and then could SSH in and I updated to the current cyberpanel version. Part way through the update, all of my sites went down and I got a 500 server error. But I can still access port 8090 and see the dashboard.

Any advice? My oldest backup of the server is from 24 hours ago.

So, I can access SSH, and I can access the cyberpanel dashboard, but all sites are giving me a 500 server error currently unable to handle this request.

Any suggestions?

My backup from 24 hours ago also is now throwing the 500 server error for every website. I’m confused. But I can access the 8090 dashboard and ssh.

Updating to the latest cyberpanel version will NOT remove the malware on your server, this is something people seem to miss.

The steps to recovery should be

1. Stop the evil people from getting in. There are many moral less evil scum of earth that have found out about this exploit and want to spread their evil. So the first step is to stop the evil which is by updating to latest version of Cyberpanel if possible.

2. Once your server is plugged from burden on earth people trying to hack you. Remove the malware that’s running on your system. Find out what malware it is, coin miner or encrypted ransomware.

3. Now, do a complete clean re-install of the OS and latest cyberpanel version and restore your sites.

4. You can do what I do, stop lscpd from running. Without a cyberpanel running, there is nothing to exploit. You can also use other security measures to protect yourself in future.

With malware running on your server, cyberpanel update isn’t going to fix anything but stop future hacking attempts

I’ve updated to the latest version of cyberpanel.

Is the 500 server error that started when I ran the update just because people are trying to continue hacking mysites from the malware on there? Very odd timing, but my sites will not come back even after restoring my backups.

I’ll work on finding the malware and removing it, assuming I can.

Since some of my sites don’t appear to have recent backups. I have access to ftp, so I can download all the raw files and then upload them manually once recovered. Ouch.

Does it look like the hack has directly affected each of the website directories? In other words, if I do a fresh OS install, but then recover the websites from a current backup, will I just be re-infecting the new install?

I was in the process of migrating sites to a fresh install, but after doing another cyberpanel update, the sites have come back online. I’m wondering if there was a new release pushed out in the last hour or two? Either that, or something I changed got rid of the 500 server error for every site.

As part of this exploit, is it possible that there is malware still installed on my server or in my website directories?

My sites are back to being down with 500 errors. I suspect it is something with php not being able to run for whatever reason. I’m trying to migrate all of my sites, but it turns out the backup function does not appear to have stored the mysql databases associated with each file.

Any advice on the easiest way to back up and restore all of the mysql databases without having access to phpmysql?

I can access the cyberpanel dashboard.

This is so frustrating.

If you still have access to CyberPanel, don’t you also have access to Database > PHPMYAdmin? You could select and export the database there. In addition, you would have to compress and download the public_html directory of the web pages. With these two files you can restore the websites.

Of course, you also need to know which plugins you have used. But the data and perhaps also the database have probably already been compromised.

phpmyadmin does not load - I get the 500 error I get from all of my other sites. I have no idea why the panel loads and static html pages load, but everything else gives me an immediate 500 error - maybe php won’t load properly?

Perhaps files have already been encrypted? Do you still have access to wp-admin from WordPress if you used it?

Nope. I can only access the cyberpanel dashboard, create and download backups from those options, and static pages on my websites, which does not include wp-admin.

If the databases are corrupted / infected, then this all may be for naught anyway. Basically all of the sites I host for customers are going to have to be redone from scratch.

No VPS snapshots taken or backup plugins such as UpdraftPlus with external backup used?

You can give this a try if you still have ssh access: GitHub - elwizard33/Cyberpanel-Server-Wizard-Cleanup: These scripts are designed to help identify and clean potential malware and ransomware infections on CyberPanel servers it will check for the attack related files and processes and try to get rid of them

Sometime ERROR 500 after copy backup data back to server is about wrong file permissions in the /home directory of your websites. You can try to fix it with the build in FileManager and the Button “Fix Permissions”. But as said before: It fixes permissions, dont´t cleanup server from encrypted files

I ran the malware scan on the server and while it did find some stuff and remove it, the server still was not stable, so I assume some malware was not detected.

I tried “fix file permissions” but that had no effect on the sites that were giving me 500 errors. It was very sporadic and I suspect it has something to do with php execution being corrupted, but I’m really not sure.

The VPS Snapshots are done daily with only the past 2 saved, so the backup was corrupted as well. I do have updraftplus backups for my clients. Those are not as current, but at least I have those.

I was able to access the dashboard and ssh into the server. I could download backups through the dashboard and those work about half the time, with the other half having issues with the mysql settings. I was able to manually backup the mysql databases and also manually fix those settings.

I did a fresh install of cyberpanel on a new server, and due to dependency changes, I was getting a lot of errors trying to run the update script. I was able to rollback some of the ubuntu package updates and then everything works on the new server. I then restored each website backup and fixed the database settings manually. I’m about 75% done recovering the websites.

Most of the websites run wordpress, so I’m installing wordfence hoping that will detect any malware, but I’m also going to have to go through and change passwords for everything, as I have to assume that everything has been compromised for now.

Once I’m done with recovery and changing passwords, I’ll start reading the threads on what the hacked servers generally consist of to make sure I don’t have to do further mitigation.

I felt the same way. On October 28 at noon, I received the Abuse message. Unfortunately, I didn’t get around to it until the evening. I made a snapshot and further backups and patched to v2.3.7. Everything still looked fine then. A few hours later, the CPU load increased. It was then clear that it was infected.

The first thing I did was to restore the snapshot. Strangely enough, the computer forced a restart shortly after logging on and a man-in-the-middle message when logging on for the second time. Then I knew that someone was probably faster and that I had at least two hacks. I suspect that one just installed the backdoor to come back later and the second installed the crypto-mining malware.

Unfortunately, I didn’t have an older snapshot either, but I still had a manual backup of the data and the database from October 14 and an UpdraftPro backup from October 27 at 1 am. Since the bug was only published around noon on October 27th, I assume that the version is clean. I’ll run a scanner over it later. So far it seems to be running again. I have further hardened the server with the new installation and will continue to upgrade it. From now on there will also be a Disaster Recovery Plan (DRP). You don’t want to experience something like this a second time.

You can try this, I have made a guide for whose who don’t have SSH or cyberpanel access.

And using this script you can get a chance to recover your websites and cyberpanel and then within 1 hour you should create backup using all in one migration.

And delete this Server as because I have discussed this with hetzner and they told me that the server is trying to get into the network subnet. So they might suspend the server. This virus type is using around 800Mbps constantly. The full shared subnet is around 1Gbps. So its a security and abuse issue also.