I see, thanks for the input.
1 Like
The current SFTP always defaults to user “root” so fixing that to allow any user name would be great.
Because of the SFTP user name fault, my current backup strategy is daily backup to local folder, delete older than 5 days and then sync using rclone to the preferred backend. That is currently SFTP with added encryption by rclone so backups are secure regardless of where they are stored.
There was some discussion a while back to integrate rclone into the CP backup giving the massive backend compatibility of rclone. Usmannasir, is that still an option or is it complicating things too much?
This is something I am 100% going to take into considerations. Will allow non-root or virtually any user to use for sftp backups.
We can use rclone for remote destinations, I am still working on the core of backups as of now (like how to generate the actual back file)
3 Likes
Fair enough, one step at a time. The non root user would be a real addition for the moment.
I can’t say it enough how much I appreciate what you and the team have created in Cyberpanel and the time you have spent on it. The odd bug can be worked round but the core is a solid CP that ticks more boxes than almost all CP’s.
1 Like
I allready proposed and addressed some issues previously.
Email notifications on failed backups
Current solution fails silently, which happened a few times. Mostly my fault, but it scares me that backups can fail for days or even weeks without me knowing.
Link backups schedules to packages
It would be nice to be able to select default backup schedules for packages and having new websites added to the schedule automatically.
This would make it possible to offer daily backups for high tier clients and weekly for low tier clients easily. And you won’t have to worry about forgetting to add a website to the backups.
Cleanup temporary backups files
Currently the backup creates random files and doesn’t cleanup after itself. The files are small, but a million small files… I’m sure you get the point.
Email backups in separate schedule
I’m not sure if email is included in the backup now. But it would be nice to be able to separate it from the rest.
Have only one backup menu in the Cyberpanel backend
There are multiple locations for different backups now. Would be nice to have them in one place
Incremental backups on Wordpress backups
I couldn’t find it in the current solution, so I’m just adding it here. I use the Wordpress backups on S3 Wasabi. I backup the DB hourly and files daily. Having an incremental option for files would drastically decrease my backup storage size.
Backup retention on S3
This is not working out of the box but would be a requirement if you ask me.
That’s what I’ve cooked up. What do you think?
Thanks for a detailed post.
I’ve updated the main thread and picked some points from your thread.
However rather then backing up specific directories, how about by default backing up all and ability to exclude some directories?
PS: Do check the original post too.
Link backups schedules to packages
We can attach backup schedules to packages, but how can you configure the run time from there? there is no such option.
But for any website under a certain package will be automatically added to schedule. But I personally believe it will confuse lots of users.
Cleanup temporary backups files
Definately, all temporary files will be removed after final zip is prepared.
Email backups in separate schedule
This will be included
Incremental backups on Wordpress backups
Backups v2 will not be incremental, unless there is a tool you can suggest apart from restic?
Incremental backups are essential - it would be a waste of time to develop Backups v2 without a solid incremental mechanism.
I previously shared a lot of ideas and information regarding cyberpanel’s implementation of incremental backups in this Github issue. RClone for Incremental Backups · Issue #831 · usmannasir/cyberpanel · GitHub.
In short, restic is completely unsuitable for this task and should have been removed long ago. However, there’s a fork/companion of it called Rustic (written in rust) that satisfies all of the concerns I shared there, has compression and more.
I would expect that cyberpanel backups v2 would incorporate all of this, along with using rclone (which is already integrated into Rustic) to allow for people to use pretty much any backend (cloud storage, ftp, etc…) that they want.
The Rustic Dev is actively developing it and is very responsive to feedback, so I’m sure would be thrilled to work together to make something great and bring a higher profile to both tools.
1 Like
Hi, nick
Glad to see you back. I will be honest with you, developing the incremnting files backup is kind of out of the scope of our development unless there is already some tool that is doing this and we can incorporate it in CyberPanel such as rustic.
I am just afraid that what if down the line even if rustic developer stops working on this tool and we will then end up wasting lots of our time. With that being said I am going to explore this option for sure and see if we can take care of incremental backups. The important thing is that I need to ensure that restore of the backups is as easy as possible be it normal or incremental backups.
I will study and revert to you as well.
thanks
2 Likes
I appreciate your concerns, though they apply to Restic and any other tool that is used by Cyberpanel.
For what it’s worth, I found both Restic and Rustic to be very easy to use, with Rustic being better because it handles incremental backups properly, is faster, has a good resume mechanism etc…
I really don’t think it would be a very difficult job to integrate into Cyberpanel - I managed to build most of the CLI stuff in the PRs that I had submitted. It was just the HTML interface (Django etc…?) that I couldn’t figure out.
I hope you’ll be able to make it work - it’s a primary request for many people!
Honestly nick I was just checking rustic I was not able to find doc links on their repo.
Plus the main concern is how to restore those rustic backups if customer loose complete access to server as these are password protected repos plus it is still really slow for large files.
These were the biggest constraints and still feels the same constraint applies.
One more thing I like to add is that for backups on the active server (server which is not yet crashed), rustic/restic works.
But incase server crashes from some reason then rustic/restic wont work, because most people won’t save the password that is required to unlock repo, thus the remote repos become useless.
Another important bottleneck is still the speed. 
I believe you can use restic and Rustic without any sort of password. But that’s a non-argument anyway - the same thing applies to mysql and cyberpanel itself.
And so long as you have access to the Rustic backup repo, you can restore it to anywhere, such as a new server.
I don’t see why it would be really slow for large files - it should be faster for them than smaller ones. Plus you can configure different compression levels, or turn it off altogether. Again, in my testing long ago (and it surely has improved since then), creating and restore backups was 2-3x faster with the compression on than with it off.
Also, incremental backups are, obviously, far faster to create and restore than a full backup and take up incomparably less storage space, which allows for setting a schedule like every 30 minutes, which is more secure/reliable than infrequent large backups.
I’m quite certain that any of your concerns could be easily alleviated with some combination of reading the cli docs/help command, experimenting, and just asking the dev for help.
For me, the current backup restore works very well. I use it for moving server’s as well as the occasional restore. The simplicity of it means I can rely on it working quickly and efficiently without passwords keys etc. Sometimes simplicity is good.
Picking the location of the restore file would be very useful rather than having to use the set /home/backup.
Individual restore options, database, email, files etc would be very useful.
For rustic which docs are you using? As far as I am aware can’t create repo without password, correct me If I am wrong?
I haven’t used Rustic in 6 months, but I recall being able to type something like rustic --help to see all docs. And, again, the dev is very responsive so you could ask questions in the Github repo if anything isn’t clear. I think he’d be particularly eager to respond and help if you said you were trying to integrate it into an open source control panel.
I think it would be better to get in touch as we are in really initial stage now. Let see how it goes, but if it works the same as restic, my hand would be tied as to avoid using it.
Here’s how to access the docs/cli options
What do you mean by “but if it works the same as restic, my hand would be tied as to avoid using it”?
As for creating a repo without a password, I’m not sure. I suspect it is possible, but maybe not. However I’m quite confident that the dev would quickly add such a possibility if it were requested.
But, again, it’s a non-issue to me… There’s already passwords for accessing a server, using cyberpanel, accessing mysql, and for using any other service in existence. If someone can’t figure out passwords, that’s their problem, not cyberpanel. But, again, I’m quite sure it would be possible to set up backups without any passwords) other than the pw required to access whatever rclone-compatible data store is being used
Moreover, restic - which is already used by cyberpanel - has the same password mechanism, which im pretty sure you’ve just been storing in a plaintext file somewhere on the server, just as you do for the mysql password.
Edit: I’ve created issues in Rustic to request being able to create repos without a password and without encryption. We’ll see what the dev says.
I also use Linode as host and they do offer S3 compatible storage - would be nice it we could use any S3 compatible storage.
The more options the users has, the better, that way we are not locked to whatever is provided. You could even make it a “paid” addon feature.
With failed email notifications as well - imagine you think your stuff is backing up and month later you check that an important backup has failed and you are not even aware. That would drive clients to not use your product.
1 Like