I think OneDrive support is required. I have an Office 365 subscription with 1TB of OneDrive cloud, which I can’t use for backups.
Suman, until the backup has been sorted I recommend rclone for almost any backend. I run a cron script after CP does a backup to a local folder. The script uploads the just created backup to the remote and then deletes the local copy. Actually I keep 3 days backup locally and delete older than that.
It is two step process but it does allow you to use almost all the storage options available.
1 Like
@Sunman @hexitnz if they end up implementing the tool (Rustic) that I recommended, you’ll be able to do compressed incremental backups to onedrive and dozens of other services, all from the Cyberpanel control panel.
1 Like
Ah I see from the github page that restic can use rclone for it’s backend.
We will start doing some work on rustic from today.
1 Like
Yes, rclone is a good option. Thank you!
We ran into some issues so contacted the dev here : GLIBC 2.29 · Issue #436 · rustic-rs/rustic · GitHub
Looks like the dev resolved it in a few minutes! Like I’ve said, they’re very responsive and eager to receive feedback and contributions (whereas Restic is semi-abandonned).
I look forward to hearing how your early tests go - I expect you’ll be able to integrate it with the existing cyberpanel incremental backup mechanism very easily.
Some things to note:
- I believe that in my testing, the default compression level (zstd 3, I think) provides the best balance between speed and size. It is obviously a bit slower when the backup is to a local drive, but considerably faster than uncompressed when sent remotely over the network.
- And, of course, incremental is infinitely faster and smaller than full backups.
And, just a reminder, I left a lot of detailed notes about issues and desired improvements for the CP backup system here RClone for Incremental Backups · Issue #831 · usmannasir/cyberpanel · GitHub
Yes they do seems really responsive. However restic documentation is really great compared to rustic.
Also we won’t touch our current implemtation of Inc Backups.
Those had some security concerns due to which we were not able to bring sftp back to them, this time we want to ensure that there are no security concerns and things are done right from the start.
For the most part you can use Rustic’s documentation for Restic. But I found it to be easy to use, through the combination of the CLI docs and responsive Dev. It is not a complicated tool to use.
And, sorry to say, but it’s a bit odd for cyberpanel to complain about someone else’s documentation, when yours is basically nonexistent (despite the hundreds of hours I spent, with minimal collaboration from others, to create a system around it…). Yet we use cyberpanel because it’s a good tool. As is rustic.
Anyway, glad to hear you’re looking to do this well. I’m sure you’ll be able to borrow some of the code from the previous mechanism.
Nick I would agree that our docs may be scattered along various location, however, we’ve youtube tutorials, actual docs and blog section.
But for rustic I am not able to find any documentation at all, except that its identical to restic.
On the other hand if you go into restic site the installation process is really easy.
PS: I really appreciate your efforts to-date regarding moving our forums to discource and arranging the docs.
thanks
Can the current simple backup to local schedule be kept as well? The simplicity is useful and great for migrating sites to other servers.
They will also stay,
1 Like
Again, your complaints about Rustic’s docs are misplaced and just seems like a poor excuse for why you shouldn’t explore this option.
Rustic is a new tool and the dev evidently has not had time to create docs. But the Restic docs apply for the most part, and the CLI docs and extremely responsive dev should be more than enough to figure out what you need.
You’re surely 100x the developer that I am, yet I figured out Rustic in a few hours. It is not complicated AT ALL .
The difficult part will be on the Cyberpanel side - integrating it into the panel gui, generating the Sql and filesystem backups, etc… I have little to contribute to that, other than the detailed suggestions that I laid out in that Github issue that I’ve linked to a couple times.
Again, If people applied the same logic to cyberpanel that you are to Rustic, no one would use it because the docs are a complete disaster - out of date, disorganized, incomplete, poorly written, etc… Yet, we persist with Cyberpanel because it is a good tool. As is rustic.
Still, there’s a huge amount of people who would never touch Cyberpanel because of the terrible docs - which is simply inexcusable for a project of this scale and longevity. So, I hope you’ll get your team to start actually writing, editing, and organizing the Cyberpanel docs, and forums in general, rather than keep wasting your time and resources writing completely superficial and largely irrelevant blog posts.
Fixing the docs would be actually useful to users, and would finally make my hard work worthwhile - as it stands, I feel that my effort was a complete waste of time, so I eventually just gave up out of futility. I’m only back now because I see that you’re finally paying attention to backups, which I already put a lot of work into
There is no excuse man, if you check v2.3.4-dev branch we have already started writing some code.
But I stand my words, our docs are not as bad as the rustic ones plus it is not 100% restic compatible, for example the --exclude does not work. But we are working on it, and we will know in couple of days if this is something we will be using in a long run
thanks
Rustic is an immensely more simple tool than CyberPanel thus docs are far less necessary. Again, I - someone who really is not particularly capable with any of this - figured it out in a few hours. It is also a considerably younger tool with essentially one contributor, so actually has an excuse for not having good docs yet - none of that is the case for CyberPanel, especially when volunteer-efforts were squandered.
But here’s a PR that is already in progress for Rustic Docs WIP: Add docu by aawsome · Pull Request #438 · rustic-rs/rustic (github.com). The dev would greatly appreciate any edits/suggestions/contributions to it.
Yes, it is not 100% Restic compatible with regards to the specific commands/api. But, that is largely irrelevant and another lame excuse. What is important is that it is compatible with the Restic Repo format such that you could switch from Restic to Rustic and maintain continuity with the same repos.
The exclude parameters are in the CLI docs (which are dead-simple to access via ./rustic backup -h and they seem quite flexible.
And they’re also listed here in the Docs PR with some examples: WIP: Add docu by aawsome · Pull Request #438 · rustic-rs/rustic (github.com).
Again, if that was a sticking point for you, the dev would have been happy to explain it if you opened an issue for it.
And, just to make it clear/easy, you can type -h after any of the commands to see the full documentation. Its quite extensive and, I think, clear. Here’s what shows up for ./rustic -h. Adding any of the subcommands (backup, init, etc…) will show their respective docs.
USAGE:
rustic [OPTIONS] <SUBCOMMAND>
OPTIONS:
-h, --help Print help information
-P, --config-profile <PROFILE> Config profile to use. This parses the file `<PROFILE>.toml`
in the config directory [default: rustic]
-V, --version Print version information
GLOBAL OPTIONS:
--log-file <LOGFILE> Write log messages to the given file instead of printing them.
Note: warnings and errors are still additionally printed unless
they are ignored by --log-level [env: RUSTIC_LOG_FILE=]
--log-level <LOG_LEVEL> Use this log level [default: info] [env: RUSTIC_LOG_LEVEL=]
REPOSITORY OPTIONS:
--cache-dir <CACHE_DIR>
Use this dir as cache dir instead of the standard cache dir [env: RUSTIC_CACHE_DIR=]
--no-cache
Don't use a cache [env: RUSTIC_NO_CACHE=]
-p, --password-file <PASSWORD_FILE>
File to read the password from [env: RUSTIC_PASSWORD_FILE=]
--password <PASSWORD>
Password of the repository - WARNING: Using --password can reveal the password in the
process list! [env: RUSTIC_PASSWORD=]
--password-command <PASSWORD_COMMAND>
Command to read the password from [env: RUSTIC_PASSWORD_COMMAND=]
-r, --repository <REPOSITORY>
Repository to use [env: RUSTIC_REPOSITORY=]
--repo-hot <REPO_HOT>
Repository to use as hot storage [env: RUSTIC_REPO_HOT=]
--warm-up
Warm up needed data pack files by only requesting them without processing
--warm-up-command <WARM_UP_COMMAND>
Warm up needed data pack files by running the command with %id replaced by pack id
--warm-up-wait <DURATION>
Duration (e.g. 10m) to wait after warm up
SUBCOMMANDS:
backup Backup to the repository
cat Show raw data of repository files and blobs
check Check the repository
completions Generate shell completions
config Change the repository configuration
copy Copy snapshots to another repository
diff Compare two snapshots/paths
forget Remove snapshots from the repository
help Print this message or the help of the given subcommand(s)
init Initialize a new repository
key Manage keys
list List repository files
ls List file contents of a snapshot
prune Remove unused data or repack repository pack files
repair Restore a snapshot/path
repoinfo Show general information about the repository
restore Restore a snapshot/path
self-update Update to the latest rustic release
snapshots Show a detailed overview of the snapshots within the repository
tag Change tags of snapshots
I think you are not following our dev branch where we are actively working on rustic. Commits · usmannasir/cyberpanel · GitHub
Once again if at any point I feel that rustic is not the tool we need (I know you like it very much) I will not be going to use it.
PS: Not everyone is like you who would like to go to rustic help section on the CLI. When we implement some tool eventually some day users may need to figure out what is going on behind the scene.
I am not going to further waste time on discussion what is good and what is not while we are already working with rustic, but as I said above if I feel at some point that this is something we don’t need (by which I mean: it wont work in our usecase) we will not use it.
1 Like
X numbers of backup keep in server.
One more suggestion - incorporate some sort of toggle to allow people to adjust the io and cpu priority of the backup. Here’s the relevant doc for restic - not sure if it applies for rustic. The dev should be helpful in figuring it out.
1 Like