Yes, if you want to limit cpu/io/network bankdwidth, all restic/rclone docs also apply to rustic.
@usmannasir I had a quick look at your changes in Backupsv2.py and have a few comments (feel free to ignore what ever you think is not helpful):
rustic also supports a config file. If you already create a config file for rclone, you can as well use one for rustic and e.g. put repository and password in it. Then your rustic calls will be much cleaner.
as @nick.chomey alredy mentioned, there are the --glob options which can be used to exclude patterns. However, the pattern must be inverted when comparing to a --exclude option, i.e. use !file in order to exclude file. I realize this is somehow strange, so I might add some --exclude options as well - if this is important for you, please open an issues for rustic.
You are using --parent to identify some snapshots. There are two other possibilities which I think would cover this use case better, namely a label (optionally set per snapshot) or tags (you can have multiple per snapshot). You can set these by using rustic backup --label <LABEL> or rustic backup --tag <TAG> and can use them to filter your snapshots list: rustic snapshots --filter-label <LABEL> --filter-tags <TAG>
Yes recently I just wrote the code for using the rclone config files which made overall code much cleaner.
noted
I am using parent, because as I was discussing with @nick.chomey in facebook that we need to make this repo independant of CyberPanel. So if CyberPanel server is completely lost, we’ve a config file attached to each thing (databases, db, emails) so that site can be rebuilt with it. I looked into label but had some issues.
As mentioned you should try out if this suffices to get an unchanged dump for an unchanged DB. If not, you might have to add additional --skip-* options.
Thanks very much. As it turns out, I’ve now decided to set up some cron backups of some web applications not managed by/associated with CyberPanel. I’ll give this a try and report back if I find anything different!
I’ve done some testing for the command you provided.
It would appear that the crucial flag is --skip-comments if you want to avoid backing up anything when there is no change.
Also, --add-drop-table and --quote-names are already set to TRUE by default so don’t need to be specified.
What purpose do you have for using --allow-keywords and --complete-insert? (reading the docs, they do appear like they probably should be used together)
Finally, it looks like if you make a minor change to the db (one word, one row, etc…), it adds 444.4 kiB (raw: 2.5 MiB) to the repo. That’s for a database that was originally 11.8 MiB (raw: 46.9 MiB). Seems a bit excessive, but it is what it is and MUCH better than the full thing. Though, I presume that if you had a 100GB DB, it would still add the same 444kiB, which would be nothing.
Thanks very much for all your help and hard work with Rustic!
My solution, for now, is to not use stdin. Instead, I’m doing everything from a script that generates the mysqldump file then includes that in the backup, then deletes the file. But it would be nice if stdin could be used alongside other paths.
Also, as mentioned in my comment above, you only need to use the --skip-comments flag. Two of them are already set by default and the other two don’t seem to be necessary.
I was going through different issues created by you on the repo, I would be looking for ideal case, because once this code gets finalized it would be hard to come back and do things differently.
Which is why I am looking for an option that takes a file from stdin and a file path from cli too…
rustic backup /path1 /path2 # gives snapshot abc
mysqldump ... | rustic backup - # gives snapshot def
rustic merge abc def # gives final snapshot
rustic forget abd def # remove temporary snapshots - you can add --prune, but there is not much temporary data left to care about
Another solution which is already supported is to generate two snapshots and manually “link” them together. But I wouldn’t use the parent field for it, but use the description instead to link and maybe labels to differentiate between them. It could look like
and find the “linked” snapshot by getting the description field and parsing for DB dump: {id}.
But this linking is of course fully outside of what rustic natively supports.
One thing to mention: You get fast backups only if you use a parent snapshot. So you might want to keep the original snapshots (or at least one) when backing up local paths. For backups from stdin, this however doesn’t help. Here the input stream has to be processed anyway.
using no config file (./rustic.toml doesn’t exist)
[INFO] repository rclone:testremote:cyberpanel.net: password is correct.
[INFO] using cache at /home/cyberpanel.net/.cache/rustic/ad21841af5644e9026c2c9a649fc03e9b1aa8fba26c08a13c75d8eb03e8361a8
[WARN] no backup source given.
I suppose you mean the rclone command called by rustic, right?
Actually it is not yet possible - except you do a manual hack like rename rclone and use a self-made script with name rclone which calls the renamed original rclone with parameter --config.
Do you need this and plan to use rustic config files? I can easily add an option to specify the rclone-command or rclone options in the rustic config file.
