Lets say you have a project running on Amazon AWS, some RDS instances, file storage on S3.
And there are some scripts running to have daily backups in a form of say
tar.gz files in some specific
backups S3 bucket. Looks good?
Well… almost, what if by some reason backups being deleted or corrupted? In real life this happens more often then most of us imagine, by either human mistake or service failure.
Backup of Backups (BoB)
To solve this problem you might want to implement
Backup of Backups (BoB). Here are some ideas about it:
- It should be different accounts. The reason for this is pretty simple: “don’t put all eggs into one basket”. Whatever can happen to one target of backups, the other will not be affected.
- If possible use different providers, like if project is hosted on AWS run BoB on Google and visa versa.
- Do not expose accounts details of BoB anywhere, store them separately.
- BoB process should have read-only (RO) access to backups.
- There should be no access to BoB from the project.
- There should be monitoring configured on BoB side to get an alert in case it breaks.
Real life example
For one of our customers we configured this:
- Client runs on AWS and has S3 bucket with backups.
- BoB was configured on Google Cloud Platform.
- Currently it is not possible to implement our idea on GCP, because of lack of scheduler there, but in other ways we did the same - we are using Autoscalling Group to have one BoB instance running all the time and we put all configuration to cloud-init.
- gsutil rsync is used to sync data between AWS S3 and GS buckets.
- gsutil du is used to send metrics to DataDog.
- On DataDog side monitor is configured to send an alert to Slack if BoB metric is not growing.
- And to store BoB files GCP nearline was chosen.
Backups are not enough. And even backup of backups can be not enough. Good plan is:
- Backup of backups.
- Monitoring of both of them.
- Periodical restoring from backups/BoB.
- Disaster Recovery (DR) plan.
- Periodical DR testing.