An Advanced Backup Strategy Using Skeddly - Skeddly

Skeddly Blog

Skeddly news and announcements...

An Advanced Backup Strategy Using Skeddly

Using Skeddly, you are able to easily create EBS snapshots on a hourly, daily, weekly, or monthly basis. In addition, Skeddly can be configured to delete those snapshots based on age, and keeping a minimum number of snapshots. Simple backup and delete strategies can be created. But more advanced strategies can be created as well.

Goal

For example, let’s say that you want to create snapshots under the following schedule:

  • Hourly snapshots kept for 24 hours
  • Daily snapshots kept for one week
  • Weekly snapshots kept for one month
  • Monthly snapshots kept for one year

Creating the Backups

We’re going to use a “Backup Multiple EC2 Instances” action to create our EBS snapshots. This action will create EBS snapshots of all EBS volumes attached to one or more EC2 instances. Selecting which instances to backup is easy with this action: we’ll backup all instances in our AWS account. However, you could choose to limit the selection by comparing EC2 tags or other criteria.

Step 1. Create a new “Backup Multiple EC2 Instances” action.

Give the action any name you want.

Select your region and credentials.

Select an hourly schedule.

For “Instance Identification”, select “All Instances”.

Use any snapshot name you’d like. Include the $(DATE) and $(TIME) macros to include the date and time if you’d like.

We’re not going to stop the instance to perform the backups, so select “None” for “Consistency Method”. This does mean that filesystems may be in an inconsistent state when the snapshots are created, but we’re OK with that for this example.

We want to add extra EC2 tags to our EBS snapshots that will help with the deletion process.

  • Skeddly-Time = $(TIME)
  • Skeddly-DayOfWeek = $(DOW)
  • Skeddly-WeekOfMonth = $(INVWEEKOFMONTH)

The reason why we prefix all the tag names with “Skeddly-“ is to distinguish snapshots created by this action from snapshots created by other actions.

Step 2. Create a new “Delete EBS Snapshots” action to delete hourly snapshots.

The first requirement is to delete snapshots which are over 24 hours old.

Give the action any name you want.

Select your region and credentials.

Select an hourly schedule. If you’re OK with some snapshots sticking around a bit longer than 24 hours, then you can reduce this schedule to every few hours or daily.

Under criteria, select:

  • Delete Comparison: “EC2 Tag”, “Does not start with”, “23”
  • EC2 Tag: “Skeddly-Time”
  • Older Than: 1 day
  • Minimum to Keep: 1

This will delete all EBS snapshots that are over 24 hours old, except those that were created at 11pm. These we’ll keep around for the later “delete” actions.

One additional comment: when Skeddly performs an EC2 tag comparison, if the tag simply does not exist on the snapshot, then Skeddly will not delete the snapshot. This fact, along with our “Skeddly-“ prefix ensures we don’t delete any snapshots create by other means.

Step 3. Create a new “Delete EBS Snapshots” action to delete daily snapshots.

The second requirement is to delete daily snapshots weekly.

Give the action any name you want.

Select your region and credentials.

Select a daily schedule. If you’re OK with some snapshots sticking around a bit longer than 7 days, then you can reduce this schedule to every few days or weekly.

Under criteria, select:

  • Delete Comparison: “EC2 Tag”, “Does not equal”, “Sat”
  • EC2 Tag: “Skeddly-DayOfWeek”
  • Older Than: 7 days
  • Minimum to Keep: 1

This will delete all EBS snapshots that are over 7 days old, except those that were created on Saturdays. These we’ll keep around for the later “delete” actions.

Step 4. Create a new “Delete EBS Snapshots” action to delete weekly snapshots.

The third requirement is to delete weekly snapshots monthly.

Give the action any name you want.

Select your region and credentials.

Select a daily schedule. If you’re OK with some snapshots sticking around a bit longer than 31 days, then you can reduce this schedule to weekly or monthly.

Under criteria, select:

  • Delete Comparison: “EC2 Tag”, “Does not equal”, “1”
  • EC2 Tag: “Skeddly-WeekOfMonth”
  • Older Than: 31 days
  • Minimum to Keep: 1

This will delete all EBS snapshots that are over 31 days old, except those that were created on the last week of each Month. These we’ll keep around for the later “delete” actions.

Step 5. Create a new “Delete EBS Snapshots” action to delete monthly snapshots.

The last requirement is to delete monthly snapshots yearly.

Give the action any name you want.

Select your region and credentials.

Select a monthly schedule.

Under criteria, select:

  • Delete Comparison: “EC2 Tag”, “Equals”, “1”
  • EC2 Tag: “Skeddly-WeekOfMonth”
  • Older Than: 365 days
  • Minimum to Keep: 1

This will delete all remaining EBS snapshots that are over 365 days old.

In all cases, we’re keeping a minimum of 1 to ensure we don’t delete all backups.

TL;DR

Skeddly is a very easy-to-use tool, and when using the basic building blocks, you can use multiple backup and delete actions to compose a very creative backup retention strategy.