s3 lifecycle policy to delete objects after n days example yaml config

s3 lifecycle policy to delete current and all old objects after n days example

We can add bucket lifecycle configuration policies to delete data older than 90 days, 30 days or whatever amount of days and to delete the delete markers from the s3 buckets as well. This includes deleting noncurrent version and current version if versioning is enabled on your bucket.

There are different ways this can be performed, you can go it from AWS console, CLI or through a s3 template from your repository. So this can be done through xml template as well as yml or yaml.j2 template code. You can modify your templates in config files in repository to include lifecycle configuration policies on dev or prod S3 buckets. So once you successfully run the pipelines and pushes these changes, your desired s3 buckets configuration will appear into your concerned account.

AWS allows us to delete all objects in s3 bucket lifecycle or we can filter through the objects with a policy in the mentioned time, the time depends on your data retention period for the particular project or how long you are going to keep spending money for historic data. s3 empty bucket lifecycle policy involves different aspects of changing things. You can move your data across different storage class and that results into saving a lot of your money.

s3 empty bucket lifecycle policy might be helpful when you are going to remove aws s3 delete non current version. The resulting S3 bucket will be free from all old versions of s3 object.

Let’s see one for deleting all the non current version or the older versions for your s3 object object from the desired s3 folder. The code snippet right below is particularly written in yaml and can be used in yaml.j2 extension template file or a jinja file or a yml jinja config file. The syntax provided below is written in YAML (YAML Ain’t Markup Language OR Yet another Markup Language). It’s fundamentally a human-readable data serialization language commonly used for configuration files and data interchange formats. YAML is often used in various contexts, including configuration management, cloud orchestration, and data serialization. Configuration management, cloud orchestration, and data serialization may sound unfamiliar but you don’t need to go that deeper if your task is just to implement AWS lifecycle policy on a bucket or two. So let’s have a look at code.

lifecycleconfiguration yaml example for all older version deletion

LifecycleConfiguration:
    Rules:
      - Id: DeleteNoncurrentVersions
        ExpiredObjectDeleteMarker: true
        NoncurrentVersionExpiration:
            NoncurrentDays: 30
        Status: Enabled

Once you deploy your s3 bucket template or run your s3 bucket configuration through a templates pipeline, you will see the below lifecycle rule to be appeared into your s3 bucket management section. You can have more than 1 rule defined for a bucket and that can be added from the template. There can be few more rules say for deleting s3 non current version of objects or current version objects deletion or for deleting the whole bucket.

If you look closely, this particular configuration defines ExpiredObjectDeleteMarker: true which mean there will be deletion of something call markers as well. As it’ll be visible under the lifecycle rule on your lifecycle console. Ideally, this parameter need to be true, however you can set it to false as per your business requirements.

Id parameter defined the name of your lifecycle rule, as you can see in below screenshot, we have Lifecycle rule name same as what we defined in Id. The next column in the attached screenshot shows the status of that particular s3 lifecycle config. It can be enabled or disabled as per your need.

Finally, NoncurrentVersionExpiration is set with NoncurrentDays value as 30. It means it will wait for the 30 days before deleting any s3 objects with non-current version.

Below is the extended view of the AWS lifecycle rule we just defined. You can go to the action dropdown and can possibly add more stuff to your existing lifecycle rule if you want.

lifecycle configuration rules on the s3 bucket to delete objects current versions

Now the below config is to delete the remaining data from your bucket. Once you are done with deletion of data that is non-current or the data with older version, you would like to get rid of the current versions as well which might be consuming a lot of space. So the config below is to free up the space that is held by any data in your concerned s3 bucket that is older than 90 days or 3 months.

LifecycleConfiguration:
    Rules:
      - Id: DeleteCurrentVersions
        ExpirationInDays: 90
        Status: Enabled

Let’s see how does it look if you do it manually in AWS console. The steps to set up such lifecycle rule are pretty much clear if you do it from console, but when it comes to automation and implementing things on larger scale, you need a pipeline to push these changes into production environment so it will be effective to all the desired bucket in s3. For that above given defined configuration can be taken into use.

As Explained earlier, when you will deploy this cloudformation yaml template, you will notice in Management section of your s3 bucket that a new rule has been added for the bucket with the name DeleteCurrentVersions as that’s what we’ve defined in here for Id parameter. You can go with any other name as per your wish. The above given cloudformation lifecycle config yaml template will result in just a single lifecycle rule for your bucket in s3. However we are defining 2 lifecycle policies for a single bucket as well. Let’s see the configuration for combining more than one AWS S3 lifecycle rule in one template configuration block using yml.j2

LifecycleConfiguration:
    Rules:
     - Id: DeleteNoncurrentVersions
        ExpiredObjectDeleteMarker: true
        NoncurrentVersionExpiration:
            NoncurrentDays: 30
        Status: Enabled
     - Id: DeleteCurrentVersions
        ExpirationInDays: 90
        Status: Enabled