S3 Batch Replication

S3 batch replication feature, help you to replicate your existing S3 objects and keep always synchronize your buckets with new objects across all the buckets you configured for replication between or different regions or different AWS accounts.

(Before this as on 08Feb2022), S3 replication was only work for new object to replicate but not by default for existing object and to replicate existing object, you had to raise request to AWS support and then AWS support enable this feature for you and also the replication for existing object was taking more time as compare to replicate new object. But with new batch replication feature, now you can simply go and do setup and replicate your existing objects and keep

always synchronize your buckets for new objects.

Use cases: For example, you can use it to minimize latency by maintaining copies of your data in AWS Regions geographically closer to your users, to meet compliance and data sovereignty requirements, and to create additional resiliency for disaster recovery planning. S3 Replication is a fully managed, low-cost feature that replicates newly uploaded objects between buckets. The buckets can belong to the same or different accounts. Objects may be replicated to a single destination bucket or to multiple destination buckets.

Demo:

Make sure you have versioning enabled for source and target buckets to use S3 Batch replication feature.

To create replication, go to source S3 bucket and them from there you can "create replication rule" from "Management" tab or you can use "Batch Operations" option from left panel.

Once you create, it will ask you to provide target buckets and there you can select bucket form same or different region or even use different AWS account S3 bucket as target S3 to get objects copied/replicated from source bucket to target bucket.

You can also specify any prefix/filter you want to use for data copy or all objects to be copied.

During replication rule creation, you can also change storage class for your target bucket:

For IAM role,

Make sure you have required permission available to list/save objects from source to target bucket and also with role, trusted assume role must be used for s3:

Permission:

Trusted - Assume role:

During role creation, at last step, it will ask you to confirm, do you want to replicate existing object, choose - yes.

When you choose yes, then it will ask some more details like how you want to run, logging bucket where you want to generate report and also generate template json file which is actually like S3 inventory which keep track of your existing objects to copied and need additional IAM role with permission to have put permission for logging bucket and list permissions to manage inventory:

IAM role permission:

IAM role trust relationship - Assume role:

And then once it created, you can find same either with S3 replication management tab rule section or from bath operation:

And you can use the clickable link to check status or execution also you can verify the bucket you had provided for storing all the execution result like below:

So in my case: with my source bucket I had few files including folders and after replication created, it copied all the objects to my target bucket. And also after replication done for my existing object(highlighted in yellow), I uploaded one new file to my source object (non highlighted one) and same have been copied to my target bucket just in few seconds.

My source bucket:

My target bucket: