Tackle Anonymous Ownership in AWS S3 Bucket Object

Tackle Anonymous Ownership in AWS S3 Bucket Object

November 8, 2019 / Nirav S

When object uploaded to s3 from third-party media, sometimes the object ownership considered as an anonymous. It’s difficult to do any object level operations on it when the ownership is anonymous and consider you have a huge amount of objects like in TB, it’s very, very difficult to change anonymous permission one by one for all the objects, it’s near to impossible. So what’s the solution !

The Issue that we faced.

  • The code used to crawl images from third party websites was uploading to the bucket under anonymous ownership which resulted in basic administrative functions like edit/delete fails.
  • The size of the bucket was 1.5 TB approx with every individual image having the size of approx 150 KB.
  • We wanted to secure the bucket & make it private but the anonymous ownership was not allowing us to make the bucket private as every image url would give a 403 Access Denied error if we make the bucket private.

The Solution

  • We searched through the web for a solution but the senorio was too unique for anybody to provide general solution.
  • Aws also didn’t have the proper solution as they suggested to delete all the existing images and reupload those.
  • which was not applicable to our scenario as there was no way to reupload the deleted images.
  • So, we had to come up with a custom script that would run on every individual image and change its ownership to aws account owner.
    • Implementation
      • We created a simple bash script that would take an object name from a list of object names in the provided file and execute 2 AWS-CLI commands.
        • First Command
          • It would change the ownership of a particular object that was in process from anonymous to AWS account owner.
        • Second Command
          • It would copy the above inprocess object into the same bucket to mimic the effect of reuploading to the bucket.
  • We had to buy a spot instance T3A.2xlarge with 8 vCPU and 32 GB Ram and ran for at least a month with the below mentioned script.

Link to install AWS cli
https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html

Script to list and change ownership of objects in s3 bucket

First of all run a command that will output only the names of the object in a particular directory in s3 bucket and pipe its output into a file.

$ aws s3 ls s3://BUCKET_NAME/FOLDER_NAME/ | perl -pe 's/^(?:\S+\s+){3}//' >> FILE_NAME.txt

Now, we have a file(FILE_NAME.txt) which contains the list containing only names of the objects that are in a buckets’ sub-directory.

Now, we use that files’s list of names as an input for our script that will change the ownership.

$ vim script.sh
#!/bin/bash
input="/FILE_PATH/FILE_NAME.txt"
while IFS= read -r line
do
aws s3 ls s3://BUCKET_NAME/FOLDER_NAME/$line --recursive | awk
'{cmd="aws s3api put-object-acl --acl bucket-owner-full-control
--no-sign-request --bucket BUCKET_NAME --key "$4; system(cmd)}' &&\
aws s3 cp --acl bucket-owner-full-control
s3://BUCKET_NAME/FOLDER_NAME/$line
s3://BUCKET_NAME/FOLDER_NAME/$line --storage-class STANDARD

Save the above lines in a file and make it executable.

Create another file which will store logs of the above script.

$ touch task.log

Now, execute the command as follows.

$ nice -n 15 sh script.sh &>> task.log &\
> disown -h %1

The Conclusion

  • We learnt AWS-CLI Commands to administer S3 buckets without GUI.
  • We learnt how to make custom bucket policy to make it secure as well as not lose the required functionality.
  • We learnt to use spot instance request and how spot instance lifecycle works.

Talk to AWS Certified Consultant

    Let Us Talk About Your AWS Development Requirements

    Have queries about your AWS project ideas and concepts? Please drop in your project details to discuss with our AWS experts, professionals and consultants.

    • Swift Hiring and Onboarding
    • Experienced and Trained AWS Team
    • Quality Consulting and Programming
    Let’s Connect and Discuss Your Project