Tackle Anonymous Ownership in AWS S3 Bucket Object.
November 8, 2019 / Eternal Team
When object uploaded to s3 from third-party media, sometimes the object ownership considered as an anonymous. It’s difficult to do any object level operations on it when the ownership is anonymous and consider you have a huge amount of objects like in TB, it’s very, very difficult to change anonymous permission one by one for all the objects, it’s near to impossible. So what’s the solution !
The Issue that we faced.
- The code used to crawl images from third party websites was uploading to the bucket under anonymous ownership which resulted in basic administrative functions like edit/delete fails.
- The size of the bucket was 1.5 TB approx with every individual image having the size of approx 150 KB.
- We wanted to secure the bucket & make it private but the anonymous ownership was not allowing us to make the bucket private as every image url would give a 403 Access Denied error if we make the bucket private.
The Solution
- We searched through the web for a solution but the senorio was too unique for anybody to provide general solution.
- Aws also didn’t have the proper solution as they suggested to delete all the existing images and reupload those.
- which was not applicable to our scenario as there was no way to reupload the deleted images.
- So, we had to come up with a custom script that would run on every individual image and change its ownership to aws account owner.
- Implementation
- We created a simple bash script that would take an object name from a list of object names in the provided file and execute 2 AWS-CLI commands.
- First Command
- It would change the ownership of a particular object that was in process from anonymous to AWS account owner.
- Second Command
- It would copy the above inprocess object into the same bucket to mimic the effect of reuploading to the bucket.
- We had to buy a spot instance T3A.2xlarge with 8 vCPU and 32 GB Ram and ran for at least a month with the below mentioned script.
Link to install AWS cli
https://docs.aws.amazon.com/cli/latest/userguide/install-linux.html
Script to list and change ownership of objects in s3 bucket
First of all run a command that will output only the names of the object in a particular directory in s3 bucket and pipe its output into a file.
$ aws s3 ls s3://BUCKET_NAME/FOLDER_NAME/ | perl -pe 's/^(?:\S+\s+){3}//' >> FILE_NAME.txt
Now, we have a file(FILE_NAME.txt) which contains the list containing only names of the objects that are in a buckets’ sub-directory.
Now, we use that files’s list of names as an input for our script that will change the ownership.
$ vim script.sh
#!/bin/bash
input="/FILE_PATH/FILE_NAME.txt"
while IFS= read -r line
do
aws s3 ls s3://BUCKET_NAME/FOLDER_NAME/$line --recursive | awk
'{cmd="aws s3api put-object-acl --acl bucket-owner-full-control
--no-sign-request --bucket BUCKET_NAME --key "$4; system(cmd)}' &&\
aws s3 cp --acl bucket-owner-full-control
s3://BUCKET_NAME/FOLDER_NAME/$line
s3://BUCKET_NAME/FOLDER_NAME/$line --storage-class STANDARD
Save the above lines in a file and make it executable.
Create another file which will store logs of the above script.
$ touch task.log
Now, execute the command as follows.
$ nice -n 15 sh script.sh &>> task.log &\
> disown -h %1
The Conclusion
- We learnt AWS-CLI Commands to administer S3 buckets without GUI.
- We learnt how to make custom bucket policy to make it secure as well as not lose the required functionality.
- We learnt to use spot instance request and how spot instance lifecycle works.
