Will & Skill Developers

Will & Skill Developers


Thoughts, snippets and ideas from the team at Will & Skill AB, Stockholm.

Faisal Mahmud
Author

“The mind is not a vessel to be filled, but a fire to be kindled.” ― Plutarch

Share


Improve teamwork by syncing files between two Amazon S3 buckets

Faisal MahmudFaisal Mahmud

When You have a project where You are working with a team of developers it can be tedious to share static/media files across computers unless You are sharing a development server.

At Will & Skill we have a solution that has served us well in both development and testing. We use Amazon S3 buckets that can be accessed by several members of a project while developing.

To keep things in sync we use a crontab that runs regularly and updates the staging/test bucket.

What You need

  1. Two Amazon S3 buckets, source and target
  2. The official awscli Python package
  3. A crontab that pushes changes from the source bucket to the target bucket on a regular basis

1. Create two buckets

For the sake of this tutorial lets just arbitrarily name the buckets as below

Source: mickeymouse-media  
Target: mickeymouse-test-media  

You can read a more detailed tutorial on how to create S3 buckets at http://tech.willandskill.se/how-to-add-a-new-bucket-on-aws-and-django/

2. Install awscli - Amazon Web Service Client

We'll use pip to install awscli. You can read more about the client and what type of commands it supports at https://github.com/aws/aws-cli

$ pip install awscli

And then configure the client

$ aws configure

The input/output from the command prompt should look something like

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE  
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY  
Default region name [None]: eu-west-1  
Default output format [None]:  

It will ask You to enter Your AWS credentials that You associated with the bucket in Step 1 above.

3. Run the script

aws s3 sync s3://mickeymouse-media s3://mickeymouse-test-media  

The output will look something similar to

copy: s3://mickeymouse-media/cache/7d/70/7d7069579d0beddb16df6663377d02d4.jpg to s3://mickeymouse-test-media/cache/7d/70/7d7069579d0beddb16df6663377d02d4.jpg  
copy: s3://mickeymouse-media/cache/7d/6f/7d6fc1d8a2b2f4b091b5fd44c5bbb966.jpg to s3://mickeymouse-test-media/cache/7d/6f/7d6fc1d8a2b2f4b091b5fd44c5bbb966.jpg  
copy: s3://mickeymouse-media/cache/7d/70/7d7026e7f7552a9f23662ee1ebb50521.jpg to s3://mickeymouse-test-media/cache/7d/70/7d7026e7f7552a9f23662ee1ebb50521.jpg  
...

NOTE: Make sure that You have files to copy from Your source bucket to your target bucket. Otherwise You might not see any activity in the console.

4. Automate the syncing process with a crontab to keep Your bucket in sync

#!/bin/bash

source /home/django/env/bin/activate  
aws s3 sync s3://mickeymouse-media s3://mickeymouse-test-media  
  1. Create bash file with a name like `bucket-sync.sh``
  2. Add the content described above
  3. Make it executable chmod +x bucket-sync.sh
  4. Test run the script ./bucket-sync.sh
  5. Add it as a crontab that runs x times per day

And voila, now You have a new bucket that You can use for test purposes and You can share it with Your colleagues!
This works perfectly especially if Your team is developing on a shared database and You are storing paths to media files and in the database and so on! No more irritating 404s! :)

Faisal Mahmud
Author

Faisal Mahmud

“The mind is not a vessel to be filled, but a fire to be kindled.” ― Plutarch

Comments