Will & Skill Developers

Will & Skill Developers

Thoughts, snippets and ideas from the team at Will & Skill AB, Stockholm.

Faisal M

“The mind is not a vessel to be filled, but a fire to be kindled.” ― Plutarch




Improve teamwork by syncing files between two Amazon S3 buckets

Faisal MFaisal M

When You have a project where You are working with a team of developers it can be tedious to share static/media files across computers unless You are sharing a development server.

At Will & Skill we have a solution that has served us well in both development and testing. We use Amazon S3 buckets that can be accessed by several members of a project while developing.

To keep things in sync we use a crontab that runs regularly and updates the staging/test bucket.

What You need

  1. Two Amazon S3 buckets, source and target
  2. The official awscli Python package
  3. A crontab that pushes changes from the source bucket to the target bucket on a regular basis

1. Create two buckets

  • We often use names like <projectname>-media if we want to store media files. This will serve as our source bucket where our production environment reads/writes media files.
  • For the target bucket we use names like <projectname>-test-media or <projectname>-staging-media depending on what environment that need access to the bucket

For the sake of this tutorial lets just arbitrarily name the buckets as below

Source: mickeymouse-media
Target: mickeymouse-test-media

You can read a more detailed tutorial on how to create S3 buckets at http://tech.willandskill.se/how-to-add-a-new-bucket-on-aws-and-django/

2. Install awscli - Amazon Web Service Client

We'll use pip to install awscli. You can read more about the client and what type of commands it supports at https://github.com/aws/aws-cli

$ pip install awscli

And then configure the client

$ aws configure

The input/output from the command prompt should look something like

AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-west-1
Default output format [None]: 

It will ask You to enter Your AWS credentials that You associated with the bucket in Step 1 above.

3. Run the script

aws s3 sync s3://mickeymouse-media s3://mickeymouse-test-media

The output will look something similar to

copy: s3://mickeymouse-media/cache/7d/70/7d7069579d0beddb16df6663377d02d4.jpg to s3://mickeymouse-test-media/cache/7d/70/7d7069579d0beddb16df6663377d02d4.jpg
copy: s3://mickeymouse-media/cache/7d/6f/7d6fc1d8a2b2f4b091b5fd44c5bbb966.jpg to s3://mickeymouse-test-media/cache/7d/6f/7d6fc1d8a2b2f4b091b5fd44c5bbb966.jpg
copy: s3://mickeymouse-media/cache/7d/70/7d7026e7f7552a9f23662ee1ebb50521.jpg to s3://mickeymouse-test-media/cache/7d/70/7d7026e7f7552a9f23662ee1ebb50521.jpg

NOTE: Make sure that You have files to copy from Your source bucket to your target bucket. Otherwise You might not see any activity in the console.

4. Automate the syncing process with a crontab to keep Your bucket in sync


source /home/django/env/bin/activate
aws s3 sync s3://mickeymouse-media s3://mickeymouse-test-media
  1. Create bash file with a name like `bucket-sync.sh``
  2. Add the content described above
  3. Make it executable chmod +x bucket-sync.sh
  4. Test run the script ./bucket-sync.sh
  5. Add it as a crontab that runs x times per day

And voila, now You have a new bucket that You can use for test purposes and You can share it with Your colleagues!
This works perfectly especially if Your team is developing on a shared database and You are storing paths to media files and in the database and so on! No more irritating 404s! :)

Faisal M

Faisal M

“The mind is not a vessel to be filled, but a fire to be kindled.” ― Plutarch

View Comments