When You have a project where You are working with a team of developers it can be tedious to share static/media files across computers unless You are sharing a development server.
At Will & Skill we have a solution that has served us well in both development and testing. We use Amazon S3 buckets that can be accessed by several members of a project while developing.
To keep things in sync we use a crontab that runs regularly and updates the staging/test bucket.
What You need
- Two Amazon S3 buckets,
- The official
- A crontab that pushes changes from the
sourcebucket to the
targetbucket on a regular basis
1. Create two buckets
- We often use names like
<projectname>-mediaif we want to store media files. This will serve as our source bucket where our production environment reads/writes media files.
- For the target bucket we use names like
<projectname>-staging-mediadepending on what environment that need access to the bucket
For the sake of this tutorial lets just arbitrarily name the buckets as below
Source: mickeymouse-media Target: mickeymouse-test-media
You can read a more detailed tutorial on how to create S3 buckets at http://tech.willandskill.se/how-to-add-a-new-bucket-on-aws-and-django/
2. Install awscli - Amazon Web Service Client
We'll use pip to install
awscli. You can read more about the client and what type of commands it supports at https://github.com/aws/aws-cli
$ pip install awscli
And then configure the client
$ aws configure
The input/output from the command prompt should look something like
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: eu-west-1 Default output format [None]:
It will ask You to enter Your AWS credentials that You associated with the bucket in Step 1 above.
3. Run the script
aws s3 sync s3://mickeymouse-media s3://mickeymouse-test-media
The output will look something similar to
copy: s3://mickeymouse-media/cache/7d/70/7d7069579d0beddb16df6663377d02d4.jpg to s3://mickeymouse-test-media/cache/7d/70/7d7069579d0beddb16df6663377d02d4.jpg copy: s3://mickeymouse-media/cache/7d/6f/7d6fc1d8a2b2f4b091b5fd44c5bbb966.jpg to s3://mickeymouse-test-media/cache/7d/6f/7d6fc1d8a2b2f4b091b5fd44c5bbb966.jpg copy: s3://mickeymouse-media/cache/7d/70/7d7026e7f7552a9f23662ee1ebb50521.jpg to s3://mickeymouse-test-media/cache/7d/70/7d7026e7f7552a9f23662ee1ebb50521.jpg ...
NOTE: Make sure that You have files to copy from Your
sourcebucket to your
targetbucket. Otherwise You might not see any activity in the console.
4. Automate the syncing process with a crontab to keep Your bucket in sync
#!/bin/bash source /home/django/env/bin/activate aws s3 sync s3://mickeymouse-media s3://mickeymouse-test-media
- Create bash file with a name like `bucket-sync.sh``
- Add the content described above
- Make it executable
chmod +x bucket-sync.sh
- Test run the script
- Add it as a crontab that runs x times per day
And voila, now You have a new bucket that You can use for test purposes and You can share it with Your colleagues!
This works perfectly especially if Your team is developing on a shared database and You are storing paths to media files and in the database and so on! No more irritating 404s! :)