Amazon S3 is a cloud storage provided by Amazon Web Services (AWS). Amazon S3 publishes a set of web services interfaces, upon which many third-party commercial services or client software are developed.
In this tutorial, I will describe how to access Amazon S3 cloud storage from the command line in Linux.
One of the most popular Amazon S3 command line clients is s3cmd, which is written in python. As a simple AWS S3 command line tool, s3cmd is ideal to use when you want to run scripted cron jobs such as daily backups.
Install s3cmd on Linux
To install s3cmd on Ubuntu or Debian:
To install s3cmd on Fedora:
To install s3cmd on CentOS or RHEL, download .rpm package from its official site, and install it manually. For 64-bit CentOS or RHEL 6:
When you run s3cmd for the first time, you need to configure it by running the following command.
It will ask you for a series of questions:
- access key and secret key for AWS S3
- encryption password for encrypted data transfer to and from AWS S3.
- path to GPG program used to encrypt data (e.g., /usr/bin/gpg)
- whether to use HTTPS protocol
- name and port of HTTP proxy if used
Configuration will then be saved as a plain text in ~/.s3cfg.
Basic Usage of s3cmd
To list all existing buckets in your AWS S3 account:
2011-05-28 22:30 s3://mybucket1 2011-05-29 00:14 s3://mybucket2
To create a new bucket:
Bucket 's3://dev99/' created
To upload files to an existing bucket:
1.png -> s3://dev99/1.png [1 of 3] 26261 of 26261 100% in 5s 4.33 kB/s done 2.png -> s3://dev99/2.png [2 of 3] 201430 of 201430 100% in 2s 98.05 kB/s done 3.png -> s3://dev99/3.png [3 of 3] 46630 of 46630 100% in 0s 56.62 kB/s done
The default access permission for the uploaded files is "private", meaning that only you, with correct access and secret keys, will be able to access the files.
To upload files to an existing bucket with public access permission:
4.png -> s3://dev99/4.png [1 of 1] 30778 of 30778 100% in 8s 3.34 kB/s done Public URL of the object is: http://dev99.s3.amazonaws.com/4.png
With public access permission granted, the uploaded file can be accessed by anyone by going to http://dev99.s3.amazonaws.com/4.png on any web browser.
To view the content of an existing bucket:
2013-06-02 02:52 26261 s3://dev99/1.png 2013-06-02 02:52 201430 s3://dev99/2.png 2013-06-02 02:52 46630 s3://dev99/3.png 2013-06-02 02:56 30778 s3://dev99/4.png
To download files (e.g., all .png files) contained in an existing bucket:
s3://dev99/1.png -> ./1.png [1 of 4] 26261 of 26261 100% in 0s 39.39 kB/s done s3://dev99/2.png -> ./2.png [2 of 4] 201430 of 201430 100% in 7s 24.64 kB/s done s3://dev99/3.png -> ./3.png [3 of 4] 46630 of 46630 100% in 1s 39.34 kB/s done s3://dev99/4.png -> ./4.png [4 of 4] 30778 of 30778 100% in 0s 97.01 kB/s done
To remove files in an existing bucket:
File s3://dev99/1.png deleted File s3://dev99/2.png deleted File s3://dev99/3.png deleted File s3://dev99/4.png deleted
To get information about an existing bucket, including its storage location and access control lists:
s3://dev99/ (bucket): Location: us-east-1 ACL: dan.nanni: READ ACL: dan.nanni: WRITE ACL: dan.nanni: READ_ACP ACL: dan.nanni: WRITE_ACP
To encrypt a file before uploading it to an existing bucket:
/tmp/tmpfile-pzT1zV3kLZlxDwqA0kwy -> s3://dev99/encrypt.png [1 of 1] 196890 of 196890 100% in 1s 99.51 kB/s done
When downloading an encrypted file with s3cmd, it will automatically detect the encryption, and decrypt the file on the fly upon downloading it. Thus, to download and access an encrypted file, simply run as usual:
s3://dev99/encrypt.png -> ./encrypt.png [1 of 1] 196890 of 196890 100% in 1s 131.29 kB/s done
To remove an existing bucket:
Bucket 's3://dev99/' removed
Note that you cannot remove a bucket if it is not empty.
Subscribe to Xmodulo
You can have daily Linux tutorials and FAQs delivered to your email inbox. Simply enter your email address below to subscribe to our mailing list. You will receive hands-on guides and carefully written tutorials related to Linux, everything for free.