Setting Up s3cmd 2.x with DigitalOcean Spaces
Spaces is an S3-compatible object storage service that lets you store and serve large amounts of data. Each Space is a bucket for you to store and serve files. The free, built-in Spaces CDN minimizes page load times, improves performance, and reduces bandwidth and infrastructure costs.
To use s3cmd with DigitalOcean Spaces, you need:
- s3cmd version 2.0.0+ or higher. You can check your version with
s3cmd --version. Versions from package managers may be out of date, so we recommend using the s3cmd download page. Homebrew users can install the latest version with the command
brew install s3cmd.
- An access key pair for your Spaces. To generate these, visit the API page in the DigitalOcean Control Panel.
Initialize the Configuration File
By default, s3cmd stores its configuration file,
.s3cfg, in the home directory of the user that ran the configuration command.
.s3cfg is a plain text file of key/value pairs which can be edited directly once it has been created.
s3cmd uses the options set in its default configuration file when you run commands. You can specify a different configuration by appending
-c ~/path/to/config/file to each command you run.
If DigitalOcean is the main or only provider you’ll connect to with
s3cmd and you don’t want to specify its configuration file every time you use s3cmd, configure the default
~/.s3cfg file with the following command:
If you’re already using s3cmd with another service, you can create an alternate configuration file by adding the
-c flag and supplying a filename. The configuration file will be created in the directory where you issue the command, so specify the path if you want it created elsewhere.
Enter Access Keys
The script begins by asking for an Access Key and Secret Key. If you don’t already have keys, you can generate a set for s3cmd by visiting the control panel’s API page.
Enter your keys, then accept
US for the Default Region because the region information isn’t relevant to DigitalOcean. If you prefer, you can use the environment variables
AWS_SECRET_ACCESS_KEY to store a set of keys.
Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key : EXAMPLE7UQOTHDTF3GK4 Secret Key : exampleb8e1ec97b97bff326955375c5 Default Region [US]:Enter the DigitalOcean Endpoint
Next, enter the DigitalOcean Spaces endpoint. The Spaces endpoint naming pattern is
nyc3.digitaloceanspaces.com. Use the endpoint for the region your Spaces are in.
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: nyc3.digitaloceanspaces.com
The next prompt asks for a URL template to access your bucket, which is the S3 equivalent of a Space. Because Spaces supports DNS-based endpoint URLs, you can use the variable
%(bucket)s to stand in for the name of your space. Enter the following template format exactly as written:
%(bucket)s.nyc3.digitaloceanspaces.com. Again, you will change this if your Space is in a different region.
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars c an be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket : %(bucket)s.nyc3.digitaloceanspaces.com
Optional: Set an Encryption Password
The next prompt is for an optional encryption password. Unlike HTTPS, which protects files only while in transit, GPG encryption prevents others from reading files both in transit and while they are stored on DigitalOcean. Setting a password now won’t cause objects to be automatically encrypted; it just makes encryption available later.
Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password:
The next prompt asks for the path to the GPG program. On Linux, you can accept the default by pressing
ENTER. If you’re following these instructions on macOS, you may have to install GPG with a tool like Homebrew (
brew install gpg). You can then find GPG’s path with
Path to GPG program [/usr/bin/gpg]:
Connect via HTTPS
The next prompt asks to use the HTTPS protocol. HTTPS protects data from being read while it’s in transit.
DigitalOcean Spaces don’t support unencrypted transfer, so you must use HTTPS. Press
ENTER to accept the default.
When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: Yes
Optional: Set a Proxy Server
The final prompt is for an HTTP proxy server. If your network requires one, enter its IP address or domain name without the protocol, e.g.
proxy.example.com. If you aren’t using one, press
ENTER to leave it blank.
On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name:
Confirm, Test, and Save Settings
After the prompt for the HTTP Proxy server name, the configuration script presents a summary of the values it will use, followed by the opportunity to test them:
New settings: Access Key: EXAMPLES7UQOTHDTF3GK4 Secret Key: b8e1ec97b97bff326955375c5example Default Region: US S3 Endpoint: nyc3.digitaloceanspaces.com DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.n yc3.digitaloceanspaces.com Encryption password: secure_password Path to GPG program: /usr/bin/gpg Use HTTPS protocol: True HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] Y
When the test completes successfully, enter
Y to save the settings:
Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Success. Encryption and decryption worked fine :-) Save settings? [y/N] Y
If the test fails or you choose
N you’ll have the opportunity to retry the configuration. Once you save the configuration, you’ll receive confirmation of its location:
Configuration saved to '/home/sammy/nyc3'
What is the command to sync ONLY files that:
- Do not exist in the destination bucket
- Do exist in the destination bucket, but have different last-modified date, or different filesize
$ s3cmd sync <source> <dest>