Setting Up s3cmd 2.x with DigitalOcean Spaces

https://www.digitalocean.com/docs/spaces/resources/s3cmd/

Spaces is an S3-compatible object storage service that lets you store and serve large amounts of data. Each Space is a bucket for you to store and serve files. The free, built-in Spaces CDN minimizes page load times, improves performance, and reduces bandwidth and infrastructure costs.

To use s3cmd with DigitalOcean Spaces, you need:

• s3cmd version 2.0.0+ or higher. You can check your version with s3cmd --version. Versions from package managers may be out of date, so we recommend using the s3cmd download page. Homebrew users can install the latest version with the command brew install s3cmd.
• An access key pair for your Spaces. To generate these, visit the API page in the DigitalOcean Control Panel.

Initialize the Configuration File

By default, s3cmd stores its configuration file, .s3cfg, in the home directory of the user that ran the configuration command. .s3cfg is a plain text file of key/value pairs which can be edited directly once it has been created.

s3cmd uses the options set in its default configuration file when you run commands. You can specify a different configuration by appending -c ~/path/to/config/file to each command you run.

If DigitalOcean is the main or only provider you’ll connect to with s3cmd and you don’t want to specify its configuration file every time you use s3cmd, configure the default ~/.s3cfg file with the following command:

s3cmd --configure

If you’re already using s3cmd with another service, you can create an alternate configuration file by adding the -c flag and supplying a filename. The configuration file will be created in the directory where you issue the command, so specify the path if you want it created elsewhere.

Enter Access Keys

The script begins by asking for an Access Key and Secret Key. If you don’t already have keys, you can generate a set for s3cmd by visiting the control panel’s API page.

Enter your keys, then accept US for the Default Region because the region information isn’t relevant to DigitalOcean. If you prefer, you can use the environment variables AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY to store a set of keys.

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key []: EXAMPLE7UQOTHDTF3GK4
Secret Key []: exampleb8e1ec97b97bff326955375c5
Default Region [US]: Enter the DigitalOcean Endpoint

Next, enter the DigitalOcean Spaces endpoint. The Spaces endpoint naming pattern is <region>.digitaloceanspaces.com, like nyc3.digitaloceanspaces.com. Use the endpoint for the region your Spaces are in.

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: nyc3.digitaloceanspaces.com

The next prompt asks for a URL template to access your bucket, which is the S3 equivalent of a Space. Because Spaces supports DNS-based endpoint URLs, you can use the variable %(bucket)s to stand in for the name of your space. Enter the following template format exactly as written: %(bucket)s.nyc3.digitaloceanspaces.com. Again, you will change this if your Space is in a different region.

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars c
an be used if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket []: %(bucket)s.nyc3.digitaloceanspaces.com

Optional: Set an Encryption Password

The next prompt is for an optional encryption password. Unlike HTTPS, which protects files only while in transit, GPG encryption prevents others from reading files both in transit and while they are stored on DigitalOcean. Setting a password now won’t cause objects to be automatically encrypted; it just makes encryption available later.

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:

The next prompt asks for the path to the GPG program. On Linux, you can accept the default by pressing ENTER. If you’re following these instructions on macOS, you may have to install GPG with a tool like Homebrew (brew install gpg). You can then find GPG’s path with which gpg.

Path to GPG program [/usr/bin/gpg]:

Connect via HTTPS

The next prompt asks to use the HTTPS protocol. HTTPS protects data from being read while it’s in transit.

DigitalOcean Spaces don’t support unencrypted transfer, so you must use HTTPS. Press ENTER to accept the default.

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: Yes

Optional: Set a Proxy Server

The final prompt is for an HTTP proxy server. If your network requires one, enter its IP address or domain name without the protocol, e.g. 203.0.113.1 or proxy.example.com. If you aren’t using one, press ENTER to leave it blank.

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

Confirm, Test, and Save Settings

After the prompt for the HTTP Proxy server name, the configuration script presents a summary of the values it will use, followed by the opportunity to test them:

New settings:
 Access Key: EXAMPLES7UQOTHDTF3GK4
 Secret Key: b8e1ec97b97bff326955375c5example
 Default Region: US
 S3 Endpoint: nyc3.digitaloceanspaces.com
 DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.n
yc3.digitaloceanspaces.com
 Encryption password: secure_password
 Path to GPG program: /usr/bin/gpg
 Use HTTPS protocol: True
 HTTP Proxy server name:
 HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y

When the test completes successfully, enter Y to save the settings:

Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] Y

If the test fails or you choose N you’ll have the opportunity to retry the configuration. Once you save the configuration, you’ll receive confirmation of its location:

Configuration saved to '/home/sammy/nyc3'

More Information

You can use our quick reference on s3cmd usage to get started. For a comprehensive guide to s3cmd, see the s3cmd usage guide or access the help file from the command line with s3cmd --help.

What is the command to sync ONLY files that:

• Do not exist in the destination bucket
• Do exist in the destination bucket, but have different last-modified date, or different filesize

answer

$ s3cmd sync <source> <dest>