Swift S3 API¶

Two ways how one can interact with OpenStack Object Storage (Swift):

Using native Object Storage API (Swift API)
Using Amazon's S3 Protocol (S3 API)

The S3 API interaction is possible due to implementation of the s3api middleware (formerly Swift3) on top of OpenStack Swift, which emulates the S3 REST API.

Note that Swift is only S3 compatible which means not all S3 API features can be used, please check compatibility matrix Swift vs S3: https://docs.openstack.org/swift/latest/s3_compat.html.

Clients¶

Among popular clients that support S3 compatible API are: s3cmd, aws, boto3, rclone, cyberduck. We provide examples of S3 API interaction using:

s3cmd command line tool
boto3 python library

Requirements¶

Configure environment using the RC file
Configure S3 compatible credentials

Configure environment with OpenStack RC file¶

In order to set up your environment, download OpenStack RC file from ScienceCloud dashboard by: API Access > Download OpenStack RC file > OpenStack RC file.

Import these variables into your shell environment:

source /path/to/downloaded/rc_file.sh

Configure S3 credentials¶

To work with S3 API including S3 clients you need "EC2-style" or S3 compatible credentials ACCESS KEY and SECRET KEY.

You can generate credentials using the following OpenStack CLI command, which generates the key pair (access, secret), see the output of the command.

openstack ec2 credentials create

+------------+-------------------------------------------------------------------------+
| Field      | Value                                                                   |
+------------+-------------------------------------------------------------------------+
| access     | <32-CHARACTER ALPHA-NUMERIC STRING>                                     |
| links      | {'self': 'https://cloud.science-it.uzh.ch:5000/v3/users/SOME_STRING'}   |
| project_id | <PROJECT-ID 32-CHARACTER ALPHA-NUMERIC STRING>                          |
| secret     | <32-CHARACTER ALPHA-NUMERIC STRING>                                     |
| trust_id   | None                                                                    |
| user_id    | <USER-ID ALPHA-NUMERIC STRING>                                          |
+------------+-------------------------------------------------------------------------+

Instead of copy-pasting the keys, we propose a short script below which generates a new pair of credentials (by calling the command above), retrieves other parameters needed for S3 API interaction and writes it to the configuration file. This configuration file can eventually be read by multiple S3 clients during authentication, see examples using s3cmd and boto3 client.

For a script below to work you need to install two libraries: OpenStackClient, for command line interaction with ScienceCloud, and jq, command-line JSON processor.

1. Install OpenStack client and JQ¶

For libraries installation we recommend to activate python virtual environment.

# create and activate python virtual environment
python -m venv .venv
source .venv/bin/activate

pip install python-openstackclient

pip install jq

Warning

If pip installation of jq fails, try to install it system-wide via brew install jq on Mac OS or sudo apt install jq on Linux. Library is available in most operating system packaging repositories.

2. Create configuration file with S3 credentials¶

After executing the following steps in the terminal configuration file s3config.cfg will be created.

S3_HOST=$(openstack catalog show swift -f json -c endpoints | jq --raw-output '.endpoints[] | select(.interface | contains("public")) | .url' | cut -f3 -d/)
ACCESS_KEY=$(openstack ec2 credentials create -f value -c access)
SECRET_KEY=$(openstack ec2 credentials show $ACCESS_KEY -f value -c secret)

cat << EOF > s3config.cfg
host_base=${S3_HOST}
host_bucket=%(bucket).${S3_HOST}
access_key=${ACCESS_KEY}
secret_key=${SECRET_KEY}
use_https=True
EOF

Check content of the configuration file s3config.cfg:

cat s3config.cfg

host_base=rgw.science-it.uzh.ch
host_bucket=%(bucket).rgw.science-it.uzh.ch
access_key=<32-CHARACTER ALPHA-NUMERIC STRING>
secret_key=<32-CHARACTER ALPHA-NUMERIC STRING>
use_https=True

Note: Every project requires separate configuration file, hence rename them accordingly.

Once S3 credentials are available you may use S3 clients to interact with OpenStack Object Storage using the S3 API, please check respective sections.

List credentials¶

You can view existing credentials using command:

openstack ec2 credentials list

+--------------+------+--------------+--------------------------------------------------------+-----------------+
| ID           | Type | User ID      | Data                                                   | Project ID      |
+--------------+------+--------------+--------------------------------------------------------+-----------------+
|              | ec2  |              | {"access": <ACCESS-KEY-ID>, "secret": <SECRET-KEY-ID>} |                 |                                |
+--------------+------+--------------+--------------------------------------------------------+-----------------+

Delete credentials¶

You can also delete created credentials with the command:

openstack ec2 credentials delete <access-key-id>