Skip to content

How to use SWIFT object store

SWIFT 3 layers architecture

Account layer

Each project in ScienceCloud owns an individual SWIFT account and quota. So at the very beginning, you need to ask for the SWIFT quota via mailing to help@s3it.uzh.ch.. Once your SWIFT quota is raised you will be able to create a container under Object Store panel of ScienceCloud dashboard.

Container layer

Is another layer of abstraction introduced in SWIFT object store architecture which allows you the possibility to assign different storage policy on a per-container basis. Containers are composed of the list of your objects and can be considered as a structural layer.

Objects

Objects are your files that are stored as binary files along with their metadata as a single unit on the filesystem.

Swift has a 5 GB limit on the size of a single uploaded object. However files larger than 5 GB can be dealt with segmenting them and uploading the resulting segments with a special manifest file that, when downloaded, sends all the segments concatenated as a single object.

Storage policies

ScienceCloud exposes  two storage policy in order to retain your data in case of failures. A storage policy can only be assigned when a container is created and once it is defined all objects will be stored in accordance to that policy.

replica-2

It is the default policy, objects are stored on two separate disks each container in a distinct server. This policy guarantees read/write access to the data in the event of a node failure or during maintenance periods.

ec104 (Erasure coding)

This policy reduces the amount of space used on the system splitting each object in 10 shards and computing 4 additional shards to allow for recovery. Each shards is then stored on a different disk and spread among the two SWIFT servers. This policy protects your data in case of disk failures (up to 3 disks hosting one of a shards of your object can fail at any given time without incurring in data loss), however your data will not be available until the system has been restored following a hardware failure or a scheduled maintenance.

Swift Roles

Swift Operator

Swift operator is a privileged role, introduced in 2016, and usually granted a subset of users in each project. The SwiftOperator user has the following access and rights within a project:

  1. List the containers
  2. Create/Delete a container
  3. Update the meta information on containers
  4. Set the read/write Access-Control List (ACL) on each container

Regular user

The non privileged users are able to:

  1. Download objects from container whose the read access is granted
  2. upload files or directories to the container with write access
  3. Issue the temporary URL for the container
  4. Obtain the statistics information for the container

But UNABLE to:

  1. List the containers in the project
  2. Set/remove ACL on containers

Available tools

There are several choices for managing your files within the SWIFT object storage which you can pick according to your liking and requisites:

OpenStack Swift tool

Swift client is the native command line utility for interacting with an OpenStack object storage environment. Although it lacks a graphical interface, it is the more popular and powerful tool for managing the data in object store. 

Setup Environment

Install the Swift python CLI and the keystone client (needed to authenticate on science cloud) by issuing below cmds on your terminal:

pip install python-swiftclient
pip install python-keystoneclient

In order to setup your environment, please download Openstack RC file from ScienceCloud GUI by:  Access & Security > API Access  > Download OpenStack RC file.

Then import this variables into your shell environment by:

source /path/to/downloaded/rc_file.sh

Manage containers

To manage the containers some operations needs SwiftOperator while others have no need for the operator role enough to be a member of the project.

SwiftOperator role operations

List containers/objects

Basically gives you the containers list and, if called against a container, the list of objects within:

swift list
swift list container_name

Warning

On MacOS the swift command has a collision with other Apple software. If using swift on MacOS, either install the package using pip within a Virtual Environment or specify the full path to the proper swift command on your machine.

Create a container with the default storage policy

To create a new container with the default storage policy (i.e. replica-2) you can simply do:

swift post <container_name>
swift post test_container
Create a container with the ec104 (Erasure Coding) storage policy

If you are aware of the downsides (see above) but would like to leverage the smaller storage overhead the ec104 storage policy has, you can create a container by:

swift post <container_name> -H 'X-Storage-Policy: <policy-name>'
swift post test_container -H 'X-Storage-Policy: ec104'

It is also advisable to allocate the _segments container that will be used by Swift when object larger than 5 GB are uploaded (failing to do so will result in a container to be automatically created with the default replica-2 policy)

swift post <container_name>_segments -H 'X-Storage-Policy: <policy-name>
swift post test_container_segments -H 'X-Storage-Policy: ec104'
Set access list

Swift post is the right command for setting up the container ACL. '-r'/'-w' option is used for read/write ACL respectively. Then we need to specify the project name and  the target user to be granted  access. Otherwise using '*' in any place is mend for 'any' user or project.

swift post -r 'project_name:shortname' <container_name>
Give the write access only to hrajab, and read right to everyone in project s3it.playground to container test.
swift post -r 's3it.playground:*' test
swift post -w 's3it.playground:hrajab' test
To delete the ACL just use the empty ''
swift post -r '' <container_name>
swift post -w '' <container_name>
swift stat test

Show the metadata for the container test

Create a public container

Public container is a public repository available to be read by everyone. The '.r,.rlistings'  are the right option for this purpose.

swift post -r '.r:*,.rlistings' <container_name>

The ".r:*" will remove all restrictions on reading. The ".rlistings" will allow also listing.

If you want to allow web listing of a container (so that you can easily access it with a browser) set the 'web-listings' meta key:

swift post -m "web-listings: true" <container_name>

Member role operations

Swift copy/download/upload

To download

# Example download
swift download <container_name> <object_name>
swift download test_container myobject.txt

# Example upload
swift upload <container_name> <path/to/object> --object-name <desired_object_name>
swift upload test_container ~/myobject.txt --object-name myobject.txt

# Example copy
swift copy <container_name> <object_name> -d </path/to/target_container>
swift copy test_container myobject.txt -d /target_container

Warning

When uploading objects, make sure to specify the --object-name flag with an appropriate object name. Otherwise, the local file path for the object will be appended to the container, which is a security concern.

Swift delete

Delete a container or objects within a container.

Note

Deleting any container or object requires the appropriate ACL.

swift delete <container_name> <object_name>
swift delete test text.txt
Swift stat

It gives you more info such as number of objects, the size, Read/Write ACL and, policy storage for given container.

swift stat <container>
swift stat test_container
         Account: AUTH_6169dd64a7ed498581cee5abff0c6cd7
       Container: test
         Objects: 0
           Bytes: 0
        Read ACL:
       Write ACL:
         Sync To:
        Sync Key:
   Accept-Ranges: bytes
X-Storage-Policy: replica-2
     X-Timestamp: 1479479622.50440
      X-Trans-Id: txb88a0f4c9128426e9a11f-00584fe7b7
    Content-Type: text/plain; charset=utf-8

Where Account point your project path.

Swift tempurl

If you need to give a temporary access to your Swift object to whom is not part of your project, generating a temporary URL would be the right solution.

swift tempurl <method> <seconds> <path> <key>
swift tempurl GET 86400 /v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/test mykey
/v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/test?temp_url_sig=4a741a28ec93ee75d86d233a9e50b87282a9e15c&temp_url_expires=1481717953

This link is valid for 24 hours download.

Rclone

A powerful command line program to copy and sync files and directories to and from SWIFT object store. This tool has some outstanding features such partial syncs, copy only changed file, check the files integrity and retain timestamps of each file that make it the preferred tool for managing large amount of data.

Rclone is a Go program that comes as a single binary file. Please refer to rclone http://rclone.org/downloads/, download and run it by following the well explained instructions.

Setup Environment

To setup a new remote SWIFT object store in rclone, there are two options:

An interactive configuration session which prompt you for the initial authentication params:

Open your terminal and enter: rclone config and enter 'n' for new remote

  1. name> choose a name for this remote
  2. storage> choose "Openstack Swift".
  3. user> ScienceCloud username
  4. key> ScienceCloud password
  5. auth> https://cloud.s3it.uzh.ch:5000/v2.0
  6. domain> default
  7. tenant> your ScienceCloud project name
  8. tenant_domain> default
  9. region> (leave empty)
  10. storage_url> (leave empty)

  11. auth_version> (leave empty)

Confirm the configuration and Quit

As an alternative, if you already downloaded the Openstack RC file, you can use RClone by simply appending to it:

export RCLONE_CONFIG_MYREMOTE_TYPE=swift

export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true

and source it as usual: in this case the name of the RClone remote will default to "myremote:"

Using this approach you will always need to source your RC file before using RClone but your Webpass will not be saved to disk.

Manage containers

SwiftOperator role operations

Here we will walk you through a very brief but must know example of this tool, however, for the further and complete guide please refer to rclone original documentation.

Create a container

To create container e.g input_files in ScienceCloud:

rclone mkdir remote_name:<container_name>
rclone mkdir myremote:test_container

To remove a container "rclone rmdir" will do the job.

List containers

To list all the containers created on your Object store:

rclone lsd remote:path
rclone lsd sciencecloud:/

Member role operations

rclone copy/sync

If a user has been given read/write access to a specific container, the user will be able to copy and sync from/to the container.

rclone copy source:path dest:path
rclone copy test.txt myremote:/test_container

Copy test.txt from your current working directory to ScienceCloud into test_container .

rclone sync source:path dest:path
rclone sync /path/to/folder myremote:/test_container

Will sync your local folder with the destination folder modifying only the destination.

rclone mount

rclone includes the ability to mount a Swift container as a local filesystem.

What follows in an example of how to mount it for simple download/upload of files (editing of files currently on swift will not work)

rclone mount --vfs-cache-mode off my_remote:/ /mnt/

There are several caching strategies that can be selected to make it behave more similarly to a regular filesystem:

we suggest you to read https://rclone.org/commands/rclone_mount/#file-caching for more information.

unmount

The Swift container should be automatically unmounted with you type Ctrl+C. If that doesn't work, you may need to manually unmount the drive:

Linux
fusermount -u /path/to/local/mount
OS X
umount /path/to/local/mount

More details are here.

Special instructions for macOS

Homebrew can be used to install both RClone and osxfuse (required for the mount functionality to work). A reboot is necessary after installing osxfuse.

brew install rclone
brew cask install osxfuse

If the mount fails, you may need to manually allow osxfuse by going to  "System Preferences → Security & Privacy → General", authenticate as an admin by clicking the 🔒, and then click "Allow" to enable kernel extensions. More details are here and here.

Additional commands

There are also some other useful commands such as:

  • rclone delete - To remove the contents of path (selective deleting).

  • rclone purge - Remove a container and all of its contents.

  • rclone check - Checks the files in the source and destination match.

  • rclone ls - List all the objects in the the path with size and path.

  • rclone lsl - List all the objects path with modification time, size and path

Cyberduck

Offers a graphical user interface and it's only supported by Windows and MacOS.

Setup connection

Check the following page: How to use CyberDuck to access SWIFT

Manage containers

SwiftOperator role operations
Create a container

On your first login, you will need to create at least one container (folder) to put your content in.

Choose File > New Folder and specify a name. There is only "RegionOne" available on ScienceCloud.

After the container has been created, you may start adding your content to the storage platform.

Member role operations
Upload/Download/Syncing

of the content is indeed straight forward by right clicking on the container and choosing:

  1. Upload: will prompt you to pick the file for upload
  2. Download: start downloading right away. To change the landing folder or file name, choose Download To or Download As.
  3. Synchronise: To keep a directory tree sync from SWIFT storage to your local host, choose Synchronize and specify the directory when you prompted for.
  4. Delete: you are able to delete an object by right-click on the object and Delete.

In addition,  bandwidth and number of concurrent connections can be limited using the toggle in the lower right of the transfer window.

Temporary URL

A private object stored in your container can be made publicly available for a limited time using a signed URL. This URL grant access to anyone to download the object, however, it also includes a date and time after which the URL will no longer work. Copy the signed URL by right-click on object and Copy URL.


Last update: March 21, 2022