Object Stores Management with rclone¶

rclone¶

A powerful command line program to copy and sync files and directories to and from SWIFT object store. This tool has some outstanding features such partial syncs, copy only changed file, check the files integrity and retain timestamps of each file that make it the preferred tool for managing large amount of data.

Rclone is a Go program that comes as a single binary file. Please see rclone downloads, download and run it by following the instructions.

Setup Environment¶

To setup a new remote SWIFT object store in rclone, there are two options:

An interactive configuration session which prompt you for the initial authentication params:

Open your terminal and enter: rclone config and enter 'n' for new remote

1. name> choose a name for this remote
2. storage> choose "OpenStack Swift".
3. user> ScienceCloud username
4. key> ScienceCloud password
5. auth> https://cloud.science-it.uzh.ch:5000/v2.0
6. domain> default
7. tenant> your ScienceCloud project name
8. tenant_domain> default
9. region> (leave empty)
10. storage_url> (leave empty)
11. auth_version> (leave empty)

Confirm the configuration and Quit

As an alternative, if you already downloaded the OpenStack RC file, you can use rclone by simply appending to it:

export RCLONE_CONFIG_MYREMOTE_TYPE=swift
export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true

and source it as usual: in this case the name of the rclone remote will default to "myremote:"

Using this approach you will always need to source your RC file before using rclone but your Webpass will not be saved to disk.

Manage containers¶

SwiftOperator role operations¶

Here we will walk you through a very brief but must know example of this tool, however, for the further and complete guide please refer to the rclone documentation.

Create a container¶

To create container e.g input_files in ScienceCloud:

rclone mkdir remote_name:<container_name>
rclone mkdir myremote:test_container

To remove a container "rclone rmdir" will do the job.

List containers¶

To list all the containers created on your Object store:

rclone lsd remote:path
rclone lsd sciencecloud:/

Member role operations¶

rclone copy/sync¶

If a user has been given read/write access to a specific container, the user will be able to copy and sync from/to the container.

rclone copy source:path dest:path
rclone copy test.txt myremote:/test_container

Copy test.txt from your current working directory to ScienceCloud into test_container .

rclone sync source:path dest:path
rclone sync /path/to/folder myremote:/test_container

Will sync your local folder with the destination folder modifying only the destination.

rclone mount¶

rclone includes the ability to mount a Swift container as a local filesystem.

What follows in an example of how to mount it for simple download/upload of files (editing of files currently on swift will not work)

rclone mount --vfs-cache-mode off my_remote:/ /mnt/

There are several caching strategies that can be selected to make it behave more similarly to a regular filesystem:

we suggest you to read https://rclone.org/commands/rclone_mount/#file-caching for more information.

unmount¶

The Swift container should be automatically unmounted with you type Ctrl+C. If that doesn't work, you may need to manually unmount the drive:

Linux¶

fusermount -u /path/to/local/mount

Mac¶

umount /path/to/local/mount

More details are here.

Special instructions for macOS¶

Homebrew can be used to install rclone.

brew install rclone

To use rclone mount on a Mac, you'll need to additionally install macFUSE. A reboot is necessary after installing macFUSE.

brew install --cask macfuse

If the mount fails, you may need to manually allow macfuse by going to "System Preferences → Security & Privacy → General", authenticate as an admin by clicking the 🔒, and then click "Allow" to enable kernel extensions. More details are here and here.

Additional commands¶

For further commands refer to section Additional commands of the generic rclone guide.