How to connect to external storage with Rclone¶

Introduction¶

This guide describes how to access external storage using Rclone. The emphasis is on the storage options provided by UZH. However, it should be easy to adapt the approach to other providers supported by Rclone. UZH users have access to the following storage options.

SWITCHdrive¶

SWITCHdrive is based on an ownCloud instance and supports WebDAV protocol, which makes it accessible via Rclone.

SMB file shares¶

An approach to mount an SMB share on a ScienceCloud instance is described in FAQs.

OneDrive (Office 365)¶

Application access to OneDrive is currently not enabled. So, it would not be possible to access it from a server even though Rclone supports OneDrive as a backend.

Installation¶

ScienceCloud¶

Most distributions have Rclone in their official repositories. For example, on a ScienceCloud instance running Ubuntu you can install it with the following command.

sudo apt install rclone

ScienceCluster¶

To use Rclone from the ScienceCluster, run the command module load rclone.

Configuration¶

SWITCHdrive¶

You can access SWITCHdrive using the WebDAV backend. Before you can configure an Rclone remote, you would need to log in to your SWITCHdrive account to obtain the WebDAV URL and an application token.

To get the WebDAV URL, click on the cogwheel in the bottom right corner of the page and copy the URL from the text box that appears. Please make sure you copy the whole URL, which should look like https://drive.switch.ch/remote.php/dav/files/xxxx.xxxx%40uzh.ch/.

To generate an app token, click on your name in the upper right corner and select Settings. Click on Security in the left navigation bar and scroll to the very bottom of the page. In the App Name text field, enter the desired application name, for example rclone-cluster. It is recommended to include the server information especially if you are using Rclone on multiple machines. Finally click on the Create new app passcode button and copy the generated token. Please note that you will not be able to look up your token afterwards. However, you can always generate a replacement token if you forget this one.

To start the configuration of a remote, type rclone config on the server where you have previously installed Rclone. This will launch the remote configuration wizard.

Enter n to create a new remote
Enter the desired remote name, e.g. switch
Enter webdav or the number corresponding to WebDAV from the list that appears
Enter the WebDAV URL you have retrieved earlier
Enter owncloud or the number corresponding to ownCloud from the list that appears
Enter your username that you use to log in to SWITCHdrive
Enter y to add the password to the configuration
Type or paste your app token
Type or paste your app token again for confirmation
Enter no to skip the advanced config
Review the remote config record and enter y to approve.
Enter q to exit the wizard

Important

While you can use your password instead of the app token, such usage is strongly discouraged.

Usage¶

You can copy files to and from the storage using the copy command, e.g.

rclone copy switch:Private/pictures/Paris.jpg data/mypictures
rclone copy /scratch/$USER/simulation/results.tar.gz switch:simulation

The first command copies Private/pictures/Paris.jpg from SWITCHdrive to your local directory data/mypictures. The second command copies /scratch/$USER/simulation/results.tar.gz to the simulation folder on SWITCHdrive.

You can also synchronise a whole directory. For instance, the following command copies all changes from your local data/results directory to archive folder on SWITCHdrive.

rclone sync data/results switch:archive

Warning

rclone sync does not ask for a confirmation. It will silently delete from the target directory any files that are not present in the source directory. You need to be especially careful when using variables and expansions like . and ~ in your local paths.

You can find more information about available Rclone commands online or by executing rclone command --help, e.g. rclone sync --help.