Skip to content

How to connect to external storage with Rclone

Introduction

This guide describes how to access external storage using Rclone. The emphasis is on the storage options provided by UZH. However, it should be easy to adapt the approach to other providers supported by Rclone. UZH users have access to the following storage options.

SWITCHdrive

SWITCHdrive is based on an ownCloud instance and supports WebDAV protocol, which makes it accessible via Rclone.

SMB file shares

An approach to mount an SMB share on a ScienceCloud instance is described in FAQs.

OneDrive (Office 365)

Application access to OneDrive is currently not enabled. So, it would not be possible to access it from a server even though Rclone supports OneDrive as a backend.

Installation

ScienceCloud

Most distributions have Rclone in their official repositories. For example, on a ScienceCloud instance running Ubuntu you can install it with the following command.

sudo apt install rclone

ScienceCluster

Since users can only install applications into their user space, the easiest way to get Rclone is by creating a conda environment. For example, the following commands create a conda environment named "xfer" and install rclone into it.

module load generic anaconda
conda create --name xfer rclone

Whenever you want to use Rclone after installation, you would need to load the anaconda module and activate the xfer environment.

module load generic anaconda
conda activate xfer

Configuration

SWITCHdrive

You can access SWITCHdrive using the WebDAV backend. Before you can configure an Rclone remote, you would need to log in to your SWITCHdrive account to obtain the WebDAV URL and an application token.

To get the WebDAV URL, click on the cogwheel in the bottom right corner of the page and copy the URL from the text box that appears. Please make sure you copy the whole URL, which should look like https://drive.switch.ch/remote.php/dav/files/xxxx.xxxx%40uzh.ch/.

To generate an app token, click on your name in the upper right corner and select Settings. Click on Security in the left navigation bar and scroll to the very bottom of the page. In the App Name text field, enter the desired application name, for example rclone-cluster. It is recommended to include the server information especially if you are using Rclone on multiple machines. Finally click on the Create new app passcode button and copy the generated token. Please note that you will not be able to look up your token afterwards. However, you can always generate a replacement token if you forget this one.

To start the configuration of a remote, type rclone config on the server where you have previously installed Rclone. This will launch the remote configuration wizard.

  • Enter n to create a new remote
  • Enter the desired remote name, e.g. switch
  • Enter webdav or the number corresponding to WebDAV from the list that appears
  • Enter the WebDAV URL you have retrieved earlier
  • Enter owncloud or the number corresponding to ownCloud from the list that appears
  • Enter your username that you use to log in to SWITCHdrive
  • Enter y to add the password to the configuration
  • Type or paste your app token
  • Type or paste your app token again for confirmation
  • Enter no to skip the advanced config
  • Review the remote config record and enter y to approve.
  • Enter q to exit the wizard

Important

While you can use your password instead of the app token, such usage is strongly discouraged.

Usage

You can copy files to and from the storage using the copy command, e.g.

rclone copy switch:Private/pictures/Paris.jpg data/mypictures
rclone copy /scratch/tander/simulation/results.tar.gz switch:simulation

The first command copies Private/pictures/Paris.jpg from SWITCHdrive to your local directory data/mypictures. The second command copies /scratch/tander/simulation/results.tar.gz to the simulation folder on SWITCHdrive.

You can also synchronise a whole directory. For instance, the following command copies all changes from your local data/results directory to archive folder on SWITCHdrive.

rclone sync data/results switch:archive

Warning

rclone sync does not ask for a confirmation. It will silently delete from the target directory any files that are not present in the source directory. You need to be especially careful when using variables and expansions like . and ~ in your local paths.

You can find more information about available Rclone commands online or by executing rclone command --help, e.g. rclone sync --help.


Last update: January 6, 2022