How to connect to external storage with Rclone¶
This guide describes how to access external storage using Rclone. The emphasis is on the storage options provided by UZH. However, it should be easy to adapt the approach to other providers supported by Rclone. UZH users have access to the following storage options.
SWITCHdrive is based on an ownCloud instance and supports WebDAV protocol, which makes it accessible via Rclone.
SMB file shares¶
An approach to mount an SMB share on a ScienceCloud instance is described in FAQs.
OneDrive (Office 365)¶
Application access to OneDrive is currently not enabled. So, it would not be possible to access it from a server even though Rclone supports OneDrive as a backend.
Most distributions have Rclone in their official repositories. For example, on a ScienceCloud instance running Ubuntu you can install it with the following command.
sudo apt install rclone
Since users can only install applications into their user space, the easiest way to get Rclone is by creating a conda environment. For example, the following commands create a conda environment named "xfer" and install rclone into it.
module load generic anaconda conda create --name xfer rclone
Whenever you want to use Rclone after installation, you would need to load the
anaconda module and activate the
module load generic anaconda conda activate xfer
You can access SWITCHdrive using the WebDAV backend. Before you can configure an Rclone remote, you would need to log in to your SWITCHdrive account to obtain the WebDAV URL and an application token.
To get the WebDAV URL, click on the cogwheel in the bottom right corner of the page and copy the URL from the text box that appears. Please make sure you copy the whole URL, which should look like
To generate an app token, click on your name in the upper right corner and select
Settings. Click on
Security in the left navigation bar and scroll to the very bottom of the page. In the
App Name text field, enter the desired application name, for example
rclone-cluster. It is recommended to include the server information especially if you are using Rclone on multiple machines. Finally click on the
Create new app passcode button and copy the generated token. Please note that you will not be able to look up your token afterwards. However, you can always generate a replacement token if you forget this one.
To start the configuration of a remote, type
rclone config on the server where you have previously installed Rclone. This will launch the remote configuration wizard.
nto create a new remote
- Enter the desired remote name, e.g.
webdavor the number corresponding to WebDAV from the list that appears
- Enter the WebDAV URL you have retrieved earlier
owncloudor the number corresponding to ownCloud from the list that appears
- Enter your username that you use to log in to SWITCHdrive
yto add the password to the configuration
- Type or paste your app token
- Type or paste your app token again for confirmation
noto skip the advanced config
- Review the remote config record and enter
qto exit the wizard
While you can use your password instead of the app token, such usage is strongly discouraged.
You can copy files to and from the storage using the copy command, e.g.
rclone copy switch:Private/pictures/Paris.jpg data/mypictures rclone copy /scratch/tander/simulation/results.tar.gz switch:simulation
The first command copies
Private/pictures/Paris.jpg from SWITCHdrive to your local directory
data/mypictures. The second command copies
/scratch/tander/simulation/results.tar.gz to the
simulation folder on SWITCHdrive.
You can also synchronise a whole directory. For instance, the following command copies all changes from your local
data/results directory to
archive folder on SWITCHdrive.
rclone sync data/results switch:archive
rclone sync does not ask for a confirmation. It will silently delete from the target directory any files that are not present in the source directory. You need to be especially careful when using variables and expansions like
~ in your local paths.
You can find more information about available Rclone commands online or by executing
rclone command --help, e.g.
rclone sync --help.