How to use Globus to transfer data¶
Globus is an open-source toolkit used for grid computing, and it offers a GridFTP implementation for the transfer of massive datasets. S3IT manages an official UZH Globus endpoint on the ScienceCluster that can be used to transfer data to external collaborators.
Globus transfers can be performed between official university endpoints free-of-charge. If you're attempting to transfer data to a collaborator at another university, first investigate whether their university has an endpoint. If so, the transfer can be performed very easily following the steps below.
If the external collaborator's university or research organization does not maintain a Globus endpoint, there is also the option of transferring from the UZH S3IT endpoint to a Globus Connect Personal based endpoint. Globus Connect Personal is an application that allows a user to turn their personal laptop or workstation into a fully functional Globus endpoint for sending/receiving data.
Globus allows transfers free of charge when either the sender or recipient is using an officially maintained Globus endpoint. In other words, only a single Globus Connect Personal based endpoint can be used in a free transfer. Transferring between 2 Globus Connect Personal endpoints requires a Globus subscription. See here for more details.
- Begin by going to https://www.globus.org/ and clicking the "Log In" button at the top of the page.
- At this point, you should be prompted to log in via your "existing organizational login". To find UZH, simply search for "Zurich".
- Click on Universität Zürich in the drop down, then click Continue to be forwarded to the SWITCH edu-ID authentication.
- After logging in, click "File Manager" on the menu on the left, then search for "S3IT" in the Collection search bar. When doing so, you should see the "S3IT UZH globus endpoint" appear in the suggestions.
- Click on the "Continue" button when prompted with "Please authenticate to access S3IT UZH globus endpoint".
- You will then be redirected to another login screen. Click on the integer username, as shown in the screenshot. At the next screen, click "Allow".
- At this point you should see your files on the ScienceCluster appear in the File Manager area. To start a transfer to another Globus endpoint, click the 2nd panel option on the top right part of the page and search for the recipient's endpoint. In the dual panel view the ScienceCluster will be shown on the left, and the recipient endpoint will be shown on the right.
- Locate the Globus endpoint that should receive the data and ensure it loads correctly within the right panel. Contact your data's recipient if you have trouble finding or loading their endpoint.
- Specify any of the "Transfer & Sync Options" that you require, click through and select all files/directories that you'd like to send, then press "Start" to begin your transfer.
We highly recommend testing the transfer of a small dataset or directory of files before performing a large transfer.
If you have your files on a ScienceCloud VM, a ScienceCloud volume, a Swift Container, or elsewhere on UZH technical infrastructure, consider copying the files to the ScienceCluster in order to take advantage of Globus.
/home folders are too large for the files you want to transfer, consider scalable storage as a temporary area for storing your files during the transfer.