How to use SWIFT object store¶
SWIFT 3 layers architecture¶
Each project in ScienceCloud owns an individual SWIFT account and quota. So at the very beginning, you need to ask for the SWIFT quota by contacting Science IT. Once your SWIFT quota is raised you will be able to create a container under Object Store panel of ScienceCloud dashboard.
Is another layer of abstraction introduced in SWIFT object store architecture which allows you the possibility to assign different storage policy on a per-container basis. Containers are composed of the list of your objects and can be considered as a structural layer.
Objects are your files that are stored as binary files along with their metadata as a single unit on the filesystem.
Swift has a 5 GB limit on the size of a single uploaded object. However files larger than 5 GB can be dealt with segmenting them and uploading the resulting segments with a special manifest file that, when downloaded, sends all the segments concatenated as a single object. Read more here.
ScienceCloud exposes two storage policy in order to retain your data in case of failures. A storage policy can only be assigned when a container is created and once it is defined all objects will be stored in accordance to that policy.
It is the default policy, objects are stored on two separate disks each container in a distinct server. This policy guarantees read/write access to the data in the event of a node failure or during maintenance periods.
ec104 (Erasure coding)¶
This policy reduces the amount of space used on the system splitting each object in 10 shards and computing 4 additional shards to allow for recovery. Each shards is then stored on a different disk and spread among the two SWIFT servers. This policy protects your data in case of disk failures (up to 3 disks hosting one of a shards of your object can fail at any given time without incurring in data loss), however your data will not be available until the system has been restored following a hardware failure or a scheduled maintenance.
Swift operator is a privileged role, introduced in 2016, and usually granted a subset of users in each project. The SwiftOperator user has the following access and rights within a project:
- List the containers
- Create/Delete a container
- Update the meta information on containers
- Set the read/write Access-Control List (ACL) on each container
The non privileged users are able to:
- Download objects from container whose the read access is granted
- upload files or directories to the container with write access
- Issue the temporary URL for the container
- Obtain the statistics information for the container
But UNABLE to:
- List the containers in the project
- Set/remove ACL on containers
There are several choices for managing your files within the SWIFT object storage which you can pick according to your liking and requisites:
OpenStack Swift tool¶
Swift client is the native command line utility for interacting with an OpenStack object storage environment. Although it lacks a graphical interface, it is the more popular and powerful tool for managing the data in object store.
Install the Swift python CLI and the keystone client (needed to authenticate on ScienceCloud) by issuing below cmds on your terminal:
pip install python-swiftclient pip install python-keystoneclient
In order to setup your environment, please download Openstack RC file from ScienceCloud GUI by: Access & Security > API Access > Download OpenStack RC file.
Then import this variables into your shell environment by:
To manage the containers some operations needs SwiftOperator while others have no need for the operator role enough to be a member of the project.
SwiftOperator role operations¶
Basically gives you the containers list and, if called against a container, the list of objects within:
swift list swift list container_name
On MacOS the
swift command has a collision with other Apple software. If using
swift on MacOS, either install the package using
pip within a Virtual Environment or specify the full path to the proper
swift command on your machine.
Create a container with the default storage policy¶
To create a new container with the default storage policy (i.e. replica-2) you can simply do:
swift post <container_name> swift post test_container
Create a container with the ec104 (Erasure Coding) storage policy¶
If you are aware of the downsides (see above) but would like to leverage the smaller storage overhead the ec104 storage policy has, you can create a container by:
swift post <container_name> -H 'X-Storage-Policy: <policy-name>' swift post test_container -H 'X-Storage-Policy: ec104'
It is also advisable to allocate the
_segments container that will be used by Swift when object larger than 5 GB are uploaded (failing to do so will result in a container to be automatically created with the default
swift post <container_name>_segments -H 'X-Storage-Policy: <policy-name> swift post test_container_segments -H 'X-Storage-Policy: ec104'
Set access list¶
Swift post is the right command for setting up the container ACL. '-r'/'-w' option is used for read/write ACL respectively. Then we need to specify the project name and the target user to be granted access. Otherwise using '*' in any place is mend for 'any' user or project.
swift post -r 'project_name:shortname' <container_name>
Give the write access only to hrajab, and read right to everyone in project s3it.playground to container test.¶
swift post -r 's3it.playground:*' test swift post -w 's3it.playground:hrajab' test
To delete the ACL just use the empty ''¶
swift post -r '' <container_name> swift post -w '' <container_name> swift stat test
Show the metadata for the container test¶
Create a public container¶
Public container is a public repository available to be read by everyone. The '.r,.rlistings' are the right option for this purpose.
swift post -r '.r:*,.rlistings' <container_name>
The ".r:*" will remove all restrictions on reading. The ".rlistings" will allow also listing.
If you want to allow web listing of a container (so that you can easily access it with a browser) set the 'web-listings' meta key:
swift post -m "web-listings: true" <container_name>
Member role operations¶
# Example download swift download <container_name> <object_name> swift download test_container myobject.txt # Example upload swift upload <container_name> <path/to/object> --object-name <desired_object_name> swift upload test_container ~/myobject.txt --object-name myobject.txt # Example copy swift copy <container_name> <object_name> -d </path/to/target_container> swift copy test_container myobject.txt -d /target_container
When uploading objects, make sure to specify the
--object-name flag with an appropriate object name. Otherwise, the local file path for the object will be appended to the container, which is a security concern.
Delete a container or objects within a container.
Deleting any container or object requires the appropriate ACL.
swift delete <container_name> <object_name> swift delete test text.txt
It gives you more info such as number of objects, the size, Read/Write ACL and, policy storage for given container.
swift stat <container> swift stat test_container Account: AUTH_6169dd64a7ed498581cee5abff0c6cd7 Container: test Objects: 0 Bytes: 0 Read ACL: Write ACL: Sync To: Sync Key: Accept-Ranges: bytes X-Storage-Policy: replica-2 X-Timestamp: 1479479622.50440 X-Trans-Id: txb88a0f4c9128426e9a11f-00584fe7b7 Content-Type: text/plain; charset=utf-8
Where Account point your project path.
If you need to give a temporary access to your Swift object to whom is not part of your project, generating a temporary URL would be the right solution.
swift tempurl <method> <seconds> <path> <key> swift tempurl GET 86400 /v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/test mykey /v1/AUTH_6169dd64a7ed498581cee5abff0c6cd7/test?temp_url_sig=4a741a28ec93ee75d86d233a9e50b87282a9e15c&temp_url_expires=1481717953
This link is valid for 24 hours download.
A powerful command line program to copy and sync files and directories to and from SWIFT object store. This tool has some outstanding features such partial syncs, copy only changed file, check the files integrity and retain timestamps of each file that make it the preferred tool for managing large amount of data.
Rclone is a Go program that comes as a single binary file. Please refer to rclone http://rclone.org/downloads/, download and run it by following the well explained instructions.
To setup a new remote SWIFT object store in rclone, there are two options:
An interactive configuration session which prompt you for the initial authentication params:
Open your terminal and enter:
rclone config and enter 'n' for new remote
name>choose a name for this remote
storage>choose "Openstack Swift".
tenant>your ScienceCloud project name
storage_url> (leave empty)
Confirm the configuration and Quit
As an alternative, if you already downloaded the Openstack RC file, you can use RClone by simply appending to it:
export RCLONE_CONFIG_MYREMOTE_TYPE=swift export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true
and source it as usual: in this case the name of the RClone remote will default to
Using this approach you will always need to source your RC file before using RClone but your Webpass will not be saved to disk.
SwiftOperator role operations¶
Here we will walk you through a very brief but must know example of this tool, however, for the further and complete guide please refer to rclone original documentation.
Create a container¶
To create container e.g input_files in ScienceCloud:
rclone mkdir remote_name:<container_name> rclone mkdir myremote:test_container
To remove a container
"rclone rmdir" will do the job.
To list all the containers created on your Object store:
rclone lsd remote:path rclone lsd sciencecloud:/
Member role operations¶
If a user has been given read/write access to a specific container, the user will be able to copy and sync from/to the container.
rclone copy source:path dest:path rclone copy test.txt myremote:/test_container
Copy test.txt from your current working directory to ScienceCloud into test_container .
rclone sync source:path dest:path rclone sync /path/to/folder myremote:/test_container
Will sync your local folder with the destination folder modifying only the destination.
rclone includes the ability to mount a Swift container as a local filesystem.
What follows in an example of how to mount it for simple download/upload of files (editing of files currently on swift will not work)
rclone mount --vfs-cache-mode off my_remote:/ /mnt/
There are several caching strategies that can be selected to make it behave more similarly to a regular filesystem:
we suggest you to read https://rclone.org/commands/rclone_mount/#file-caching for more information.
The Swift container should be automatically unmounted with you type Ctrl+C. If that doesn't work, you may need to manually unmount the drive:
fusermount -u /path/to/local/mount
More details are here.
Special instructions for macOS¶
brew install rclone brew cask install macfuse
If the mount fails, you may need to manually allow macfuse by going to "System Preferences → Security & Privacy → General", authenticate as an admin by clicking the 🔒, and then click "Allow" to enable kernel extensions. More details are here and here.
There are also some other useful commands such as:
rclone delete - To remove the contents of path (selective deleting).
rclone purge - Remove a container and all of its contents.
rclone check - Checks the files in the source and destination match.
rclone ls - List all the objects in the the path with size and path.
rclone lsl - List all the objects path with modification time, size and path
Offers a graphical user interface and it's only supported by Windows and MacOS.
Check the following page: How to use CyberDuck to access SWIFT
SwiftOperator role operations¶
Create a container¶
On your first login, you will need to create at least one container (folder) to put your content in.
Choose File > New Folder and specify a name. There is only "RegionOne" available on ScienceCloud.
After the container has been created, you may start adding your content to the storage platform.
Member role operations¶
of the content is indeed straight forward by right clicking on the container and choosing:
- Upload: will prompt you to pick the file for upload
- Download: start downloading right away. To change the landing folder or file name, choose Download To or Download As.
- Synchronise: To keep a directory tree sync from SWIFT storage to your local host, choose Synchronize and specify the directory when you prompted for.
- Delete: you are able to delete an object by right-click on the object and Delete.
In addition, bandwidth and number of concurrent connections can be limited using the toggle in the lower right of the transfer window.
A private object stored in your container can be made publicly available for a limited time using a signed URL. This URL grant access to anyone to download the object, however, it also includes a date and time after which the URL will no longer work. Copy the signed URL by right-click on object and Copy URL.