Skip to content

Object Stores (Containers)

🔧 Under Development

This page is under active development. Content is being updated as we prepare the new version of ScienceCloud for production. Some sections may be incomplete or subject to change.

SWIFT 3 layers architecture

Account layer

Each project in ScienceCloud has an individual SWIFT account and quota. So at the very beginning, you need to ask for the SWIFT quota by contacting Science IT. Once your SWIFT quota is raised you will be able to create a container under Object Store panel of ScienceCloud dashboard.

Container layer

Is another layer of abstraction introduced in SWIFT object store architecture which allows you the possibility to assign different storage policy on a per-container basis. Containers are composed of the list of your objects and can be considered as a structural layer.

Objects

Objects are your files that are stored as binary files along with their metadata as a single unit on the filesystem.

Swift has a 5 GB limit on the size of a single uploaded object. However files larger than 5 GB can be dealt with segmenting them and uploading the resulting segments with a special manifest file that, when downloaded, sends all the segments concatenated as a single object. Read more here.

Storage policies

ScienceCloud exposes two storage policy in order to retain your data in case of failures. A storage policy can only be assigned when a container is created and once it is defined all objects will be stored in accordance to that policy.

replica-2

It is the default policy, objects are stored on two separate disks each container in a distinct server. This policy guarantees read/write access to the data in the event of a node failure or during maintenance periods.

ec104 (Erasure coding)

This policy reduces the amount of space used on the system splitting each object in 10 shards and computing 4 additional shards to allow for recovery. Each shards is then stored on a different disk and spread among the two SWIFT servers. This policy protects your data in case of disk failures (up to 3 disks hosting one of a shards of your object can fail at any given time without incurring in data loss), however your data will not be available until the system has been restored following a hardware failure or a scheduled maintenance.

Swift Roles

Swift Operator

Swift operator is a privileged role, introduced in 2016, and usually granted a subset of users in each project. The SwiftOperator user has the following access and rights within a project:

  1. List the containers
  2. Create/Delete a container
  3. Update the meta information on containers
  4. Set the read/write Access-Control List (ACL) on each container

Regular user

The non privileged users are able to:

  1. Download objects from container whose the read access is granted
  2. upload files or directories to the container with write access
  3. Issue the temporary URL for the container
  4. Obtain the statistics information for the container

But UNABLE to:

  1. List the containers in the project
  2. Set/remove ACL on containers

Available tools

There are several choices for managing your files within the SWIFT object storage which you can pick according to your liking and requisites: