This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

S3 Storage Design Overview

Buckets

There is but one bucket per customer. Production, test system as well as when developing locally, the same bucket is used.

Configuration is done via the s3.properties file:

s3.main.endpoint=https://objects.rma.cloudscale.ch
s3.main.bucketName=XXXXXXX
s3.main.accessKeyId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
s3.main.secretAccessKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

All buckets are created by Ansible. There should be no need to change the configuration manuallly.

Object Removal / Retention

Objects are deduplicated by means of reference counting. Once the reference counter reaches zero, the object is marked for removal. However, the object itself and the reference in the _nice_binary table are not removed immediately.

There are several reasons for this behavior:

  1. Removal of S3 objects is slow. Doing so synchronously would lead to delays.

  2. In case a backup of the database needs to be restored, there is no need to restore a backup of the objects too.

  3. There is no way to create atomic and thus consistent backups, i.e. backups where the database and S3 storage are guaranteed to contain the same set of objects.

Permissions

digraph { label="Users used for S3 access and their permissions." rankdir=LR subgraph cluster1 { label="Installations Accessing" subgraph cluster1a { label="User \"abc\"" abc [ label="Installation \"abc\"" ] abctest [ label="Installation \"abctest\"" ] } subgraph cluster1b { label="User \"xyz\"" xyz [ label=" Installation \"xyz\"" ] xyztest [ label="Installation \"xyztest\"" ] } } subgraph cluster2 { label="Deveopers Acessing" dev_a [ label="Peter" ] dev_b [ label="Adrian" ] dev_c [ label="Juan" ] } bucket_abc [ label="Bucket of\ncustomer \"abc\"" ] bucket_xyz [ label="Bucket of\ncustomer \"xyz\"" ] { abc abctest } -> bucket_abc [ label="full access" ] { xyz xyztest } -> bucket_xyz [ label="full access" ] { bucket_abc bucket_xyz } -> { dev_a dev_b dev_c } [ dir=back label="restricted rw" ] }

Installation User (Left-Hand Side on Graph)

Every customer has a dedicated user whose access is restricted to their respective bucket. However, all installations of a customer, production or test, share that one bucket.

Developer User (Right-Hand Side on Graph)

Every developer has an account allowing read/write access to the buckets of all customers. However, some operation are restricted. For instance, developers cannot remove any objects or change permissions.

Implementation of Access Permissions

Permissions are granted via S3 policy. The policy itselfs is set by Ansible

Source of Credentials

Credentials can be configured in s3.[local.]properties. Should no credentials be found in said files, credentials from the [nice2] section in ~/.aws/credentials are used.

digraph { start [ shape=circle ] s3_local_prop [ shape=diamond, label="Does s3.local.properties\n contain security/access key?" ] s3_local_prop_yes [ label="Yes"] s3_local_prop_no [ label="No"] s3_prop [ shape=diamond, label="Does s3.properties\n contain security/access key?" ] s3_prop_yes [ label="Yes"] s3_prop_no [ label="No"] aws_cred [ shape=diamond, label="Does ~/.aws/credentials\ncontain a \"nice2\" section\nwith a security/access key?" ] aws_cred_yes [ label="Yes"] aws_cred_no [ label="No"] end [ shape=circle ] { rank=same aws_cred_yes aws_cred_no } { rank=same s3_prop_yes s3_prop_no } { rank=same s3_local_prop_yes s3_local_prop_no } start -> s3_local_prop s3_local_prop -> { s3_local_prop_yes s3_local_prop_no } s3_local_prop_yes -> end [ label="use keys", color=green ] s3_local_prop_no -> s3_prop s3_prop -> { s3_prop_yes s3_prop_no } s3_prop_yes -> end [ label="use keys", color=green ] s3_prop_no -> aws_cred aws_cred -> { aws_cred_yes aws_cred_no } aws_cred_yes -> end [ label="use keys", color=green ] aws_cred_no -> end [ label="no keys found", color=red ] }