This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.
S3 Storage Design Overview¶
There is but one bucket per customer. Production, test system as well as when developing locally, the same bucket is used.
Configuration is done via the
s3.main.endpoint=https://objects.rma.cloudscale.ch s3.main.bucketName=XXXXXXX s3.main.accessKeyId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX s3.main.secretAccessKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
All buckets are created by Ansible. There should be no need to change the configuration manuallly.
Object Removal / Retention¶
Objects are deduplicated by means of reference counting. Once the reference counter reaches
zero, the object is marked for removal. However, the object itself and the reference
_nice_binary table are not removed immediately.
There are several reasons for this behavior:
Removal of S3 objects is slow. Doing so synchronously would lead to delays.
In case a backup of the database needs to be restored, there is no need to restore a backup of the objects too.
There is no way to create atomic and thus consistent backups, i.e. backups where the database and S3 storage are guaranteed to contain the same set of objects.
Installation User (Left-Hand Side on Graph)¶
Every customer has a dedicated user whose access is restricted to their respective bucket. However, all installations of a customer, production or test, share that one bucket.
Developer User (Right-Hand Side on Graph)¶
Every developer has an account allowing read/write access to the buckets of all customers. However, some operation are restricted. For instance, developers cannot remove any objects or change permissions.
Source of Credentials¶
Credentials can be configured in
s3.[local.]properties. Should no credentials be found in
said files, credentials from the
[nice2] section in
~/.aws/credentials are used.