This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.
S3 Storage Design Overview¶
Buckets¶
There is but one bucket per customer. Production, test system as well as when developing locally, the same bucket is used.
Configuration is done via the s3.properties
file:
s3.main.endpoint=https://objects.rma.cloudscale.ch
s3.main.bucketName=XXXXXXX
s3.main.accessKeyId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
s3.main.secretAccessKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
All buckets are created by Ansible. There should be no need to change the configuration manuallly.
Object Removal / Retention¶
Permissions¶
![digraph {
label="Users used for S3 access and their permissions."
rankdir=LR
subgraph cluster1 {
label="Installations Accessing"
subgraph cluster1a {
label="User \"abc\""
abc [ label="Installation \"abc\"" ]
abctest [ label="Installation \"abctest\"" ]
}
subgraph cluster1b {
label="User \"xyz\""
xyz [ label=" Installation \"xyz\"" ]
xyztest [ label="Installation \"xyztest\"" ]
}
}
subgraph cluster2 {
label="Deveopers Acessing"
dev_a [ label="Peter" ]
dev_b [ label="Adrian" ]
dev_c [ label="Juan" ]
}
bucket_abc [ label="Bucket of\ncustomer \"abc\"" ]
bucket_xyz [ label="Bucket of\ncustomer \"xyz\"" ]
{ abc abctest } -> bucket_abc [ label="full access" ]
{ xyz xyztest } -> bucket_xyz [ label="full access" ]
{ bucket_abc bucket_xyz } -> { dev_a dev_b dev_c } [ dir=back label="restricted rw" ]
}](../../../_images/graphviz-a62537b61c3dc8434e2799200009cf72b4e03f9d.png)
Installation User (Left-Hand Side on Graph)¶
Every customer has a dedicated user whose access is restricted to their respective bucket. However, all installations of a customer, production or test, share that one bucket.
Developer User (Right-Hand Side on Graph)¶
Every developer has an account allowing read/write access to the buckets of all customers. However, some operation are restricted. For instance, developers cannot remove any objects or change permissions.
Implementation of Access Permissions¶
Permissions are granted via S3 policy. The policy itselfs is set by Ansible
Source of Credentials¶
Credentials can be configured in s3.[local.]properties
. Should no credentials be found in
said files, credentials from the [nice2]
section in ~/.aws/credentials
are used.
![digraph {
start [ shape=circle ]
s3_local_prop [ shape=diamond, label="Does s3.local.properties\n contain security/access key?" ]
s3_local_prop_yes [ label="Yes"]
s3_local_prop_no [ label="No"]
s3_prop [ shape=diamond, label="Does s3.properties\n contain security/access key?" ]
s3_prop_yes [ label="Yes"]
s3_prop_no [ label="No"]
aws_cred [ shape=diamond, label="Does ~/.aws/credentials\ncontain a \"nice2\" section\nwith a security/access key?" ]
aws_cred_yes [ label="Yes"]
aws_cred_no [ label="No"]
end [ shape=circle ]
{ rank=same aws_cred_yes aws_cred_no }
{ rank=same s3_prop_yes s3_prop_no }
{ rank=same s3_local_prop_yes s3_local_prop_no }
start -> s3_local_prop
s3_local_prop -> { s3_local_prop_yes s3_local_prop_no }
s3_local_prop_yes -> end [ label="use keys", color=green ]
s3_local_prop_no -> s3_prop
s3_prop -> { s3_prop_yes s3_prop_no }
s3_prop_yes -> end [ label="use keys", color=green ]
s3_prop_no -> aws_cred
aws_cred -> { aws_cred_yes aws_cred_no }
aws_cred_yes -> end [ label="use keys", color=green ]
aws_cred_no -> end [ label="no keys found", color=red ]
}](../../../_images/graphviz-d995e3e5be27dcd019acce8e8f705b2a1a73aa7d.png)