This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

S3 Storage Design Overview

Buckets

There is but one bucket per customer. Production, test system as well as when developing locally, the same bucket is used.

Configuration is done via the s3.properties file:

s3.main.endpoint=https://objects.rma.cloudscale.ch
s3.main.bucketName=XXXXXXX
s3.main.accessKeyId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
s3.main.secretAccessKey=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

All buckets are created by Ansible. There should be no need to change the configuration manuallly.

Object Removal / Retention

See S3 Backups and Object Removal

Permissions

digraph {
  label="Users used for S3 access and their permissions."
  rankdir=LR

  subgraph cluster1 {
      label="Installations Accessing"

      subgraph cluster1a {
          label="User \"abc\""

          abc [ label="Installation \"abc\"" ]
          abctest [ label="Installation \"abctest\"" ]
      }
      subgraph cluster1b {
          label="User \"xyz\""

          xyz [ label=" Installation \"xyz\"" ]
          xyztest [ label="Installation \"xyztest\"" ]
      }
  }

  subgraph cluster2 {
      label="Deveopers Acessing"

      dev_a [ label="Peter" ]
      dev_b [ label="Adrian" ]
      dev_c [ label="Juan" ]
  }

  bucket_abc [ label="Bucket of\ncustomer \"abc\"" ]
  bucket_xyz [ label="Bucket of\ncustomer \"xyz\"" ]

  { abc abctest } -> bucket_abc [ label="full access" ]
  { xyz xyztest } -> bucket_xyz [ label="full access" ]
  { bucket_abc bucket_xyz } -> { dev_a dev_b dev_c } [ dir=back label="restricted rw" ]
}

Installation User (Left-Hand Side on Graph)

Every customer has a dedicated user whose access is restricted to their respective bucket. However, all installations of a customer, production or test, share that one bucket.

Developer User (Right-Hand Side on Graph)

Every developer has an account allowing read/write access to the buckets of all customers. However, some operation are restricted. For instance, developers cannot remove any objects or change permissions.

Implementation of Access Permissions

Permissions are granted via S3 policy. The policy itselfs is set by Ansible

Source of Credentials

Credentials can be configured in s3.[local.]properties. Should no credentials be found in said files, credentials from the [nice2] section in ~/.aws/credentials are used.

digraph {
  start [ shape=circle ]
  s3_local_prop [ shape=diamond, label="Does s3.local.properties\n contain security/access key?" ]
  s3_local_prop_yes [ label="Yes"]
  s3_local_prop_no [ label="No"]
  s3_prop [ shape=diamond, label="Does s3.properties\n contain security/access key?" ]
  s3_prop_yes [ label="Yes"]
  s3_prop_no [ label="No"]
  aws_cred [ shape=diamond, label="Does ~/.aws/credentials\ncontain a \"nice2\" section\nwith a security/access key?" ]
  aws_cred_yes [ label="Yes"]
  aws_cred_no [ label="No"]
  end [ shape=circle ]

  { rank=same aws_cred_yes aws_cred_no }
  { rank=same s3_prop_yes s3_prop_no }
  { rank=same s3_local_prop_yes s3_local_prop_no }

  start -> s3_local_prop
  s3_local_prop -> { s3_local_prop_yes s3_local_prop_no }
  s3_local_prop_yes -> end [ label="use keys", color=green ]
  s3_local_prop_no -> s3_prop
  s3_prop -> { s3_prop_yes s3_prop_no }
  s3_prop_yes -> end [ label="use keys", color=green ]
  s3_prop_no -> aws_cred
  aws_cred -> { aws_cred_yes aws_cred_no }
  aws_cred_yes -> end [ label="use keys", color=green ]
  aws_cred_no -> end [ label="no keys found", color=red ]
}