This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Migration

Tip

See Introduction to Deployments and Migrations for a simpler introduction to migrations.

Legend

This document describes the DevOps part of updating from one major or minor version to another.

Pre-Migration / Non-Migration State

When no migration is in progress, there is exatly one production and one test system.

Migration of Test

First, a testnew system is created and upgraded. During these phase hotfixes can still be deployed independently by deploying them to test first and then production.

Instructions

1. Create an installation named ${customer}testnew via Ansible. Copy the config from ${customer}test to ensure custom configurations are preserved. You should end up with something like this in config.yml:

    example:  # <-- customer
    # ...
    installations:
      example:  # <-- production
        db_server: db1.example.net             # <-- ${prod_db_server}
        # ...
      exampletest:  # <-- pre-existing test
        branch: releases/1.0
        env:                                    #
          custom_option: example                #        COPY THIS
        db_server: db1.example.net              # (incl. db_server/solr_server)
        solr_server: solr1.example.net          #
      exampletestnew:  # <-- new test system
        branch: releases/1.0                    # <-- set this to correct (target) version
        env:
          custom_option: abc
        db_server: db1.example.net              # <-- ${testnew_db_server}
   

   For details see Create new Customer or Installation.

2. If you don’t want to replace ${customer}, ${prod_db_server} and ${test_db_server} in the following steps manually, set them as variables in your terminal:

   customer=${customer}                #
   prod_db_server=${prod_db_server}    # Replace ${…}
   test_db_server=${test_db_server}    #
   

3. Check if DNS record exists:

   dig ${customer}testnew.tocco.ch
   

   If it doesn’t (no output from dig), create records for both, ${customer}newtest and ${customer}testold [1]. See ${INSTALLATION_NAME}.tocco.ch.

4. Create installation ${customer}testnew:

   cd ${GIT_ROOT}/tocco
   ansible-playbook playbook.yml -l ${customer}testnew --skip-tags monitoring
   

   Note

   If you see errors that don’t make sense to you, make sure that your local ansible setup is according to the latest tocco docs ansible setup guide.

5. Copy prod database:

   copy_db ${prod_db_server} nice_${customer} ${testnew_db_server} nice_${customer}testnew
   

   See Copy Database for details about copy_db.

6. Deploy installation ${customer}testnew

   See Deliver (Simple)

7. Enable monitoring:

   cd ${GIT_ROOT}/tocco
   ansible-playbook playbook.yml -l ${customer}testnew -t monitoring
   

Migration of Production

During this phase, the old test system is renamed testold and testnew renamed test. Once this is done, production can be upgraded to the new version.

Instructions

1. Update config.yml:

   1. Rename ${customer}test to ${customer}testold

   2. Rename ${customer}testnew to ${customer}test

   3. Update or add branch for production

   4. Add branch for testold if missing

   Example:

   example:
     # ...
     installations:
       example:
         branch: releases/1.0    # <-- set to target version
          # ...
       exampletestnew:    # <-- rename to 'exampletest'
         # ...
       exampletest:       # <-- rename to 'exampletestold'
         # ...
         branch: release/1.0     # <-- add if missing
         # ...
   

2. If you don’t want to replace ${customer} and ${db_server} in the following steps manually, set them as variables in your terminal:

   customer=${customer}      # Replace ${…}
   db_server=${db_server}    #
   

3. Create ${customer}testold and update configs of ${customer}test and ${customer}:

   cd ${GIT_ROOT}/tocco
   ansible-playbook playbook.yml -l ${customer}test,${customer}testold,${customer} --skip-tags monitoring
   

4. Set downtime for ${customer}test and ${customer}testnew.

   See Schedule Downtime

5. Stop test systems:

   oc -n nice-${customer}test scale --replicas 0 dc/nice
   oc -n nice-${customer}testnew scale --replicas 0 dc/nice
   

6. Rename databases:

   ssh ${db_server} psql postgres -c "\"ALTER DATABASE nice_${customer}test RENAME TO nice_${customer}testold\""
   ssh ${db_server} psql nice_${customer}testold -c "\"REASSIGN OWNED BY nice_${customer}test TO nice_${customer}testold\""
   
   ssh ${db_server} psql postgres -c "\"ALTER DATABASE nice_${customer}testnew RENAME TO nice_${customer}test\""
   ssh ${db_server} psql nice_${customer}test -c "\"REASSIGN OWNED BY nice_${customer}testnew TO nice_${customer}test\""
   

   Warning

   REASSIGN OWNED will change the owner of all objects in the DB connected to matching BY ${ROLE_NAME} and the owner of all DBs matching. Be careful when there are other DBs that have ${ROLE_NAME} as owner!

7. Start ${customer}test again:

   oc -n nice-${customer}test scale --replicas 1 dc/nice
   

8. Remove ${customer}testnew

   Skip removal of DNS record [1].

   See Remove Installation/Customer

9. Deploy installation ${customer}test

   See Deliver (Simple)

10. Deploy installation ${customer} (production)

    See Deliver (Simple)

11. Deploy installation ${customer}testold

    See Deliver (Simple)

12. Enable monitoring for testold:

    cd ${GIT_ROOT}/tocco
    ansible-playbook playbook.yml -l ${customer}testold -t monitoring
    

Complete Migration

Weeks later, when sure it isn’t needed anymore, testold can be removed.

Instructions

1. Remove ${customer}testold

   Skip removal of DNS record [1].

   See Remove Installation/Customer

Footnotes

[1] (1,2,3)

DNS records for ${customer}testnew and ${customer}testold are needed on every migration. To avoid additional work, they are created but once and kept.