This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Migration

Tip

See Introduction to Deployments and Migrations for a simpler introduction to migrations.

This document describes the DevOps part of updating from one major or minor version to another.

Pre-Migration / Non-Migration State

When no migration is in progress, there is exatly one production and one test system.

digraph {
    label="Normal Operations (v3.0)"

    subgraph cluster_git {
        label="Git repository"
        r300 [ label="releases/3.0", shape="hexagon" ]
    }
    prod [ label="prod (v3.0)" ]
    test [ label="test (v3.0)" ]

    r300 -> test [ style=dotted ]
    test -> prod [ style=dotted ]
}

Migration of Test

First, a testnew system is created and upgraded. During these phase hotfixes can still be deployed independently by deploying them to test first and then production.

digraph {
    label="Migration of Test System (v3.0 → v3.5)"

    subgraph cluster_git {
        label="Git repository"
        r300 [ label="releases/3.0", shape="hexagon" ]
        r305 [ label="releases/3.5", shape="hexagon" ]
    }
    prod [ label="prod (v3.0)" ]
    test [ label="test (v3.0)" ]
    testnew [ label="testnew (v3.0 → v3.5)", color=green ]

    { rank=same test testnew }

    r300 -> test [ style=dotted ]
    r305 -> testnew [ color=red, fontcolor=red, label=deploy ]
    test -> prod [ style=dotted ]

    prod -> testnew [ color=blue, fontcolor=blue, label="copy DB" ]
}

Instructions

  1. Create an installation named ${customer}testnew via Ansible. Copy the config from ${customer}test to ensure custom configurations are preserved. You should end up with something like this in config.yml:

     example:  # <-- customer
     # ...
     installations:
       example:  # <-- production
         db_server: db1.example.net             # <-- ${prod_db_server}
         # ...
       exampletest:  # <-- pre-existing test
         branch: releases/1.0
         env:                                    #
           custom_option: example                #        COPY THIS
         db_server: db1.example.net              #
       exampletestnew:  # <-- new test system
         branch: releases/1.0                    # <-- set this to correct (target) version
         env:
           custom_option: abc
         db_server: db1.example.net              # <-- ${testnew_db_server}
    

    For details see Create new Customer or Installation.

  2. If you don’t want to replace ${customer}, ${prod_db_server} and ${test_db_server} in the following steps manually, set them as variables in your terminal:

    customer=${customer}                #
    prod_db_server=${prod_db_server}    # Replace ${…}
    test_db_server=${test_db_server}    #
    
  3. Check if DNS record exists:

    dig ${customer}testnew.tocco.ch
    

    If it doesn’t (no output from dig), create records for both, ${customer}newtest and ${customer}testold [1]. See ${INSTALLATION_NAME}.tocco.ch.

  4. Create installation ${customer}testnew:

    cd ${GIT_ROOT}/tocco
    ansible-playbook playbook.yml -l ${customer}testnew --skip-tags monitoring
    

    Note

    If you see errors that don’t make sense to you, make sure that your local ansible setup is according to the latest tocco docs ansible setup guide.

  5. Copy prod database:

    copy_db ${prod_db_server} nice_${customer} ${testnew_db_server} nice_${customer}testnew
    

    See Copy Database for details about copy_db.

  6. Deploy installation ${customer}testnew

    See Deliver (Simple)

  7. Enable monitoring:

    cd ${GIT_ROOT}/tocco
    ansible-playbook playbook.yml -l ${customer}testnew -t monitoring
    

Migration of Production

During this phase, the old test system is renamed testold and testnew renamed test. Once this is done, production can be upgraded to the new version.

digraph {
    label="Migration of Production System (v3.0 → v3.5)"

    subgraph cluster_git {
        label="Git repository"
        r305 [ label="releases/3.5", shape="hexagon" ]
    }
    prod [ label="prod (v3.0 → v3.5)" ]
    test [ label="test (v3.5)" ]
    testnew [ label="testnew", style=dotted ]
    testold [ label="testold (v3.0)", color=green ]

    { rank=same test testnew testold }

    r305 -> test [ style=dotted ]
    r305 -> testnew [ style=invis ]
    test -> prod [ color=red, fontcolor=red, label=deploy ]
    r305 -> testold [ style=invis ]

    testnew -> test [ label="rename DB", color=blue, fontcolor=blue ]
    test -> testold [ label="rename DB", color=blue, fontcolor=blue ]
}

Instructions

  1. Update config.yml:

    1. Rename ${customer}test to ${customer}testold

    2. Rename ${customer}testnew to ${customer}test

    3. Update or add branch for production

    4. Add branch for testold if missing

    Example:

    example:
      # ...
      installations:
        example:
          branch: releases/1.0    # <-- set to target version
           # ...
        exampletestnew:    # <-- rename to 'exampletest'
          # ...
        exampletest:       # <-- rename to 'exampletestold'
          # ...
          branch: release/1.0     # <-- add if missing
          # ...
    
  2. If you don’t want to replace ${customer} and ${db_server} in the following steps manually, set them as variables in your terminal:

    customer=${customer}      # Replace ${…}
    db_server=${db_server}    #
    
  3. Create ${customer}testold and update configs of ${customer}test and ${customer}:

    cd ${GIT_ROOT}/tocco
    ansible-playbook playbook.yml -l ${customer}test,${customer}testold,${customer} --skip-tags monitoring
    
  4. Set downtime for ${customer}test and ${customer}testnew.

    See Schedule Downtime

  5. Stop test systems:

    oc -n nice-${customer}test scale --replicas 0 dc/nice
    oc -n nice-${customer}testnew scale --replicas 0 dc/nice
    
  6. Rename databases:

    ssh ${db_server} psql postgres -c "\"ALTER DATABASE nice_${customer}test RENAME TO nice_${customer}testold\""
    ssh ${db_server} psql nice_${customer}testold -c "\"REASSIGN OWNED BY nice_${customer}test TO nice_${customer}testold\""
    
    ssh ${db_server} psql postgres -c "\"ALTER DATABASE nice_${customer}test_history RENAME TO nice_${customer}testold_history\""
    ssh ${db_server} psql nice_${customer}testold_history -c "\"REASSIGN OWNED BY nice_${customer}test TO nice_${customer}testold\""
    
    ssh ${db_server} psql postgres -c "\"ALTER DATABASE nice_${customer}testnew RENAME TO nice_${customer}test\""
    ssh ${db_server} psql nice_${customer}test -c "\"REASSIGN OWNED BY nice_${customer}testnew TO nice_${customer}test\""
    
    ssh ${db_server} psql postgres -c "\"ALTER DATABASE nice_${customer}testnew_history RENAME TO nice_${customer}test_history\""
    ssh ${db_server} psql nice_${customer}test_history -c "\"REASSIGN OWNED BY nice_${customer}testnew TO nice_${customer}test\""
    

    Warning

    REASSIGN OWNED will change the owner of all objects in the DB connected to matching BY ${ROLE_NAME} and the owner of all DBs matching. Be careful when there are other DBs that have ${ROLE_NAME} as owner!

  7. Start ${customer}test again:

    oc -n nice-${customer}test scale --replicas 1 dc/nice
    
  8. Remove ${customer}testnew

    Skip removal of DNS record [1].

    See Remove Installation/Customer

  9. Deploy installation ${customer}test

    See Deliver (Simple)

  10. Deploy installation ${customer} (production)

    See Deliver (Simple)

  11. Deploy installation ${customer}testold

    See Deliver (Simple)

  12. Enable monitoring for testold:

    cd ${GIT_ROOT}/tocco
    ansible-playbook playbook.yml -l ${customer}testold -t monitoring
    

Complete Migration

Weeks later, when sure it isn’t needed anymore, testold can be removed.

digraph {
    label="Returning to Normal Operations (v3.5)"

    subgraph cluster_git {
        label="Git repository"
        r305 [ label="releases/3.5", shape=hexagon ]
    }
    prod [ label="prod (v3.5)" ]
    test [ label="test (v3.5)" ]
    testold [ style=dotted, label="testold (v3.0)" ]

    { rank=same test testold }

    r305 -> test [ style=dotted ]
    test -> prod [ style=dotted ]
}

Instructions

  1. Remove ${customer}testold

    Skip removal of DNS record [1].

    See Remove Installation/Customer

Footnotes