This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Move Database between Servers

Checklist for moving DBs between clusters.

Pre-Move

(prod-only) Announce downtime ~one week in advance.

Check disk space:

$ tco db-size ${installation}
$ ssh ${target_server} df -h /var/lib/postgresql/

(unannounced / test-only) check for users:
```
$ tco sessions ${installation}
```
If anyone is using the installation, consider postponing the move. Use tco sessions -v to see additional details.

Move

Update db_server in config.yml and create new DBs and user.

When you copy the history DB (prod-only):
```
$ ansible-playbook playbook.yml -l ${installation} -t postgres --skip-tags no-init-history-db
```
When you skip copying the history DB:
```
$ ansible-playbook playbook.yml -l ${installation} -t postgres
```
Note

Do not bother copying DB for test systems unless explicitly requested.
Start maintenance page:
```
$ oc project nice-${installation}
$ tocco-mntnc --auto-scale start
```
This will stop the installation.

See Maintenance Page

Rename DBs on old server:

$ tco db -t postgres ${installation}
$ ALTER DATABASE nice_${installation} RENAME TO nice_${installation}_moved;
$ ALTER DATABASE nice_${installation}_history RENAME TO nice_${installation}_history_moved;

Note

Safety measure Nice2 cannot be started again with old DB settings.

Remember to include the “_moved” suffix in ${source_db} from now on.

(prod-only) move / initialize history DB:
```
$ tco cp ${source_host}/${source_db} ${target_host}/${target_db}
```
Hint

Alternatives
1. Use direct server-to-server copying
  
  Some additional setup is required but this is generally much faster as it avoids downloading and reuploading the DB.
  1. Get public key from target server:
    $ ssh ${target_server} cat .ssh/id_ed25519.pub
    Create a key if file doesn’t exist:
    $ ssh ${target_server} ssh-keygen -N '' -t ed25519 -f '~/.ssh/id_ed25519'
  2. Create authorized_keys2 file on source server:
    $ ssh ${source_server} tee '~/.ssh/authorized_keys2' <<<'command="pg_dump -Fc -Z 0 ${source_db}",restrict ${public_key_from_previous_step}'
  3. Copy:
    $ ssh ${target_server} 'ssh ${source_server} | pg_restore --role nice_${installation_name} --no-owner --no-acl -d ${target_db}'
  4. Remove authorization again:
    $ ssh ${source_server} rm .ssh/authorized_keys2
2. Use copy_history_db
  
  An alternative is to use copy_history_db. It allows initializing the history DB now and delaying copying. Nice2 will be allowed to add new history records while delaying copying to after the move:
  $ copy_history_db --init-only ${source_host} ${source_db}_moved ${target_host} ${target_db}
  Use this to reduce downtime when copying very large DBs.
  
  Caveats:
  - Copying is much, much slower. It’ll potentially take days.
  - History records from before the move will not show up until copying is completed.
Move main DB:
```
$ tco cp ${source_host}/${source_db} ${target_host}/${target_db}
```
Hint

Direct server-to-server copying described above is also possible here.
Update OpenShift config:
```
$ ansible-playbook playbook.yml -l ${installation} --skip-tags ingress
```
Note

Skipping ingress to prevent reverting changes made by tocco-mntnc.

Verify deployment succeeded:

$ oc project nice-${installation}
$ oc get pods

Check pod is READY.

Stop maintenance page:
```
$ tocco-mntnc --auto-scale stop
```
Verify installation is back online

Post-Move

(prod-only) On the next day complete move of history DB

Only do this when you used copy_history_db --init earlier:
```
$ copy_history_db --skip-init ${source_host} ${source_db} ${target_host} ${target_db}
```
Note

Do not do this overnight as we run into issues before. Specifically, we ran out of disk space.
Remove old DBs four days later:
```
tco dbs unused --ask-delete
```