This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

CI

Setup

Gitlab is used as host provider and CI of the tocco-client project.

The triggered pipelines can be inspected here: Gitlab CI Pipelines.

Stages and Jobs

Alls jobs can be found in the .gitlab-ci.yml file in the root.

There are a set of jobs that are triggered for every merge request and push to master. The jobs ensure, that the linting command and all unit tests run successfully and storybook can be build. Also the builded storybook gets pushed to the Gitlab Storybook Repository and a link to this storybook deployment is written as comment to the merge request.

There is also a scheduled pipeline that runs daily on master testing all the above and also runs the more time consuming end-to-end tests with cypress.

Also there are two jobs that support the development workflow. One job handles the auto-releasing of packages on the release branches (if the changelog is not empty a package is released) on each commit. The last job handles cherry-picking across different versions and is described here Cherry-pick job

Caching

All node_modules folder are getting cached since a installation of all dependencies from scratch takes very long. The yarn.lock files act as keys. That means if a dependency changes, a new cache is created. Each job should define if it pull, pushed or pull and pushed to the cache to speed things up.

Image

As docker image the linux alpine version is used to speed up the image initialization phase by several seconds. As downside of this optimization, some dependencies must be installed by a job with apk.

Releasing

Manual trigger releasing

Using schedule (releasing all packages)

If all packages should be released open Pipeline Schedules and run the job of the corresponding branch.

Using pipeline (releasing only some packages)

Open Run Pipeline on Gitlab. Select the branch where you want to manually release (either master or nice-releases/VERSION). The following variables can be passed:

  • CI_RUN_AUTO_RELEASE (mandatory): must be set to 1 that the releasing is started

  • CI_RELEASING_PACKAGES (optional): per default the releasing is started for all releasable packages. if a comma-separated list is passed (e.g. admin,devcon) only these packages are release candidates

  • CI_RELEASING_FORCE (optional): per default a package is only released if the changelog is not empty. if true is passed as value the package is always released

Fix failing job

If the releasing job fails on the Gitlab CI, some packages are already published to NPM but the new version is not committed to the release/master branch. First you must manually change these versions (check log to know which packages were already released). Commit these changes and merge them. Last you must move for each package the git tag. You must execute:

git checkout ${COMMIT_ID}
git tag ${PACKAGE_NAME}/${RELEASE_TAG} -f
git push origin ${PACKAGE_NAME}/${RELEASE_TAG} -f

  • COMMIT_ID: commit id on the release/master branch of your created commit on Gitlab

  • PACKAGE_NAME: npm package name which always has the tocco prefix (e.g. tocco-admin)

  • RELEASE_TAG: npm release tag name (e.g. nice34). On the master the tag is named like the upcoming version

Cherry-pick job

To explain this rater complex job.

Naming

  • source branch: branch on which the CI job is running (e.g. nice-releases/227)

  • target branch: branch to which the commits should be cherry-picked (next release branch or master; e.g. nice-releases/228)

  • auto merge tag: each release branch has a git tag like auto-merge/${VERSION}. All commits before this tag (including the commit itself) are already cherry-picked (if necessary)

Workflow

  • setup variables (auto merge tag, source branch, nice version)

  • configure git and clone repo (only source & target branch and all tags of these branches)

  • get commit id which belongs to the auto merge tag of the source branch (e.g. tag auto-merge/227)

  • checkout source branch

  • retrieve commits which should be cherry-picked. Such a commit has a Cherry-pick:Up in the commit message body and are after the auto merge tag in the history (=not already cherry-picked)

  • stop if nothing should be cherry-picked

  • checkout target branch

  • cherry pick commit by commit

  • create and push new branch cherry-picking/_TARGET-BRANCH_-auto-xxxxxxxx

  • move auto merge tag to head of source branch

Fix failing cherry-pick job

For example there could be an auto merge conflict. In such a case there is log message like this:

CONFLICT (content): Merge conflict in packages/entity-browser/src/routes/list/containers/ListViewContainer.js

Workflow for fixing (in the example the source branch is nice-releases/227 and the target branch nice-releases/228) :

  • Check log which commits are cherry-picked (see line 2 und 3):

c2b6b34cdca570fd36bdef75682b702a9702938b is commit id of tag auto-merge/227
00f9bed1a0969efceaaed43e10c952f1e0fd57f0 has 'Cherry-pick: Up' in commit message
952f1e0fd57f000f9bed1a0969efceaaed43e10c has 'Cherry-pick: Up' in commit message
Switched to branch 'nice-releases/228'

  • Checkout the target branch

  • Cherry pick the commits with git cherry-pick ${COMMIT_ID} -x and fix conflicts

  • Create a PR

  • Optional: if the PR is approved release all necessary packages (all necessary packages will be released automatically every night)

  • Move the auto merge tag to the head of the source branch

git checkout ${HEAD_COMMIT_ID}
git tag -f auto-merge/${SOURCE_BRANCH_NICE_VERSION}
git push origin auto-merge/${SOURCE_BRANCH_NICE_VERSION} -f

Merge Request

There is one job that automates the creation of merge requests after a successful release of a package (or cherry picking), the branch containing the release (or cherry picked) commits must be rebased to its target branch using a merge request.

CI scripts on different versions

In version 2.27 many client CI scripts were introduced or adjusted such as auto-releasing or cherry-picking. If a CI feature is added or modified this should be done ideally on version 2.27 if possible. However between 2.27 and the latest version are some differences in the scripts which are documented here. The command git diff origin/nice-releases/34 .gitlab-ci.yml scripts ':!scripts/nice2' ':!scripts/widgets' (on branch nice-releases/35) is used to extract the difference (Folder nice2 and widgets are excluded as the are only used by Cypress which runs on the master branch).

Changed in version 2.28

tagging.sh

  • exit 0 removed

Changed in version 2.29

helpers.sh

  • in method setNiceVersion command adjusted

release-package.sh

  • return statement adjusted

Changed in version 3.1

.gitlab-ci.yml

  • image from node 14 to node 16 updated

  • formatting job added

  • cypress job

    • CYPRESS_INSTALL_BINARY version changed

    • execution command adjusted

    • artifacts storage added

  • sonarcube job added

msg-validation.js

  • code reformatted

Changed in version 3.2

.gitlab-ci.yml

  • yarn_setup job packages/*/ replaced with packages/*/*/

  • test job artifacts storage added

  • cypress job expire_in changed

  • sonarqube job before_script added

helpers.sh

  • in method getDevDependenciesGreps command adjusted to get dev dependencies

  • method setPackageDir added

  • method checkPackage adjusted

release-message.js

  • script added

release-package.sh

  • adjust navigation to folder

  • add slack message for package release => release-message.js

release-packages.sh

  • adjust iterating over folders

tag-all-packages.sh

  • script added

Changed in version 3.3

.gitlab-ci.yml

  • cypress job

    • upgrade cypress image

    • CYPRESS_INSTALL_BINARY version changed

helpers.sh

  • setPackageDir supports now bundle packages

tag-all-packages.sh

  • add registry to npm dist-tag add ... command

Changed in version 3.4

.gitlab-ci.yml

  • variables NICE_VERSION, DOCKER_NICE_VERSION, NICE_TIMEOUT_SECS, S3_SECRET_ACCESS_KEY, CYPRESS_API_KEY, CYPRESS_PASSWORD, CYPRESS_USER, CYPRESS_USER_API_KEY_HASH, CYPRESS_USER_PASSWORD_HASH added

  • .nice-as-service added

  • yarn_setup job runs now also on schedules

  • cypress job

    • extends .nice-as-service

    • CYPRESS_INSTALL_BINARY version changed

    • variables HIBERNATE_MAIN_SERVERNAME, HIBERNATE_MAIN_DATABASENAME, POSTGRES_USER, CYPRESS_BASE_URL, BACKEND_URL

    • replace scripts

msg-validation.js

  • add command

  • remove unused code

Changed in version 3.5

.gitlab-ci.yml

  • variable NICE_VERSION removed

  • variable POSTGRES_VERSION added and Postgres updated to v15

  • test job

    • artifacts are no longer saved

    • command yarn test:ci instead of yarn test --ci

  • cypress job

    • CYPRESS_INSTALL_BINARY version changed

    • add command yarn db:create-template

    • run cypress with --browser chrome

  • sonarqube job no longer depends on test job

helpers.sh

  • in method getDevDependenciesGreps command adjusted to extract dev dependencies after updating lerna

Changed in version 3.6

.gitlab-ci.yml

  • image from node 16 to node 18 updated

  • cypress job

    • CYPRESS_INSTALL_BINARY version changed

Changed in version 3.8

.gitlab-ci.yml

  • removed tagging script

  • adjust command for releasing

else

  • rewrite of releasing script

    • added files helpers/*, releasePackages.js, releasePackagesHelpers.js, releasePackagesHelpers.spec.js and variables.json

    • removed files auto-release.sh, helpers.sh, release-package.sh, release-packages.sh and tagging.sh.sh