This is an internal documentation. There is a good chance you’re looking for something else. See Disclaimer.

Grafana

Login

Hint

Access to Grafana is only granted to members of Openshift groups: tocco-log-viewer, tocco-dev and tocco-admin.

  1. Log in on Tocco-Grafana.

  2. Open the Main menu via the Hamburger icon at the left upper corner and go to Dashboards.

Here you should see the folder Tocco Dashboards, which includes the dashboards for Nice-Logs, Webserver-Logs and Alerts among others.

Dashboards

Alerts

The Alerts dashboard displays firing and silenced alerts. silenced alerts are hidden by default, but can be included by setting the Include Silenced Alerts value in the upper left corner to true.

firing alerts include links to get to the nice and webserver logs, as well as one to silence the alert.

silenced alerts contain the same links as active alerts except for the silence action. They also show information about who silenced the alert, its start and end times.

../../_images/grafana_alerts-dashboard-example-1.png

Go to Alerts Dashboard

Log Dashboards

Log message dashboards are available for Nice-Logs and Webserver-Logs

Nice Logs

Logs messages from the nice2 container setup on openshift. In the upper dashboard area we can see the following filter options.

../../_images/grafana_nice_logs_1.png

  • namespace: sets one or multiple namespaces as target.

  • filter: Fulltext-Search over all log messages within the set time range.

  • level: define what log levels should be included (info, warn, error, debug)

  • Ad hoc Filters: Filter that can be added manually or when searching through the logs, via magnifying glass icon

../../_images/grafana_nice_logs_2.png

Per default the log message is displayed as one line and can be unfolded by clicking on it.

If we wan’t to filter or un-filter for a particular field inside the log message, we can do this by , clicking the magnifying glass with the + sign, to filter only for this value, or the - sign, to remove all messages matching this fields value.

../../_images/grafana_nice_logs_3.png

We can for example add the hostname by clicking the magnifying glass with the + sign, which results in this filter being added to the Ad hoc Filters

../../_images/grafana_nice_logs_4.png

Go to Nice-Logs Dashboard

Webserver Logs

Webserver logs are basically the same as Nice Logs but have two different filter options for http

../../_images/grafana_nginx_logs_1.png

  • method: Filter for GET, POST etc.

  • status: Filter for status 404, 200 etc.

Go to Webserver-Logs Dashboard

OOM Logs

Listens and displays OOM related log messages over all namespaces. Specific namespaces can also be defined by filter.

../../_images/OOM_figure1.png

A registered OOM log message triggers an alert notification sent to the Grafana-Slack channel and admin inbox. By adjusting the time range, we can also view instances that suffered an OOM within the specified time period.

Go to OOM Dashboard

Certificates

Shows current SSL-Certificate states and their expiry times for all namespaces routes.

../../_images/certificates_figure1.png

Go to Certificate Dashboard

Openshift-Node-Stats

This dashboard shows various stats about nodes on the OpenShift cluster. The left side displays a general overview of how many worker, master, and infra nodes are currently in use.

The right side shows details such as pod count and average memory usage for the node selected from the dropdown menu in the left upper corner.

Go to Openshift-Node-Stats Dashboard

Alert Setup and Silencing

Alert rules [1] are can be found at Alert Rules under the Grafana-managed section. Alert Rules for Nice and WordPress installations are located inside the OnCall_Alerts-Folder.

Alert Rules for OOM-Events and Services are located inside the NiceAlerts-Folder.

../../_images/grafana_alertrules_1.png

For details check the official Documentation Alert-Rule-Documentation.

Silencing alerts

Alerts can be silenced at Alert Silencing

  1. Click on the +Add Silence button on the right side

  2. Set a time range

  3. Choose one or multiple labels to specify what should be silenced. E.g. if you want to silence the whole namespace choose the label namespace and the value nice-${CUSTOMERNAME} check out the Hint box below for other labels.

  4. Add a comment and save

Hint

By using a different and/or combine multiple labels, we can specify the silence e.g. to a single route. The following labels might be helpful:

  • namespace : nice-${CUSTOMERNAME}.ch

  • domain : tocco.${CUSTOMERNAME}.ch

  • instance : https://tocco.${CUSTOMERNAME}.ch/status-tocco

  • job : nice or wordpress

According to the labels added, grafana will show a list of currently active Alerts, that would be affected by the created Silence. This list contains more labels that can be used if needed.

../../_images/silenced_alerts.png

The created Silence should then be listed together with other active Silences.

../../_images/grafana_alertrules_2.png

Hint

Silences can also be directly created via the Alerts Dashboards create silence link.

Via the Expired Silences dropdown, we see a Silence history and can reactivate them with Recreate.

Notification policies and fixed Downtimes (mute timings)

To which endpoints the alerts are delivered and the setup of fixed downtimes/maintenance windows can be configured at Alert Routes.

Notification policies

Policies to determine various aspects on alert delivery, repetition and contact point definition etc. By now only the default-policy is set, that sends all alerts to the Tocco-Oncall setting in Jira.

Mute Timings

A mute timing is a recurring interval that stops notifications for one or multiple notification policies during a specified period. They can be used to pause notifications for a specific recurring period, such as a regular maintenance window or weekends.

../../_images/grafana_mute_timing_1.png

At the moment only VSHN’s maintenance window on Tuesday evening is set as a mute timing. The process of setting a new mute timing is pretty self-explaining and can be done via the + Add mute timing button.

Hint

When trying to delete a mute timing, we have to make sure that all its references/evaluation points are either gone or not active, otherwise we get a conflict message back.

Contact points and Notifications

Contact points allow us to define various endpoints for alerts and notifications coming from within Grafana. There are already a lot of presets for most common messenger Services like Slack, OpsGenie, Telegram etc. of course also eMail. Defined at Contact Points.

As of now, some alerts are higher prioritized than others and got to the grafana Slack channel s#grafana-alerts and the admin mailbox:

  • oom-alerts - slack & mail

  • nice-prod - slack & mail

  • nice-stage - mail

  • wp-prod - slack & mail

  • wp-stage - no alert

Adjustments to the notification messages etc. can be done at Notification Setting.

Git sync

Grafana version 13.1 introduced git sync, which allows us to synchronize and store dashboards as JSON or YAML files in any Git provider and manage them as code. For more details, see git-sync.

The repository is located at git-sync-Repo.

Folders managed by git sync display a purple icon next to their names:

../../_images/git-sync-icon.png

If we make changes to any dashboard inside these folders and try to save them, the following menu appears:

../../_images/git-sync-savecontext1.png

Since the master branch is protected, we must create a new branch and submit a merge request to add our changes to master. At this point, the changes are only visible locally, as shown by a the banner appearing at the top of the dashboard:

../../_images/git-sync-banner.png

Once changes are merged into master, an automated sync job that runs hourly will deploy the newest version across the organization.

For very urgent changes, we can trigger the job manually at Provisioning Settings:

../../_images/provisioning_settings1.png

Hint

Why is there a warning during some sync tasks ?

At this point, git sync is only compatible with dashboard objects. The warning appears because some of these folders also contain alert objects. These objects will hopefully be implemented into git sync in the near future.

This also means that alert rules still need to be exported and uploaded to Git manually.

Footnotes