Viewing Alert Logger

The Alert Logger stores all the generated alerts by default. You can view the stored alerts using the following show commands.

show alert history { detail | summary }

show alert active { detail | summary }

You can narrow down the result using the following filtering options:

  • annotations – Specifies the annotations of the alert.

  • endsAt – Specifies the end time of the alert.

  • labels – Specifies the additional labels of the alert.

  • severity – Specifies the severity of the alert.

  • source – Specifies the source of the alert.

  • startsAt – Specifies the start time of the alert.

  • type – Specifies the type of the alert.

You can view the history of configured alerts using show alerts history command.

The following examples displays the history of the alerts configured in the system:

Example:
cee# show alerts history summary
NAME UID SEVERITY STARTS AT DURATION SOURCE SUMMARY
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
k8s-pod-crashing-loop 13218bfedfb7 critical 11-02T19:42:40 3m50s upf-cm-tb16-2-cm1 Pod cee-global/alert-logger-56f85f54df-wdpbp (alert-logger) is restarting 1.01 times / 5 minutes.
k8s-pod-crashing-loop bf8f6b0e167c critical 11-02T19:42:40 3m50s upf-cm-tb16-2-cm1 Pod cee-global/pgpool-5cc9d4b44f-4kklz (pgpool) is restarting 1.01 times / 5 minutes.
k8s-pod-crashing-loop 840f362e970e critical 11-02T19:42:40 3m50s upf-cm-tb16-2-cm1 Pod cee-global/grafana-5b9779c7d6-hmptk (grafana) is restarting 1.01 times / 5 minutes.
k8s-pod-crashing-loop 40f4de09d667 critical 11-02T19:42:30 3m50s upf-cm-tb16-2-cm1 Pod cee-global/pgpool-5cc9d4b44f-gwdpp (pgpool) is restarting 1.01 times / 5 minutes.
k8s-pod-not-ready 3ade1624bfa8 critical 11-02T19:40:40 40s postgres-0 Pod cee-global/postgres-0 has been in a non-ready state for longer than 1 minute.

The following examples displays a detailed history of the alerts configured in the system:

cee# show alerts history detail
alerts history detail k8s-pod-crashing-loop 13218bfedfb7
severity critical
type "Processing Error Alarm"
startsAt 2020-11-02T19:42:40.400Z
endsAt 2020-11-02T19:46:30.400Z
source upf-cm-tb16-2-cm1
summary "Pod cee-global/alert-logger-56f85f54df-wdpbp (alert-logger) is restarting 1.01 times / 5 minutes."
labels [ "alertname: k8s-pod-crashing-loop" "cluster: upf-cm_cee-global" "component: kube-state-metrics" "container: alert-logger"
 "hostname: upf-cm-tb16-2-cm1" "instance: 192.168.211.203:8080" "job: kubernetes-pods" "monitor: prometheus" 
"namespace: cee-global" "pod: alert-logger-56f85f54df-wdpbp" "pod_template_hash: db7bf9f7" 
"release: cee-global-cnat-monitoring" "replica: upf-cm_cee-global" "severity: critical" ]
annotations [ "summary: Pod cee-global/alert-logger-56f85f54df-wdpbp (alert-logger) is restarting 1.01 times / 5 minutes."
 "type: Processing Error Alarm" ]

You can view the active using the show alerts active command.

Example:
show alerts active summary
NAME UID SEVERITY STARTS AT SOURCE SUMMARY
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
server-alert 02232d49cccd minor 10-29T06:09:04 upf-4 PS_RDNDNT_MODE: Power Supply redundancy is lost or non-redundant: Check Redundancy Policy or reseat/replace Power Supply
server-alert f97ec27bc318 minor 10-29T06:09:04 cm-2 PS_RDNDNT_MODE: Power Supply redundancy is lost or non-redundant: Check Redundancy Policy or reseat/replace Power Supply
watchdog 0dbfe73527ad minor 10-29T06:07:58 System This is an alert meant to ensure that the entire alerting pipeline is functional. This alert is always firing, therefore it should always be firing...
Example:
show alerts active detail
alerts active detail server-alert 359fe8fd1dd8
severity warning
type "Equipment Alarm"
startsAt 2020-10-29T06:09:04.243Z
source cm-2
summary "Storage Virtual Drive 0 Degraded: please check the storage controller, or reseat the storage drive"
labels [ "alertname: server-alert" "cluster: tb16-2" "description: Storage Virtual Drive 0 Degraded: 
please check the storage controller, or reseat the storage drive" "fault_id: sys/rack-unit-1/board/
storage-SAS-MRAID/vd-0/fault-F1008" "id: 3523411968" "monitor: prometheus" "replica: tb16-2" 
"server: cm-2" "severity: warning" ]
annotations [ "dn: tb16-2/cm-2/sys/rack-unit-1/board/storage-SAS-MRAID/vd-0/fault-F1008/3523411968" 
"summary: Storage Virtual Drive 0 Degraded: please check the storage controller, or reseat the 
storage drive" "type: Equipment Alarm" ]