Notifications

The SMI comes equipped with different notification streams to verify the event status (cluster or node synchronization) or synchronization state of the current cluster or nodes in it.

The show notification stream provides notification details for alert notifications, license status and system status.

show notification stream ? 
Possible completions:
  alert-notification   Notifications for alerts
  license-status       Notifications for smart agent licensing will be generated here
  system-status        Notifications for system status

The NSO must subscribe to the alert-notification notification stream to receive alerts.

You can access the notification stream using the following command show command in the SMI Cluster Manager CLI:

show notification stream {node-state | node-status | sync-state | sync-status} 

Sync-State Notification

You can access the synchronization state of a given cluster using the sync-state notification. After every cluster synchronization, a sync-state notification is registered as sent.

The following example displays the deployment of a stacked cluster:

SMI Cluster Deployer# show notification stream sync-state
notification 
 eventTime 2020-10-07T18:58:27.368+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
notification 
 eventTime 2020-10-07T19:07:56.114+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.531+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
notification 
 eventTime 2020-10-08T15:59:56.697+00:00
 sync-state-notification 
  cluster <stacked_cluster_name>
  state DEPLOYED
 !
!
The node-state-notification could be used to assess the state of the node in a cluster deloyment sync process. Below example shows the state of control plane and worker nodes are joined in <stacked_cluster_name> cluster. Before reaching JOINED state if the join process is still running they would show up as JOINING.
[installer-controlplane] SMI Cluster Deployer# show notification stream node-state
notification 
 eventTime 2020-10-07T19:46:54.531+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.532+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.534+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane1
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.54+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker1
  state JOINED
 !
!

Node-State Notification

You can view the node state in a cluster deployment synchronization process using the node-state notification.

The following example displays the state of control plane and worker nodes that have joined the stacked cluster. The node state is displayed as JOINING , if the process is running. Otherwise, the node state is displayed as JOINED .

SMI Cluster Deployer# show notification stream node-state
notification 
 eventTime 2020-10-07T19:46:54.531+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.532+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.534+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker2
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane3
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.535+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node controlplane1
  state JOINED
 !
!
notification 
 eventTime 2020-10-07T19:46:54.54+00:00
 node-state-notification 
  cluster <stacked_cluster_name>
  node cmts-worker1
  state JOINED
 !
!

Sync-Status Notification

You view the progression of the cluster synchronization process using the sync-status notification.

The following example displays the different states (SYNC-START, ERROR and DONE) of the cluster synchronization process:

SMI Cluster Deployer# show notification stream sync-status
notification 
 eventTime 2020-10-08T16:05:53.326+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status ERROR
 !
!
notification 
 eventTime 2020-10-08T16:12:48.042+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T16:26:23.444+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status ERROR
 !
!
notification 
 eventTime 2020-10-08T16:26:23.446+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status ERROR
 !
!
notification 
 eventTime 2020-10-08T16:31:42.551+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T16:43:33.498+00:00
 sync-status-notification 
  cluster <stacked_cluster_name>
  status DONE
 !
!
notification 
 eventTime 2020-10-08T20:04:21.386+00:00
 sync-status-notification 
  cluster kali-stacked
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T20:04:21.386+00:00
 sync-status-notification 
  cluster kali-stacked
  status SYNC-START
 !
!
notification 
 eventTime 2020-10-08T20:27:32.155+00:00
 sync-status-notification 
  cluster kali-stacked
  status DONE
 !
!

Node-Status Notification

You can view the status of the node synchronization using the node-status notification. The following example displays the status of the nodes in a stacked cluster:

SMI Cluster Deployer# show notification stream node-status
notification 
 eventTime 2020-09-25T19:59:56.728+00:00
 node-status-notification 
cluster kali-vm3
node controlplane
status DONE
 !
!
notification 
 eventTime 2020-09-29T16:36:05.962+00:00
 node-status-notification 
cluster kali
node etcd1
status DONE
 !
!
notification 
 eventTime 2020-09-29T20:54:18.415+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status DONE
 !
!
notification 
 eventTime 2020-09-30T01:28:57.355+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status DONE
 !
!
notification 
 eventTime 2020-09-30T01:46:53.034+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status DONE
 !
!
notification 
 eventTime 2020-09-30T04:17:25.701+00:00
 node-status-notification 
cluster kali-stacked
node controlplane1
status ERROR
 !
!
notification 
 eventTime 2020-10-06T20:02:30.108+00:00
 node-status-notification 
cluster kali-stacked
node cmts-worker1
status DONE
 !
!

VM Status Alerts

The NSO needs to know the state of the NF (Network Function) that has been deployed and notifications for the following states:

  • DEPLOYED

  • ALIVE

  • UNDEPLOYED

  • ERROR

  • RECOVERING

  • RECOVERY_FAILED

Note

The SMI supports VM status notification for UPF only.

The following parameters are introduced in the notification streams for the VM status notifications:

Parameter

Description

vm-state-notification stream

The vm-state-notification stream has the following details:

  • cluster_name: the cluster holds KVM nodes

  • node_name: KVM node name

  • vm_name: UPF name

  • state: VM state

  • message: Carries useful information such as error message or mgmt-ip when the state of the VM is ALIVE.

alert-notification stream

The alert-notification stream has the following labels:

  • node_name: KVM node name

  • vm_name: UPF name

  • state: VM state

  • message: Carries useful info such as error message or mgmt-ip when state is ALIVE.

Depending on the life cycle stage of the VM, notifications are generated from either the Cluster Manager or CEE Ops-Center and sent to the NSO. When VM gets deleted or redeployed, the UNDEPLOYED notification is sent to the Cluster Manager notification stream. All other notifications are generated by the Alert Manager and then sent to the CEE Ops-Center notification stream.

Cluster Manager Notification

The Cluster Manager sends vm-state-notification for the UNDEPLOYED VM state.

CEE Ops-Center Notification

The CEE Ops-Center alert-notification sends the following alerts for different VM states:

  • vm-deployed: minor - DEPLOYED

  • vm-alive: minor – ALIVE (alert lasts for a short time and disappears automatically)

  • vm-error: major - ERROR

  • vm-recovering: warning - RECOVERING

  • vm-recovery_failed: critical - RECOVERY_FAILED

All required fields are included in alert labels for notification from alert-notification. All VM alerts are viewable on the Grafana dashboard.

VM Action Notifications

Delete Action: When delete VM action is triggered, CM sends notifications that the VM is deleted. The VM states are UNDEPLOYED and ERROR for vm delete action.

clusters abc-cluster-15 nodes kvm-1 vms upf1 actions delete

Redeploy Action: When VM is in RECOVERY_FAILED state, NSO sends a request to redeploy the VM. A redeploy action does both delete action and sync action.

clusters abc-cluster-15 nodes kvm-1 vms upf1 actions redeploy

Redeploy Action Notification: The redeploy action sends a notification to the CM. The redeploy vm action has the following 4 states: UNDEPLOYED, ERROR, REDEPLOYED, REDEPLOY_ERROR.

show notification stream vm-state 

notification
eventTime 2021-02-23T21:27:28.692+00:00
vm-state-notification
cluster_name cndp-testbed
node_name kvm-1
vm_name upf2
state UNDEPLOYED
message
!
!
notification
eventTime 2021-02-23T21:29:18.699+00:00
vm-state-notification
cluster_name cndp-testbed
node_name kvm-1
vm_name upf2
state REDEPLOYED
message
!
!

Configuring the Alert Notification in CEE

The user must configure alert notifications when they deploy the UPF VMs. Log in to the CEE cli to add the following configuration:


config
bulk-stats prune-interval-days 3
prometheus kvm-metrics defaults private-key "-----BEGIN OPENSSH PRIVATE KEY-----LGXtil23N4YV=\n-----END OPENSSH PRIVATE KEY-----\n"
prometheus kvm-metrics defaults user cloud-user
prometheus kvm-metrics monitor-server 10.194.62.41
hostname abc-bm-15-controlplane
exit
Note

The user must replace the IP, hostname, private key and user details.

Sample Notification from the Alert Notification Stream


notification
eventTime 2021-01-08T03:28:54.501+00:00
smi-alert-notification
starts-at 2021-01-08T03:28:24.493874101Z
ends-at 0001-01-01T00:00:00Z
alert-status firing
smi-alert-notification alert-label
name alertname
value vm-recovery-failed
!
smi-alert-notification alert-label
name cluster
value test-cee-kvm_cee-voice
!
smi-alert-notification alert-label
name hostname
value test-bm-15-controlplane
!
smi-alert-notification alert-label
name instance
value metrics-proxy-test-bm-15-controlplane:9100
!
smi-alert-notification alert-label
name job
value metrics-proxy
!
smi-alert-notification alert-label
name message
value 10.1.1.3
!
smi-alert-notification alert-label
name monitor
value prometheus
!
smi-alert-notification alert-label
name node_name
value controlplane
!
smi-alert-notification alert-label
name replica
value test-cee-kvm_cee-voice
!
smi-alert-notification alert-label
name severity
value critical
!
smi-alert-notification alert-label
name state
value RECOVERY_FAILED
!
smi-alert-notification alert-label
name vm_name
value upf2
!
smi-alert-notification alert-annotation
name summary
value upf2 failed to recover.
!
smi-alert-notification alert-annotation
name type
value Equipment Alarm
!
!
!