System Event Messages Reference

The following table lists all system event messages organized by type. These messages can be viewed through third-party notification applications. For details about configuring notifications, refer to Configure Notifications.

For a list of threshold-based events supported, refer to TCA Supported Events. For a list of What Just Happened events supported, refer to WJH Supported Events.

Agent Events

TypeTriggerSeverityMessage FormatExample
agentNetQ Agent state changed to Rotten (not heard from in over 15 seconds)CriticalAgent state changed to rottenAgent state changed to rotten
agentNetQ Agent rebootedCriticalNetq-agent rebooted at (@last_boot)Netq-agent rebooted at 1573166417
agentNode running NetQ Agent rebootedCriticalSwitch rebooted at (@sys_uptime)Switch rebooted at 1573166131
agentNetQ Agent state changed to FreshInfoAgent state changed to freshAgent state changed to fresh
agentNetQ Agent state was resetInfoAgent state was paused and resumed at (@last_reinit)Agent state was paused and resumed at 1573166125
agentVersion of NetQ Agent has changedInfoAgent version has been changed old_version:@old_version and new_version:@new_version. Agent reset at @sys_uptimeAgent version has been changed old_version:2.1.2 and new_version:2.3.1. Agent reset at 1573079725

BGP Events

TypeTriggerSeverityMessage FormatExample
bgpBGP Session state changedCriticalBGP session with peer @peer @neighbor vrf @vrf state changed from @old_state to @new_stateBGP session with peer leaf03 leaf04 vrf mgmt state changed from Established to Failed
bgpBGP Session state changed from Failed to EstablishedInfoBGP session with peer @peer @peerhost @neighbor vrf @vrf session state changed from Failed to EstablishedBGP session with peer swp5 spine02 spine03 vrf default session state changed from Failed to Established
bgpBGP Session state changed from Established to FailedInfoBGP session with peer @peer @neighbor vrf @vrf state changed from established to failedBGP session with peer leaf03 leaf04 vrf mgmt state changed from down to up
bgpThe reset time for a BGP session changedInfoBGP session with peer @peer @neighbor vrf @vrf reset time changed from @old_last_reset_time to @new_last_reset_timeBGP session with peer spine03 swp9 vrf vrf2 reset time changed from 1559427694 to 1559837484

BTRFS Events

TypeTriggerSeverityMessage FormatExample
btrfsinfoDisk space available after BTRFS allocation is less than 80% of partition size or only 2 GB remain.Critical@info : @detailshigh btrfs allocation space : greater than 80% of partition size, 61708420
btrfsinfoIndicates if space would be freed by a rebalance operation on the diskCritical@info : @detailsdata storage efficiency : space left after allocation greater than chunk size 6170849.2","

Cable Events

TypeTriggerSeverityMessage FormatExample
cableLink speed is not the same on both ends of the linkCritical@ifname speed @speed, mismatched with peer @peer @peer_if speed @peer_speedswp2 speed 10, mismatched with peer server02 swp8 speed 40
cableThe speed setting for a given port changedInfo@ifname speed changed from @old_speed to @new_speedswp9 speed changed from 10 to 40
cableThe transceiver status for a given port changedInfo@ifname transceiver changed from @old_transceiver to @new_transceiverswp4 transceiver changed from disabled to enabled
cableThe vendor of a given transceiver changedInfo@ifname vendor name changed from @old_vendor_name to @new_vendor_nameswp23 vendor name changed from Broadcom to Mellanox
cableThe part number of a given transceiver changedInfo@ifname part number changed from @old_part_number to @new_part_numberswp7 part number changed from FP1ZZ5654002A to MSN2700-CS2F0
cableThe serial number of a given transceiver changedInfo@ifname serial number changed from @old_serial_number to @new_serial_numberswp4 serial number changed from 571254X1507020 to MT1552X12041
cableThe status of forward error correction (FEC) support for a given port changedInfo@ifname supported fec changed from @old_supported_fec to @new_supported_fecswp12 supported fec changed from supported to unsupported

swp12 supported fec changed from unsupported to supported

cableThe advertised support for FEC for a given port changedInfo@ifname supported fec changed from @old_advertised_fec to @new_advertised_fecswp24 supported FEC changed from advertised to not advertised
cableThe FEC status for a given port changedInfo@ifname fec changed from @old_fec to @new_fecswp15 fec changed from disabled to enabled

CLAG/MLAG Events

TypeTriggerSeverityMessage FormatExample
clagCLAG remote peer state changed from up to downCriticalPeer state changed to downPeer state changed to down
clagLocal CLAG host MTU does not match its remote peer MTUCriticalSVI @svi1 on vlan @vlan mtu @mtu1 mismatched with peer mtu @mtu2SVI svi7 on vlan 4 mtu 1592 mistmatched with peer mtu 1680
clagCLAG SVI on VLAN is missing from remote peer stateWarningSVI on vlan @vlan is missing from peerSVI on vlan vlan4 is missing from peer
clagCLAG peerlink is not opperating at full capacity. At least one link is down.Warning

Clag peerlink not at full redundancy, member link @slave is downClag peerlink not at full redundancy, member link swp40 is down
clagCLAG remote peer state changed from down to upInfoPeer state changed to upPeer state changed to up
clagLocal CLAG host state changed from down to upInfoClag state changed from down to upClag state changed from down to up
clagCLAG bond in Conflicted state was updated with new bondsInfoClag conflicted bond changed from @old_conflicted_bonds to @new_conflicted_bondsClag conflicted bond changed from swp7 swp8 to @swp9 swp10
clagCLAG bond changed state from protodown to up stateInfoClag conflicted bond changed from @old_state_protodownbond to @new_state_protodownbondClag conflicted bond changed from protodown to up

CL Support Evemts

TypeTriggerSeverityMessage FormatExample
clsupportA new CL Support file has been created for the given nodeCriticalHostName @hostname has new CL SUPPORT fileHostName leaf01 has new CL SUPPORT file

Config Diff Events

TypeTriggerSeverityMessage FormatExample
configdiffConfiguration file deleted on a deviceCritical@hostname config file @type was deletedspine03 config file /etc/frr/frr.conf was deleted
configdiffConfiguration file has been createdInfo@hostname config file @type was createdleaf12 config file /etc/lldp.d/README.conf was created
configdiffConfiguration file has been modifiedInfo@hostname config file @type was modifiedspine03 config file /etc/frr/frr.conf was modified

EVPN Events

TypeTriggerSeverityMessage FormatExample
evpnA VNI was configured and moved from the up state to the down stateCriticalVNI @vni state changed from up to downVNI 36 state changed from up to down
evpnA VNI was configured and moved from the down state to the up stateInfoVNI @vni state changed from down to upVNI 36 state changed from down to up
evpnThe kernel state changed on a VNIInfoVNI @vni kernel state changed from @old_in_kernel_state to @new_in_kernel_stateVNI 3 kernel state changed from down to up
evpnA VNI state changed from not advertising all VNIs to advertising all VNIsInfoVNI @vni vni state changed from @old_adv_all_vni_state to @new_adv_all_vni_stateVNI 11 vni state changed from false to true

Lifecycle Management Events

TypeTriggerSeverityMessage FormatExample
lcmCumulus Linux backup started for a switch or hostInfoCL configuration backup started for hostname @hostnameCL configuration backup started for hostname spine01
lcmCumulus Linux backup completed for a switch or hostInfoCL configuration backup completed for hostname @hostnameCL configuration backup completed for hostname spine01
lcmCumulus Linux backup failed for a switch or hostCriticalCL configuration backup failed for hostname @hostnameCL configuration backup failed for hostname spine01
lcmCumulus Linux upgrade from one version to a newer version has started for a switch or hostCriticalCL Image upgrade from version @old_cl_version to version @new_cl_version started for hostname @hostnameCL Image upgrade from version 4.1.0 to version 4.2.1 started for hostname server01
lcmCumulus Linux upgrade from one version to a newer version has completed successfully for a switch or hostInfoCL Image upgrade from version @old_cl_version to version @new_cl_version completed for hostname @hostnameCL Image upgrade from version 4.1.0 to version 4.2.1 completed for hostname server01
lcmCumulus Linux upgrade from one version to a newer version has failed for a switch or hostCriticalCL Image upgrade from version @old_cl_version to version @new_cl_version failed for hostname @hostnameCL Image upgrade from version 4.1.0 to version 4.2.1 failed for hostname server01
lcmRestoration of a Cumulus Linux configuration started for a switch or hostInfoCL configuration restore started for hostname @hostnameCL configuration restore started for hostname leaf01
lcmRestoration of a Cumulus Linux configuration completed successfully for a switch or hostInfoCL configuration restore completed for hostname @hostnameCL configuration restore completed for hostname leaf01
lcmRestoration of a Cumulus Linux configuration failed for a switch or hostCriticalCL configuration restore failed for hostname @hostnameCL configuration restore failed for hostname leaf01
lcmRollback of a Cumulus Linux image has started for a switch or hostCriticalCL Image rollback from version @old_cl_version to version @new_cl_version started for hostname @hostnameCL Image rollback from version 4.2.1 to version 4.1.0 started for hostname leaf01
lcmRollback of a Cumulus Linux image has completed successfully for a switch or hostInfoCL Image rollback from version @old_cl_version to version @new_cl_version completed for hostname @hostnameCL Image rollback from version 4.2.1 to version 4.1.0 completed for hostname leaf01
lcmRollback of a Cumulus Linux image has failed for a switch or hostCriticalCL Image rollback from version @old_cl_version to version @new_cl_version failed for hostname @hostnameCL Image rollback from version 4.2.1 to version 4.1.0 failed for hostname leaf01
lcmInstallation of a Cumulus NetQ image has started for a switch or hostInfoNetQ Image version @netq_version installation started for hostname @hostnameNetQ Image version 3.2.0 installation started for hostname spine02
lcmInstallation of a Cumulus NetQ image has completed successfully for a switch or hostInfoNetQ Image version @netq_version installation completed for hostname @hostnameNetQ Image version 3.2.0 installation completed for hostname spine02
lcmInstallation of a Cumulus NetQ image has failed for a switch or hostCriticalNetQ Image version @netq_version installation failed for hostname @hostnameNetQ Image version 3.2.0 installation failed for hostname spine02
lcmUpgrade of a Cumulus NetQ image has started for a switch or hostInfoNetQ Image upgrade from version @old_netq_version to version @netq_version started for hostname @hostnameNetQ Image upgrade from version 3.1.0 to version 3.2.0 started for hostname spine02
lcmUpgrade of a Cumulus NetQ image has completed successfully for a switch or hostInfoNetQ Image upgrade from version @old_netq_version to version @netq_version completed for hostname @hostnameNetQ Image upgrade from version 3.1.0 to version 3.2.0 completed for hostname spine02
lcmUpgrade of a Cumulus NetQ image has failed for a switch or hostCriticalNetQ Image upgrade from version @old_netq_version to version @netq_version failed for hostname @hostnameNetQ Image upgrade from version 3.1.0 to version 3.2.0 failed for hostname spine02

Cumulus Linux License Events

TypeTriggerSeverityMessage FormatExample
licenseLicense state is missing or invalidCriticalLicense check failed, name @lic_name state @stateLicense check failed, name agent.lic state invalid
licenseLicense state is missing or invalid on a particular deviceCriticalLicense check failed on @hostnameLicense check failed on leaf03
TypeTriggerSeverityMessage FormatExample
linkLink operational state changed from up to downCriticalHostName @hostname changed state from @old_state to @new_state Interface:@ifnameHostName leaf01 changed state from up to down Interface:swp34
linkLink operational state changed from down to upInfoHostName @hostname changed state from @old_state to @new_state Interface:@ifnameHostName leaf04 changed state from down to up Interface:swp11

LLDP Events

TypeTriggerSeverityMessage FormatExample
lldpLocal LLDP host has new neighbor informationInfoLLDP Session with host @hostname and @ifname modified fields @changed_fieldsLLDP Session with host leaf02 swp6 modified fields leaf06 swp21
lldpLocal LLDP host has new peer interface nameInfoLLDP Session with host @hostname and @ifname @old_peer_ifname changed to @new_peer_ifnameLLDP Session with host spine01 and swp5 swp12 changed to port12
lldpLocal LLDP host has new peer hostnameInfoLLDP Session with host @hostname and @ifname @old_peer_hostname changed to @new_peer_hostnameLLDP Session with host leaf03 and swp2 leaf07 changed to exit01

MTU Events

TypeTriggerSeverityMessage FormatExample
mtuVLAN interface link MTU is smaller than that of its parent MTUWarningvlan interface @link mtu @mtu is smaller than parent @parent mtu @parent_mtuvlan interface swp3 mtu 1500 is smaller than parent peerlink-1 mtu 1690
mtuBridge interface MTU is smaller than the member interface with the smallest MTUWarningbridge @link mtu @mtu is smaller than least of member interface mtu @minbridge swp0 mtu 1280 is smaller than least of member interface mtu 1500

NTP Events

TypeTriggerSeverityMessage FormatExample
ntpNTP sync state changed from in sync to not in syncCriticalSync state changed from @old_state to @new_state for @hostnameSync state changed from in sync to not sync for leaf06
ntpNTP sync state changed from not in sync to in syncInfoSync state changed from @old_state to @new_state for @hostnameSync state changed from not sync to in sync for leaf06

OSPF Events

TypeTriggerSeverityMessage FormatExample
ospfOSPF session state on a given interface changed from Full to a down stateCriticalOSPF session @ifname with @peer_address changed from Full to @down_state

OSPF session swp7 with 27.0.0.18 state changed from Full to Fail

OSPF session swp7 with 27.0.0.18 state changed from Full to ExStart

ospfOSPF session state on a given interface changed from a down state to fullInfoOSPF session @ifname with @peer_address changed from @down_state to Full

OSPF session swp7 with 27.0.0.18 state changed from Down to Full

OSPF session swp7 with 27.0.0.18 state changed from Init to Full

OSPF session swp7 with 27.0.0.18 state changed from Fail to Full

Package Information Events

TypeTriggerSeverityMessage FormatExample
packageinfoPackage version on device does not match the version identified in the existing manifestCritical@package_name manifest version mismatchnetq-apps manifest version mismatch

PTM Events

TypeTriggerSeverityMessage FormatExample
ptmPhysical interface cabling does not match configuration specified in topology.dot fileCriticalPTM cable status failedPTM cable status failed
ptmPhysical interface cabling matches configuration specified in topology.dot fileCriticalPTM cable status passedPTM cable status passed

Resource Events

TypeTriggerSeverityMessage FormatExample
resourceA physical resource has been deleted from a deviceCriticalResource Utils deleted for @hostnameResource Utils deleted for spine02
resourceRoot file system access on a device has changed from Read/Write to Read OnlyCritical@hostname root file system access mode set to Read Onlyserver03 root file system access mode set to Read Only
resourceRoot file system access on a device has changed from Read Only to Read/WriteInfo@hostname root file system access mode set to Read/Writeleaf11 root file system access mode set to Read/Write
resourceA physical resource has been added to a deviceInfoResource Utils added for @hostnameResource Utils added for spine04

Running Config Diff Events

TypeTriggerSeverityMessage FormatExample
runningconfigdiffRunning configuration file has been modifiedInfo@commandname config result was modified@commandname config result was modified

Sensor Events

TypeTriggerSeverityMessage FormatExample
sensorA fan or power supply unit sensor has changed stateCriticalSensor @sensor state changed from @old_s_state to @new_s_stateSensor fan state changed from up to down
sensorA temperature sensor has crossed the maximum threshold for that sensorCriticalSensor @sensor max value @new_s_max exceeds threshold @new_s_critSensor temp max value 110 exceeds the threshold 95
sensorA temperature sensor has crossed the minimum threshold for that sensorCriticalSensor @sensor min value @new_s_lcrit fall behind threshold @new_s_minSensor psu min value 10 fell below threshold 25
sensorA temperature, fan, or power supply sensor state changedInfoSensor @sensor state changed from @old_state to @new_state

Sensor temperature state changed from critical to ok

Sensor fan state changed from absent to ok

Sensor psu state changed from bad to ok

sensorA fan or power supply sensor state changedInfoSensor @sensor state changed from @old_s_state to @new_s_state

Sensor fan state changed from down to up

Sensor psu state changed from down to up

Services Events

TypeTriggerSeverityMessage FormatExample
servicesA service status changed from down to upCriticalService @name status changed from @old_status to @new_statusService bgp status changed from down to up
servicesA service status changed from up to downCriticalService @name status changed from @old_status to @new_statusService lldp status changed from up to down
servicesA service changed state from inactive to activeInfoService @name changed state from inactive to active

Service bgp changed state from inactive to active

Service lldp changed state from inactive to active

SSD Utilization Events

TypeTriggerSeverityMessage FormatExample
ssdutil3ME3 disk health has dropped below 10%Critical@info: @detailslow health : 5.0%
ssdutilA dip in 3ME3 disk health of more than 2% has occured within the last 24 hoursCritical@info: @detailssignificant health drop : 3.0%

Threshold-based Events

TypeTriggerSeverityMessage FormatExample
tcaPercentage of CPU utilization exceeded user-defined maximum threshold on a switchCriticalCPU Utilization for host @hostname exceed configured mark @cpu_utilizationCPU Utilization for host leaf11 exceed configured mark 85
tcaPercentage of disk utilization exceeded user-defined maximum threshold on a switchCriticalDisk Utilization for host @hostname exceed configured mark @disk_utilizationDisk Utilization for host leaf11 exceed configured mark 90
tcaPercentage of memory utilization exceeded user-defined maximum threshold on a switchCriticalMemory Utilization for host @hostname exceed configured mark @mem_utilizationMemory Utilization for host leaf11 exceed configured mark 95
tcaNumber of transmit bytes exceeded user-defined maximum threshold on a switch interfaceCriticalTX bytes upper threshold breached for host @hostname ifname:@ifname value: @tx_bytesTX bytes upper threshold breached for host spine02 ifname:swp4 value: 20000
tcaNumber of broadcast transmit bytes exceeded user-defined maximum threshold on a switch interfaceCriticalTX broadcast upper threshold breached for host @hostname ifname:@ifname value: @rx_broadcastTX broadcast upper threshold breached for host leaf04 ifname:swp45 value: 40200
tcaNumber of multicast transmit bytes exceeded user-defined maximum threshold on a switch interfaceCriticalTX multicast upper threshold breached for host @hostname ifname:@ifname value: @rx_broadcastTX multicast upper threshold breached for host leaf04 ifname:swp45 value: 30000
tcaNumber of receive bytes exceeded user-defined maximum threshold on a switch interfaceCriticalRX bytes upper threshold breached for host @hostname ifname:@ifname value: @tx_bytesRX bytes upper threshold breached for host spine02 ifname:swp4 value: 20000
tcaNumber of broadcast receive bytes exceeded user-defined maximum threshold on a switch interfaceCriticalRX broadcast upper threshold breached for host @hostname ifname:@ifname value: @rx_broadcastRX broadcast upper threshold breached for host leaf04 ifname:swp45 value: 40200
tcaNumber of multicast receive bytes exceeded user-defined maximum threshold on a switch interfaceCriticalRX multicast upper threshold breached for host @hostname ifname:@ifname value: @rx_broadcastRX multicast upper threshold breached for host leaf04 ifname:swp45 value: 30000
tcaFan speed exceeded user-defined maximum threshold on a switchCriticalSensor for @hostname exceeded threshold fan speed @s_input for sensor @s_nameSensor for spine03 exceeded threshold fan speed 700 for sensor fan2
tcaPower supply output exceeded user-defined maximum threshold on a switchCriticalSensor for @hostname exceeded threshold power @s_input watts for sensor @s_nameSensor for leaf14 exceeded threshold power 120 watts for sensor psu1
tcaTemperature (° C) exceeded user-defined maximum threshold on a switchCriticalSensor for @hostname exceeded threshold temperature @s_input for sensor @s_nameSensor for leaf14 exceeded threshold temperature 90 for sensor temp1
tcaPower supply voltage exceeded user-defined maximum threshold on a switchCriticalSensor for @hostname exceeded threshold voltage @s_input volts for sensor @s_nameSensor for leaf14 exceeded threshold voltage 12 volts for sensor psu2

Version Events

TypeTriggerSeverityMessage FormatExample
versionAn unknown version of the operating system was detectedCriticalunexpected os version @my_verunexpected os version cl3.2
versionDesired version of the operating system is not availableCriticalos version @veros version cl3.7.9
versionAn unknown version of a software package was detectedCriticalexpected release version @verexpected release version cl3.6.2
versionDesired version of a software package is not availableCriticaldifferent from version @verdifferent from version cl4.0

VXLAN Events

TypeTriggerSeverityMessage FormatExample
vxlanReplication list is contains an inconsistent set of nodes<>Critical<>VNI @vni replication list inconsistent with @conflicts diff:@diff<>VNI 14 replication list inconsistent with ["leaf03","leaf04"] diff:+:["leaf03","leaf04"] -:["leaf07","leaf08"]