This documentation is for an older version of the software. If you are using the current version of Cumulus Linux, this content may not be up to date. The current version of the documentation is available here. If you are redirected to the main page of the user guide, then this page may have been renamed; please search for it there.

Integrate NetQ with Notification Applications

After you have installed the NetQ applications package and the NetQ Agents, you may want to configure some of the additional capabilities that NetQ offers. This topic describes how to integrate NetQ with an event notification application.

Integrate NetQ with an Event Notification Application

To take advantage of the numerous event messages generated and processed by NetQ, you must integrate with third-party event notification applications. You can integrate NetQ with the PagerDuty and Slack tools. You may integrate with one or both of these applications.

Each network protocol and service in the NetQ Platform receives the raw data stream from the NetQ Agents, processes the data and delivers events to the Notification function. Notification then stores, filters and sends messages to any configured notification applications. Filters are based on rules you create. You must have at least one rule per filter.

You may choose to implement a proxy server (that sits between the NetQ Platform and the integration channels) that receives, processes and distributes the notifications rather than having them sent directly to the integration channel. If you use such a proxy, you must configure NetQ with the proxy information.

In either case, notifications are generated for the following types of events:

  • Network Protocols
    • BGP status and session state
    • CLAG (MLAG) status and session state
    • EVPN status and session state
    • LLDP status
    • LNV status and session state *
    • OSFP status and session state
    • VLAN status and session state *
    • VXLAN status and session state *
  • Interfaces
    • Link status
    • Ports and cables status
  • Services status
    • NetQ Agent status
    • PTM
    • SSH *
    • NTP status *
  • Trace status
  • Sensors
    • Fan status
    • PSU (power supply unit) status
    • Temperature status
  • System
    • Configuration File changes
    • Cumulus Linux License status
    • Cumulus Linux Support status

* This type of event can only be viewed in the CLI with this release.

Event Message Format

Messages have the following structure: <message-type><timestamp><opid><hostname><severity><message>

ElementDescription
message typeCategory of event; bgp, clag, configdiff, evpn, link, lldp, lnv, node, ntp, ospf, port, sensor, services, trace, vlan or vxlan
timestampDate and time event occurred
opidIdentifier of the service or process that generated the event
hostnameHostname of network device where event occurred
severitySeverity level in which the given event is classified; debug, error, info, warning, or critical
messageText description of event

For example:

To set up the integrations, you must configure NetQ with at least one channel. Optionally, you can define rules and filters to refine what messages you want to view and where to send them. You can also configure a proxy server to receive, process, and forward the messages. This is accomplished using the NetQ CLI in the following order:

Notification Commands Overview

The NetQ Command Line Interface (CLI) is used to filter and send notifications to third-party tools based on severity, service, event-type, and device. You can use TAB completion or the help option to assist when needed. The command syntax is:

##Proxy
netq add notification proxy <text-proxy-hostname> [port <text-proxy-port>]
netq show notification proxy
netq del notification proxy
 
##Channels
netq add notification channel slack <text-channel-name> webhook <text-webhook-url> [severity info|severity warning|severity error|severity debug] [tag <text-slack-tag>]
netq add notification channel pagerduty <text-channel-name> integration-key <text-integration-key> [severity info|severity warning|severity error|severity debug]
 
##Rules and Filters
netq add notification rule <text-rule-name> key <text-rule-key> value <text-rule-value>
netq add notification filter <text-filter-name> [severity info|severity warning|severity error|severity debug] [rule <text-rule-name-anchor>] [channel <text-channel-name-anchor>] [before <text-filter-name-anchor>|after <text-filter-name-anchor>]
 
##Management
netq del notification channel <text-channel-name-anchor>
netq del notification filter <text-filter-name-anchor>
netq del notification rule <text-rule-name-anchor>
netq show notification [channel|filter|rule] [json]

The options are described in the following sections where they are used.

Configure Basic NetQ Event Notification

The simplest configuration you can create is one that sends all events generated by all interfaces to a single notification application. This is described here. For more granular configurations and examples, refer to Configure Advanced NetQ Event Notifications.

A notification configuration must contain one channel, one rule, and one filter. Creation of the configuration follows this same path:

  1. Add a channel (slack, pagerduty, syslog)
  2. Add a rule that accepts all interface events
  3. Add a filter that associates this rule with the newly created channel

Create Your Channel

For Pager Duty–

Configure a channel using the integration key for your Pager Duty setup. Verify the configuration.

```
cumulus@switch:~$ netq add notification channel pagerduty pd-netq-events integration-key c6d666e210a8425298ef7abde0d1998
Successfully added/updated channel pd-netq-events

cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity         Channel Info
--------------- ---------------- ---------------- ------------------------
pd-netq-events  pagerduty        info             integration-key: c6d666e
                                                210a8425298ef7abde0d1998      
```

For Slack–

Create an *incoming webhook* as described in the documentation for your version of Slack. Verify the configuration.

    ```
    cumulus@switch:~$ netq add notification channel slack slk-netq-events webhook https://hooks.slack.com/services/text/moretext/evenmoretext
    Successfully added/updated channel slk-netq-events
    
    cumulus@switch:~$ netq show notification channel
    Matching config_notify records:
    Name            Type             Severity Channel Info
    --------------- ---------------- -------- ----------------------
    slk-netq-events slack            info     webhook:https://hooks.s
                                            lack.com/services/text/
                                            moretext/evenmoretext
    ```

For Syslog–

Create the channel using the syslog server hostname (or IP address) and port. Verify the configuration.

    ```
    cumulus@switch:~$ netq add notification channel syslog syslog-netq-events hostname syslog-server port 514
    Successfully added/updated channel syslog-netq-events
    
    cumulus@switch:~$ netq show notification channel
    Matching config_notify records:
    Name            Type             Severity Channel Info
    --------------- ---------------- -------- ----------------------
    syslog-netq-eve syslog            info     host:syslog-server
    nts                                        port: 514
    ```

Create a Rule

Create and verify a rule that accepts all interface events. Verify the configuration.

```
cumulus@switch:~$ netq add notification rule all-ifs key ifname value ALL
Successfully added/updated rule all-ifs

cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
all-interfaces  ifname           ALL
```

Create a Filter

Create a filter to tie the rule to the channel. Verify the configuration.

For PagerDuty–

```
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-ifs channel pd-netq-events
Successfully added/updated filter notify-all-ifs

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs  1          info             pd-netq-events   all-ifs
```

For Slack–

```
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-ifs channel slk-netq-events
Successfully added/updated filter notify-all-ifs

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs  1          info             slk-netq-events   all-ifs
```

For Syslog–

```
cumulus@switch:~$ netq add notification filter notify-all-ifs rule all-ifs channel syslog-netq-events
Successfully added/updated filter notify-all-ifs

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
notify-all-ifs  1          info             syslog-netq-events all-ifs
```

NetQ is now configured to send all interface events to your selected channel.

Configure Advanced NetQ Event Notifications

If you want to create more granular notifications based on such items as selected devices, characteristics of devices, or protocols, or you want to use a proxy server, you need more than the basic notification configuration. Details for creating these more complex notification configurations are included here.

Configure a Proxy Server

To send notification messages through a proxy server instead of directly to a notification channel, you configure NetQ with the hostname and optionally a port of a proxy server. If no port is specified, NetQ defaults to port 80. Only one proxy server is currently supported. To simplify deployment, configure your proxy server before configuring channels, rules, or filters.To configure the proxy server:

cumulus@switch:~$ netq add notification proxy <text-proxy-hostname> [port <text-proxy-port]
cumulus@switch:~$ netq add notification proxy proxy4
Successfully configured notifier proxy proxy4:80

You can view the proxy server settings by running the netq show notification proxy command.

cumulus@switch:~$ netq show notification proxy
Matching config_notify records:
Proxy URL          Slack Enabled              PagerDuty Enabled
------------------ -------------------------- ----------------------------------
proxy4:80          yes                        yes

You can remove the proxy server by running the netq del notification proxy command. This changes the NetQ behavior to send events directly to the notification channels.

cumulus@switch:~$ netq del notification proxy
Successfully overwrote notifier proxy to null

Create Channels

Create one or more PagerDuty, Slack, or syslog channels to present the notifications.

Configure a PagerDuty Channel

NetQ sends notifications to PagerDuty as PagerDuty events.

For example:

To configure the NetQ notifier to send notifications to PagerDuty:

  1. Configure the following options using the netq add notification channel command:

    OptionDescription
    CHANNEL_TYPE <text-channel-name>The third-party notification channel and name; use pagerduty in this case.
    integration-key <text-integration-key>The integration key is also called the service_key or routing_key. The default is an empty string ("").
    severity(Optional) The log level to set, which can be one of info, warning, error, critical or debug. The severity defaults to info.
    cumulus@switch:~$ netq add notification channel pagerduty pd-netq-events integration-key c6d666e210a8425298ef7abde0d1998
    Successfully added/updated channel pd-netq-events
    
  2. Verify that the channel is configured properly.

    cumulus@switch:~$ netq show notification channel
    Matching config_notify records:
    Name            Type             Severity         Channel Info
    --------------- ---------------- ---------------- ------------------------
    pd-netq-events  pagerduty        info             integration-key: c6d666e
                                                      210a8425298ef7abde0d1998      
    

Configure a Slack Channel

NetQ Notifier sends notifications to Slack as incoming webhooks for a Slack channel you configure. For example:

To configure NetQ to send notifications to Slack:

  1. If needed, create one or more Slack channels on which to receive the notifications.

    1. Click + next to Channels.
    2. Enter a name for the channel, and click Create Channel.
    3. Navigate to the new channel.
    4. Click + Add an app link below the channel name to open the application directory.
    5. In the search box, start typing incoming and select ** Incoming WebHooks when it appears.
    6. Click Add Configuration and enter the name of the channel you created (where you want to post notifications).
    7. Click Add Incoming WebHooks integration.
    8. Save WebHook URL in a text file for use in next step.
  2. Configure the following options in the netq config add notification channel command:

    Option

    Description

    CHANNEL_TYPE <text-channel-name>

    The third-party notification channel name; use slack in this case.

    WEBHOOK

    Copy the WebHook URL from the text file OR in the desired channel, locate the initial message indicating the addition of the webhook, click incoming-webhook link, click Settings.

    Example URL: https://hooks.slack.com/services/text/moretext/evenmoretext

    severity

    The log level to set, which can be one of error, warning, info, or debug. The severity defaults to info.

    tag

    Optional tag appended to the Slack notification to highlight particular channels or people. The tag value must be preceded by the @ sign. For example, @netq-info.

    cumulus@switch:~$ netq add notification channel slack slk-netq-events webhook https://hooks.slack.com/services/text/moretext/evenmoretext
    Successfully added/updated channel netq-events
    
  3. Verify the channel is configured correctly.
    From the CLI:

    cumulus@switch:~$ netq show notification channel
    Matching config_notify records:
    Name            Type             Severity Channel Info
    --------------- ---------------- -------- ----------------------
    slk-netq-events slack            info     webhook:https://hooks.s
                                              lack.com/services/text/
                                              moretext/evenmoretext
    

    From the Slack Channel:

Create Rules

Each rule is comprised of a single key-value pair. The key-value pair indicates what messages to include or drop from event information sent to a notification channel. You can create more than one rule for a single filter. Creating multiple rules for a given filter can provide a very defined filter. For example, you can specify rules around hostnames or interface names, enabling you to filter messages specific to those hosts or interfaces. You should have already defined the PagerDuty or Slack channels (as described earlier).

There is a fixed set of valid rule keys. Values are entered as regular expressions and vary according to your deployment.

Service

Rule Key

Description

Example Rule Values

BGP

message_type

Network protocol or service identifier

bgp

hostname

User-defined, text-based name for a switch or host

server02, leaf11, exit01, spine-4

peer

User-defined, text-based name for a peer switch or host

server4, leaf-3, exit02, spine06

desc

Text description

vrf

Name of VRF interface

mgmt, default

old_state

Previous state of the BGP service

Established, Failed

new_state

Current state of the BGP service

Established, Failed

old_last_reset_time

Previous time that BGP service was reset

Apr3, 2019, 4:17 pm

new_last_reset_time

Most recent time that BGP service was reset

Apr8, 2019, 11:38 am

MLAG (CLAG)

message_type

Network protocol or service identifier

clag

hostname

User-defined, text-based name for a switch or host

server02, leaf-9, exit01, spine04

old_conflicted_bonds

Previous pair of interfaces in a conflicted bond

swp7 swp8, swp3 swp4

new_conflicted_bonds

Current pair of interfaces in a conflicted bond

swp11 swp12, swp23 swp24

old_state_protodownbond

Previous state of the bond

protodown, up

new_state_protodownbond

Current state of the bond

protodown, up

ConfigDiff

message_type

Network protocol or service identifier

configdiff

hostname

User-defined, text-based name for a switch or host

server02, leaf11, exit01, spine-4

vni

Virtual Network Instance identifier

12, 23

old_state

Previous state of the configuration file

created, modified

new_state

Current state of the configuration file

created, modified

EVPN

message_type

Network protocol or service identifier

evpn

hostname

User-defined, text-based name for a switch or host

server02, leaf-9, exit01, spine04

vni

Virtual Network Instance identifier

12, 23

old_in_kernel_state

Previous VNI state, in kernel or not

true, false

new_in_kernel_state

Current VNI state, in kernel or not

true, false

old_adv_all_vni_state

Previous VNI advertising state, advertising all or not

true, false

new_adv_all_vni_state

Current VNI advertising state, advertising all or not

true, false

Link

message_type

Network protocol or service identifier

link

hostname

User-defined, text-based name for a switch or host

server02, leaf-6, exit01, spine7

ifname

Software interface name

eth0, swp53

LLDP

message_type

Network protocol or service identifier

lldp

hostname

User-defined, text-based name for a switch or host

server02, leaf41, exit01, spine-5, tor-36

ifname

Software interface name

eth1, swp12

old_peer_ifname

Previous software interface name

eth1, swp12, swp27

new_peer_ifname

Curent software interface name

eth1, swp12, swp27

old_peer_hostname

Previous user-defined, text-based name for a peer switch or host

server02, leaf41, exit01, spine-5, tor-36

new_peer_hostname

Current user-defined, text-based name for a peer switch or host

server02, leaf41, exit01, spine-5, tor-36

Node

message_type

Network protocol or service identifier

node

hostname

User-defined, text-based name for a switch or host

server02, leaf41, exit01, spine-5, tor-36

ntp_state

Current state of NTP service

in sync, not sync

db_state

Current state of DB

Add, Update, Del, Dead

NTP

message_type

Network protocol or service identifier

ntp

hostname

User-defined, text-based name for a switch or host

server02, leaf-9, exit01, spine04

old_state

Previous state of service

in sync, not sync

new_state

Current state of service

in sync, not sync

Port

message_type

Network protocol or service identifier

port

hostname

User-defined, text-based name for a switch or host

server02, leaf13, exit01, spine-8, tor-36

ifname

Interface name

eth0, swp14

old_speed

Previous speed rating of port

10 G, 25 G, 40 G, unknown

old_transreceiver

Previous transceiver

40G Base-CR4, 25G Base-CR

old_vendor_name

Previous vendor name of installed port module

Amphenol, OEM, Mellanox, Fiberstore, Finisar

old_serial_number

Previous serial number of installed port module

MT1507VS05177, AVE1823402U, PTN1VH2

old_supported_fec

Previous forward error correction (FEC) support status

none, Base R, RS

old_advertised_fec

Previous FEC advertising state

true, false, not reported

old_fec

Previous FEC capability

none

old_autoneg

Previous activation state of auto-negotiation

on, off

new_speed

Current speed rating of port

10 G, 25 G, 40 G

new_transreceiver

Current transceiver

40G Base-CR4, 25G Base-CR

new_vendor_name

Current vendor name of installed port module

Amphenol, OEM, Mellanox, Fiberstore, Finisar

new_part_number

Current part number of installed port module

SFP-H10GB-CU1M, MC3309130-001, 603020003

new_serial_number

Current serial number of installed port module

MT1507VS05177, AVE1823402U, PTN1VH2

new_supported_fec

Current FEC support status

none, Base R, RS

new_advertised_fec

Current FEC advertising state

true, false

new_fec

Current FEC capability

none

new_autoneg

Current activation state of auto-negotiation

on, off

Sensors

sensor

Network protocol or service identifier

Fan: fan1, fan-2
Power Supply Unit: psu1, psu2
Temperature: psu1temp1, temp2

hostname

User-defined, text-based name for a switch or host

server02, leaf-26, exit01, spine2-4

old_state

Previous state of a fan, power supply unit, or thermal sensor

Fan: ok, absent, bad
PSU: ok, absent, bad
Temp: ok, busted, bad, critical

new_state

Current state of a fan, power supply unit, or thermal sensor

Fan: ok, absent, bad
PSU: ok, absent, bad
Temp: ok, busted, bad, critical

old_s_state

Previous state of a fan or power supply unit.

Fan: up, down
PSU: up, down

new_s_state

Current state of a fan or power supply unit.

Fan: up, down
PSU: up, down

new_s_max

Current maximum temperature threshold value

Temp: 110

new_s_crit

Current critical high temperature threshold value

Temp: 85

new_s_lcrit

Current critical low temperature threshold value

Temp: -25

new_s_min

Current minimum temperature threshold value

Temp: -50

Services

message_type

Network protocol or service identifier

services

hostname

User-defined, text-based name for a switch or host

server02, leaf03, exit01, spine-8

name

Name of service

clagd, lldpd, ssh, ntp, netqd, net-agent

old_pid

Previous process or service identifier

12323, 52941

new_pid

Current process or service identifier

12323, 52941

old_status

Previous status of service

up, down

new_status

Current status of service

up, down

Rule names are case sensitive, and no wildcards are permitted. Rule names may contain spaces, but must be enclosed with single quotes in commands. It is easier to use dashes in place of spaces or mixed case for better readability. For example, use bgpSessionChanges or BGP-session-changes or BGPsessions, instead of ‘BGP Session Changes’.

Use Tab completion to view the command options syntax.

Example Rules

Create a BGP Rule Based on Hostname:

cumulus@switch:~$ netq add notification rule bgpHostname key hostname value spine-01
Successfully added/updated rule bgpHostname 

Create a Rule Based on a Configuration File State Change:

cumulus@switch:~$ netq add notification rule sysconf key configdiff value updated
Successfully added/updated rule sysconf

Create an EVPN Rule Based on a VNI:

cumulus@switch:~$ netq add notification rule evpnVni key vni value 42
Successfully added/updated rule evpnVni

Create an Interface Rule Based on FEC Support:

cumulus@switch:~$ netq add notification rule fecSupport key new_supported_fec value supported
Successfully added/updated rule fecSupport

Create a Service Rule Based on a Status Change:

cumulus@switch:~$ netq add notification rule svcStatus key new_status value down
Successfully added/updated rule svcStatus

Create a Sensor Rule Based on a Threshold:

cumulus@switch:~$ netq add notification rule overTemp key new_s_crit value 24
Successfully added/updated rule overTemp

Create an Interface Rule Based on Port:

cumulus@switch:~$ netq add notification rule swp52 key port value swp52
Successfully added/updated rule swp52 

View the Rule Configurations

Use the netq show notification command to view the rules on your platform.

cumulus@switch:~$ netq show notification rule
 
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
fecSupport      new_supported_fe supported
                c
overTemp        new_s_crit       24
svcStatus       new_status       down
swp52           port             swp52
sysconf         configdiff       updated

Create Filters

You can limit or direct event messages using filters. Filters are created based on rules you define; like those in the previous section. Each filter contains one or more rules. When a message matches the rule, it is sent to the indicated destination. Before you can create filters, you need to have already defined the rules and configured PagerDuty and/or Slack channels (as described earlier).

As filters are created, they are added to the bottom of a filter list. By default, filters are processed in the order they appear in this list (from top to bottom) until a match is found. This means that each event message is first evaluated by the first filter listed, and if it matches then it is processed, ignoring all other filters, and the system moves on to the next event message received. If the event does not match the first filter, it is tested against the second filter, and if it matches then it is processed and the system moves on to the next event received. And so forth. Events that do not match any filter are ignored.

You may need to change the order of filters in the list to ensure you capture the events you want and drop the events you do not want. This is possible using the before or after keywords to ensure one rule is processed before or after another.

This diagram shows an example with four defined filters with sample output results.

Filter names may contain spaces, but must be enclosed with single quotes in commands. It is easier to use dashes in place of spaces or mixed case for better readability. For example, use bgpSessionChanges or BGP-session-changes or BGPsessions, instead of ‘BGP Session Changes’. Filter names are also case sensitive.

Example Filters

Create a filter for BGP Events on a Particular Device:

cumulus@switch:~$ netq add notification filter bgpSpine rule bgpHostname channel pd-netq-events
Successfully added/updated filter bgpSpine

Create a Filter for a Given VNI in Your EVPN Overlay:

cumulus@switch:~$ netq add notification filter vni42 severity warning rule evpnVni channel pd-netq-events
Successfully added/updated filter vni42

Create a Filter for when a Configuration File has been Updated:

cumulus@switch:~$ netq add notification filter configChange severity info rule sysconf channel slk-netq-events
Successfully added/updated filter configChange

Create a Filter to Monitor Ports with FEC Support:

cumulus@switch:~$ netq add notification filter newFEC rule fecSupport channel slk-netq-events
Successfully added/updated filter newFEC

Create a Filter to Monitor for Services that Change to a Down State:

cumulus@switch:~$ netq add notification filter svcDown severity error rule svcStatus channel slk-netq-events
Successfully added/updated filter svcDown

Create a Filter to Monitor Overheating Platforms:

cumulus@switch:~$ netq add notification filter critTemp severity error rule overTemp channel pd-netq-events
Successfully added/updated filter critTemp

Create a Filter to Drop Messages from a Given Interface, and match against this filter before any other filters. To create a drop style filter, do not specify a channel. To put the filter first, use the before option.

cumulus@switch:~$ netq add notification filter swp52Drop severity error rule swp52 before bgpSpine
Successfully added/updated filter swp52Drop

View the Filter Configurations

Use the netq show notification command to view the filters on your platform.

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
swp52Drop       1          error            NetqDefaultChann swp52
                                            el
bgpSpine        2          info             pd-netq-events   bgpHostnam
                                                             e
vni42           3          warning          pd-netq-events   evpnVni
configChange    4          info             slk-netq-events  sysconf
newFEC          5          info             slk-netq-events  fecSupport
svcDown         6          critical         slk-netq-events  svcStatus
critTemp        7          critical         pd-netq-events   overTemp

Reorder Filters

When you look at the results of the netq show notification filter command above, you might notice that although you have the drop-based filter first (no point in looking at something you are going to drop anyway, so that is good), but the critical severity events are processed last, per the current definitions. If you wanted to process those before lesser severity events, you can reorder the list using the before and after options.

For example, to put the two critical severity event filters just below the drop filter:

cumulus@switch:~$ netq add notification filter critTemp after swp52Drop
Successfully added/updated filter critTemp
cumulus@switch:~$ netq add notification filter svcDown before bgpSpine
Successfully added/updated filter svcDown

You do not need to reenter all the severity, channel, and rule information for existing rules if you only want to change their processing order.

Run the netq show notification command again to verify the changes:

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
swp52Drop       1          error            NetqDefaultChann swp52
                                            el
critTemp        2          critical         pd-netq-events   overTemp
svcDown         3          critical         slk-netq-events  svcStatus
bgpSpine        4          info             pd-netq-events   bgpHostnam
                                                             e
vni42           5          warning          pd-netq-events   evpnVni
configChange    6          info             slk-netq-events  sysconf
newFEC          7          info             slk-netq-events  fecSupport

Examples of Advanced Notification Configurations

Putting all of these channel, rule, and filter definitions together you create a complete notification configuration. The following are example notification configurations are created using the three-step process outlined above. Refer to Integrate NetQ with an Event Notification Application for details and instructions for creating channels, rules, and filters.

Create a Notification for BGP Events from a Selected Switch

In this example, we created a notification integration with a PagerDuty channel called pd-netq-events. We then created a rule bgpHostname and a filter called 4bgpSpine for any notifications from spine-01. The result is that any info severity event messages from Spine-01 are filtered to the pd-netq-events ** channel.

cumulus@switch:~$ netq add notification channel pagerduty pd-netq-events integration-key 1234567890
Successfully added/updated channel pd-netq-events
cumulus@switch:~$ netq add notification rule bgpHostname key node value spine-01
Successfully added/updated rule bgpHostname
 
cumulus@switch:~$ netq add notification filter bgpSpine rule bgpHostname channel pd-netq-events
Successfully added/updated filter bgpSpine
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity         Channel Info
--------------- ---------------- ---------------- ------------------------
pd-netq-events  pagerduty        info             integration-key: 1234567
                                                  890   

cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
 
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
bgpSpine        1          info             pd-netq-events   bgpHostnam
                                                             e

Create a Notification for Warnings on a Given EVPN VNI

In this example, we created a notification integration with a PagerDuty channel called pd-netq-events. We then created a rule evpnVni and a filter called 3vni42 for any warnings messages from VNI 42 on the EVPN overlay network. The result is that any warning severity event messages from VNI 42 are filtered to the pd-netq-events channel.

cumulus@switch:~$ netq add notification channel pagerduty pd-netq-events integration-key 1234567890
Successfully added/updated channel pd-netq-events
 
cumulus@switch:~$ netq add notification rule evpnVni key vni value 42
Successfully added/updated rule evpnVni
 
cumulus@switch:~$ netq add notification filter vni42 rule evpnVni channel pd-netq-events
Successfully added/updated filter vni42
 
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity         Channel Info
--------------- ---------------- ---------------- ------------------------
pd-netq-events  pagerduty        info             integration-key: 1234567
                                                  890   

cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
 
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
bgpSpine        1          info             pd-netq-events   bgpHostnam
                                                             e
vni42           2          warning          pd-netq-events   evpnVni

Create a Notification for Configuration File Changes

In this example, we created a notification integration with a Slack channel called slk-netq-events. We then created a rule sysconf and a filter called configChange for any configuration file update messages. The result is that any configuration update messages are filtered to the slk-netq-events channel.

cumulus@switch:~$ netq add notification channel slack slk-netq-events webhook https://hooks.slack.com/services/text/moretext/evenmoretext
Successfully added/updated channel slk-netq-events
 
cumulus@switch:~$ netq add notification rule sysconf key configdiff value updated
Successfully added/updated rule sysconf
 
cumulus@switch:~$ netq add notification filter configChange severity info rule sysconf channel slk-netq-events
Successfully added/updated filter configChange
 
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity Channel Info
--------------- ---------------- -------- ----------------------
slk-netq-events slack            info     webhook:https://hooks.s
                                          lack.com/services/text/
                                          moretext/evenmoretext     
 
cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
sysconf         configdiff       updated

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
bgpSpine        1          info             pd-netq-events   bgpHostnam
                                                             e
vni42           2          warning          pd-netq-events   evpnVni
configChange    3          info             slk-netq-events  sysconf

Create a Notification for When a Service Goes Down

In this example, we created a notification integration with a Slack channel called slk-netq-events. We then created a rule svcStatus and a filter called svcDown for any services state messages indicating a service is no longer operational. The result is that any service down messages are filtered to the slk-netq-events channel.

cumulus@switch:~$ netq add notification channel slack slk-netq-events webhook https://hooks.slack.com/services/text/moretext/evenmoretext
Successfully added/updated channel slk-netq-events
 
cumulus@switch:~$ netq add notification rule svcStatus key new_status value down
Successfully added/updated rule svcStatus
 
cumulus@switch:~$ netq add notification filter svcDown severity error rule svcStatus channel slk-netq-events
Successfully added/updated filter svcDown
 
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity Channel Info
--------------- ---------------- -------- ----------------------
slk-netq-events slack            info     webhook:https://hooks.s
                                          lack.com/services/text/
                                          moretext/evenmoretext     
 
cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
svcStatus       new_status       down
sysconf         configdiff       updated

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
bgpSpine        1          info             pd-netq-events   bgpHostnam
                                                             e
vni42           2          warning          pd-netq-events   evpnVni
configChange    3          info             slk-netq-events  sysconf
svcDown         4          critical         slk-netq-events  svcStatus

Create a Filter to Drop Notifications from a Given Interface

In this example, we created a notification integration with a Slack channel called slk-netq-events. We then created a rule swp52 and a filter called swp52Drop that drops all notifications for events from interface swp52.

cumulus@switch:~$ netq add notification channel slack slk-netq-events webhook https://hooks.slack.com/services/text/moretext/evenmoretext
Successfully added/updated channel slk-netq-events
 
cumulus@switch:~$ netq add notification rule swp52 key port value swp52
Successfully added/updated rule swp52
 
cumulus@switch:~$ netq add notification filter swp52Drop severity error rule swp52 before bgpSpine
Successfully added/updated filter swp52Drop
 
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity Channel Info
--------------- ---------------- -------- ----------------------
slk-netq-events slack            info     webhook:https://hooks.s
                                          lack.com/services/text/
                                          moretext/evenmoretext     
 
cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
svcStatus       new_status       down
swp52           port             swp52
sysconf         configdiff       updated

cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
swp52Drop       1          error            NetqDefaultChann swp52
                                            el
bgpSpine        2          info             pd-netq-events   bgpHostnam
                                                             e
vni42           3          warning          pd-netq-events   evpnVni
configChange    4          info             slk-netq-events  sysconf
svcDown         5          critical         slk-netq-events  svcStatus

Create a Notification for a Given Device that has a Tendency to Overheat (using multiple rules)

In this example, we created a notification when switch leaf04 has passed over the high temperature threshold. Two rules were needed to create this notification, one to identify the specific device and one to identify the temperature trigger. We sent the message to the pd-netq-events channel.

cumulus@switch:~$ netq add notification channel pagerduty pd-netq-events integration-key 1234567890
Successfully added/updated channel pd-netq-events
 
cumulus@switch:~$ netq add notification rule switchLeaf04 key hostname value leaf04
Successfully added/updated rule switchLeaf04
cumulus@switch:~$ netq add notification rule overTemp key new_s_crit value 24
Successfully added/updated rule overTemp
 
cumulus@switch:~$ netq add notification filter critTemp rule switchLeaf04 channel pd-netq-events
Successfully added/updated filter critTemp
cumulus@switch:~$ netq add notification filter critTemp severity critical rule overTemp channel pd-netq-events
Successfully added/updated filter critTemp
 
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity         Channel Info
--------------- ---------------- ---------------- ------------------------
pd-netq-events  pagerduty        info             integration-key: 1234567
                                                  890

cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
overTemp        new_s_crit       24
svcStatus       new_status       down
switchLeaf04    hostname         leaf04
swp52           port             swp52
sysconf         configdiff       updated
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
swp52Drop       1          error            NetqDefaultChann swp52
                                            el
bgpSpine        2          info             pd-netq-events   bgpHostnam
                                                             e
vni42           3          warning          pd-netq-events   evpnVni
configChange    4          info             slk-netq-events  sysconf
svcDown         5          critical         slk-netq-events  svcStatus
critTemp        6          critical         pd-netq-events   switchLeaf
                                                             04
                                                             overTemp                                                

View Notification Configurations in JSON Format

You can view configured integrations using the netq show notification commands. To view the channels, filters, and rules, run the three flavors of the command. Include the json option to display JSON-formatted output.

For example:

cumulus@switch:~$ netq show notification channel json
{
    "config_notify":[
        {
            "type":"slack",
            "name":"slk-netq-events",
            "channelInfo":"webhook:https://hooks.slack.com/services/text/moretext/evenmoretext",
            "severity":"info"
        },
        {
            "type":"pagerduty",
            "name":"pd-netq-events",
            "channelInfo":"integration-key: 1234567890",
            "severity":"info"
    }
    ],
    "truncatedResult":false
}
 
cumulus@switch:~$ netq show notification rule json
{
    "config_notify":[
        {
            "ruleKey":"hostname",
            "ruleValue":"spine-01",
            "name":"bgpHostname"
        },
        {
            "ruleKey":"vni",
            "ruleValue":42,
            "name":"evpnVni"
        },
        {
            "ruleKey":"new_supported_fec",
            "ruleValue":"supported",
            "name":"fecSupport"
        },
        {
            "ruleKey":"new_s_crit",
            "ruleValue":24,
            "name":"overTemp"
        },
        {
            "ruleKey":"new_status",
            "ruleValue":"down",
            "name":"svcStatus"
        },
        {
            "ruleKey":"configdiff",
            "ruleValue":"updated",
            "name":"sysconf"
    }
    ],
    "truncatedResult":false
}
 
cumulus@switch:~$ netq show notification filter json
{
    "config_notify":[
        {
            "channels":"pd-netq-events",
            "rules":"overTemp",
            "name":"1critTemp",
            "severity":"critical"
        },
        {
            "channels":"pd-netq-events",
            "rules":"evpnVni",
            "name":"3vni42",
            "severity":"warning"
        },
        {
            "channels":"pd-netq-events",
            "rules":"bgpHostname",
            "name":"4bgpSpine",
            "severity":"info"
        },
        {
            "channels":"slk-netq-events",
            "rules":"sysconf",
            "name":"configChange",
            "severity":"info"
        },
        {
            "channels":"slk-netq-events",
            "rules":"fecSupport",
            "name":"newFEC",
            "severity":"info"
        },
        {
            "channels":"slk-netq-events",
            "rules":"svcStatus",
            "name":"svcDown",
            "severity":"critical"
    }
    ],
    "truncatedResult":false
}

Manage Event Notification Integrations

You might need to modify event notification configurations at some point in the lifecycle of your deployment. Optionally, you might want to configure a proxy.

Remove an Event Notification Channel

You can delete an event notification integration using the netq config del notification command. You can verify it has been removed using the related show command.

For example, to remove a Slack integration and verify it is no longer in the configuration:

cumulus@switch:~$ netq del notification channel slk-netq-events
cumulus@switch:~$ netq show notification channel
Matching config_notify records:
Name            Type             Severity         Channel Info
--------------- ---------------- ---------------- ------------------------
pd-netq-events  pagerduty        info             integration-key: 1234567
                                                  890

Delete an Event Notification Rule

To delete a rule, use the following command, then verify it has been removed:

cumulus@switch:~$ netq del notification rule swp52
cumulus@switch:~$ netq show notification rule
Matching config_notify records:
Name            Rule Key         Rule Value
--------------- ---------------- --------------------
bgpHostname     hostname         spine-01
evpnVni         vni              42
overTemp        new_s_crit       24
svcStatus       new_status       down
switchLeaf04    hostname         leaf04
sysconf         configdiff       updated

Delete an Event Notification Filter

To delete a filter, use the following command, then verify it has been removed:

cumulus@switch:~$ netq del notification filter bgpSpine
cumulus@switch:~$ netq show notification filter
Matching config_notify records:
Name            Order      Severity         Channels         Rules
--------------- ---------- ---------------- ---------------- ----------
swp52Drop       1          error            NetqDefaultChann swp52
                                            el
vni42           2          warning          pd-netq-events   evpnVni
configChange    3          info             slk-netq-events  sysconf
svcDown         4          critical         slk-netq-events  svcStatus
critTemp        5          critical         pd-netq-events   switchLeaf
                                                             04
                                                             overTemp