Monitor Network Connectivity

It is helpful to verify that communications are freely flowing between the various devices in your network. You can verify the connectivity between two devices in both an adhoc fashion and by defining connectivity checks to occur on a scheduled basis. There are three card workflows which enable you to view connectivity, the Trace Request, On-demand Trace Results, and Scheduled Trace Results.

Create a Trace Request

Two types of connectivity checks can be run–an immediate (on-demand) trace and a scheduled trace. The Trace Request card workflow is used to configure and run both of these trace types.

Trace Request Card Workflow Summary

The small Trace Request card displays:

Item

Description

Indicates a trace request

Select Trace list

Select a scheduled trace request from the list

Go

Click to start the trace now

The medium Trace Request card displays:

Item

Description

Indicates a trace request

Title

New Trace Request

New Trace Request

Create a new layer 3 trace request. Use the large Trace Request card to create a new layer 2 or 3 request.

Source

(Required) Hostname or IP address of device where to begin the trace

Destination

(Required) IP address of device where to end the trace

Run Now

Start the trace now

The large Trace Request card displays:

Item

Description

Indicates a trace request

Title

New Trace Request

Trace selection

Leave New Trace Request selected to create a new request, or choose a scheduled request from the list.

Source

(Required) Hostname or IP address of device where to begin the trace.

Destination

(Required) Ending point for the trace. For layer 2 traces, value must be a MAC address. For layer 3 traces, value must be an IP address.

VRF

Optional for layer 3 traces. Virtual Route Forwarding interface to be used as part of the trace path.

VLAN ID

Required for layer 2 traces. Virtual LAN to be used as part of the trace path.

Schedule

Sets the frequency with which to run a new trace (Run every) and when to start the trace for the first time (Starting).

Run Now

Start the trace now

Update

Update is available when a scheduled trace request is selected from the dropdown list and you make a change to its configuration. Clicking Update saves the changes to the existing scheduled trace.

Save As New

Save As New is available in two instances:

  • When you enter a source, destination, and schedule for a new trace. Clicking Save As New in this instance saves the new scheduled trace.

  • When changes are made to a selected scheduled trace request. Clicking Save As New in this instance saves the modified scheduled trace without changing the original trace on which it was based.

The full screen Trace Request card displays:

Item

Description

Title

Trace Request

Closes full screen card and returns to workbench

Time period

Range of time in which the displayed data was collected; applies to all card sizes; select an alternate time period by clicking

Results

Number of results found for the selected tab

Schedule Preview tab

Displays all scheduled trace requests for the given user. By default, the listing is sorted by Start Time, with the most recently started traces listed at the top. The tab provides the following additional data about each event:

  • Action: Indicates latest action taken on the trace job. Values include Add, Deleted, Update.

  • Frequency: How often the trace is scheduled to run

  • Active: Indicates if trace is actively running (true), or stopped from running (false)

  • ID: Internal system identifier for the trace job

  • Trace Name: User-defined name for a trace

  • Trace Params: Indicates source and destination, optional VLAN or VRF specified, and whether to alert on failure

Export

Enables export of all or selected items in a CSV or JSON formatted file

Enables manipulation of table display; choose columns to display and reorder columns

Create a Layer 3 On-demand Trace Request

It is helpful to verify the connectivity between two devices when you suspect an issue is preventing proper communication between them. It you cannot find a path through a layer 3 path, you might also try checking connectivity through a layer 2 path.

To create a layer 3 trace request:

  1. Open the medium Trace Request card.

  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.

  3. In the Destination field, enter the IP address of the device where you want to end the trace.

    In this example, we are starting our trace at server02 and ending it at 10.1.3.103.

    If you mistype an address, you must double-click it, or backspace over the error, and retype the address. You cannot select the address by dragging over it as this action attempts to move the card to another location.

  4. Click Run Now. A corresponding Trace Results card is opened on your workbench. Refer to View Layer 3 Trace Results for details.

Create a Layer 3 Trace Through a Given VRF

If you want to guide a trace through a particular VRF interface, you can do so using the large New Trace Request card.

To create the trace request:

  1. Open the large Trace Request card.
  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.
  3. In the Destination field, enter the IP address of the device where you want to end the trace.
  4. In the VRF field, enter the identifier for the VRF interface you want to use.

    In this example, we are starting our trace at leaf01 and ending it at 10.1.3.103 using VRF vrf1.

  5. Click Run Now. A corresponding Trace Results card is opened on your workbench. Refer to View Layer 3 Trace Results for details.

Create a Layer 2 Trace

It is helpful to verify the connectivity between two devices when you suspect an issue is preventing proper communication between them. It you cannot find a path through a layer 2 path, you might also try checking connectivity through a layer 3 path.

To create a layer 2 trace request:

  1. Open the large Trace Request card.

  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.

  3. In the Destination field, enter the MAC address for where you want to end the trace.

  4. In the VLAN ID field, enter the identifier for the VLAN you want to use.

    In this example, we are starting our trace at leaf01 and ending it at 00:03:00:33:33:01 using VLAN 13.

  5. Click Run Now. A corresponding Trace Results card is opened on your workbench. Refer to View Layer 2 Trace Results for details.

Create a Trace to Run on a Regular Basis (Scheduled Trace)

There may be paths through your network that you consider critical to your everyday or particularly important operations. In that case, it might be useful to create one or more traces to periodically confirm that at least one path is available between the relevant two devices. Scheduling a trace request can be performed from the large Trace Request card.

To schedule a trace:

  1. Open the large Trace Request card.
  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.
  3. In the Destination field, enter the MAC address (layer 2) or IP address (layer 3) of the device where you want to end the trace.
  4. Optionally, enter a VLAN ID (layer 2) or VRF interface (layer 3).

  5. Select a timeframe under Schedule to specify how often you want to run the trace.

  6. Accept the default starting time, or click in the Starting field to specify the day you want the trace to run for the first time.

  7. Click Next.

  8. Click the time you want the trace to run for the first time.

  9. Click OK.

  10. Verify your entries are correct, then click Save As New.

  11. Provide a name for the trace. Note: This name must be unique for a given user.

  12. Click Save. You can now run this trace on demand by selecting it from the dropdown list, or wait for it to run on its defined schedule.

Run a Scheduled Trace on Demand

You may find that, although you have a schedule for a particular trace, you want to have visibility into the connectivity data now. You can run a scheduled trace on demand from the small, medium and large Trace Request cards.

To run a scheduled trace now:

  1. Open the small or medium or large Trace Request card.

  2. Select the scheduled trace from the Select Trace or New Trace Request list. Note: In the medium and large cards, the trace details are filled in on selection of the scheduled trace.

  3. Click Go or Run Now. A corresponding Trace Results card is opened on your workbench.

View On-demand Trace Results

Once you have started an on-demand trace, the results are displayed in the medium Trace Results card by default. You may view the results in more or less detail by switching to the large or small Trace Results card, respectively.

On-demand Trace Results Card Workflow Summary

The small On-demand Trace Results card displays:

Item

Description

Indicates an on-demand trace result

 

Source and destination of the trace, identified by their address or hostname. Source is listed on top with arrow pointing to destination.

,

Indicates success or failure of the trace request. A successful result implies all paths were successful without any warnings or failures. A failure result implies there was at least one path with warnings or errors.

The medium On-demand Trace Results card displays:

Item

Description

Indicates an on-demand trace result

Title

On-demand Trace Result

 

Source and destination of the trace, identified by their address or hostname. Source is listed on top with arrow pointing to destination.

,

Indicates success or failure of the trace request. A successful result implies all paths were successful without any warnings or failures. A failure result implies there was at least one path with warnings or errors.

Total Paths Found

Number of paths found between the two devices

MTU Overall

Average size of the maximum transmission unit for all paths

Minimum Hops

Smallest number of hops along a path between the devices

Maximum Hops

Largest number of hops along a path between the devices

The large On-demand Trace Results card contains two tabs.

The On-demand Trace Result tab displays:

Item

Description

Indicates an on-demand trace result

Title

On-demand Trace Result

,

Indicates success or failure of the trace request. A successful result implies all paths were successful without any warnings or failures. A failure result implies there was at least one path with warnings or errors.

Source and destination of the trace, identified by their address or hostname. Source is listed on top with arrow pointing to destination.

Distribution by Hops chart

Displays the distributions of various hop counts for the available paths

Distribution by MTU chart

Displays the distribution of MTUs used on the interfaces used in the available paths

Table

Provides detailed path information, sorted by the route identifier, including:

  • Route ID: Identifier of each path

  • MTU: Average speed of the interfaces used

  • Hops: Number of hops to get from the source to the destination device

  • Warnings: Number of warnings encountered during the trace on a given path

  • Errors: Number of errors encountered during the trace on a given path

Total Paths Found

Number of paths found between the two devices

MTU Overall

Average size of the maximum transmission unit for all paths

Minimum Hops

Smallest number of hops along a path between the devices

The On-demand Trace Settings tab displays:

Item

Description

Indicates an on-demand trace setting

Title

On-demand Trace Settings

Source

Starting point for the trace

Destination

Ending point for the trace

Schedule

Does not apply to on-demand traces

VRF

Associated virtual route forwarding interface, when used with layer 3 traces

VLAN

Associated virtual local area network, when used with layer 2 traces

Job ID

Identifier of the job; used internally

Re-run Trace

Clicking this button runs the trace again

The full screen On-demand Trace Results card displays:

Item

Description

Title

On-demand Trace Results

Closes full screen card and returns to workbench

Time period

Range of time in which the displayed data was collected; applies to all card sizes; select an alternate time period by clicking

Results

Number of results found for the selected tab

Trace Results tab

Provides detailed path information, sorted by the Resolution Time (date and time results completed), including:

  • SCR.IP: Source IP address

  • DST.IP: Destination IP address

  • Max Hop Count: Largest number of hops along a path between the devices

  • Min Hop Count: Smallest number of hops along a path between the devices

  • Total Paths: Number of paths found between the two devices

  • PMTU: Average size of the maximum transmission unit for all interfaces along the paths

  • Errors: Message provided for analysis when a trace fails

Export

Enables export of all or selected items in a CSV or JSON formatted file

Enables manipulation of table display; choose columns to display and reorder columns

View Layer 2 Trace Results

When you start the trace, the corresponding results card is opened on your workbench. While it is working on the trace, a notice is shown on the card indicating it is running.

Once the job is completed, the results are displayed.

In this example, we see that the trace was successful. Four paths were found between the devices, each with four hops and with an overall MTU of 1500. If there was a difference between the minimum and maximum number of hops or other failures, viewing the results on the large card would provide additional information.

In our example, we can verify that every path option had four hops since the distribution chart only shows one hop count and the table indicates each path had a value of four hops. Similarly, you can view the MTU data. If there had been any warnings, the count would have been visible in the table.

View Layer 3 Trace Results

When you start the trace, the corresponding results card is opened on your workbench. While it is working on the trace, a notice is shown on the card indicating it is running.

Once results are obtained, it displays them. Using our example from earlier, the following results are shown:

In this example, we see that the trace was successful. Six paths were found between the devices, each with five hops and with an overall MTU of 1500. If there was a difference between the minimum and maximum number of hops or other failures, viewing the results on the large card would provide additional information.

In our example, we can verify that every path option had five hops since the distribution chart only shows one hop count and the table indicates each path had a value of five hops. Similarly, you can view the MTU data. If there had been any warnings, the count would have been visible in the table.

View Scheduled Trace Results

You can view the results of scheduled traces at any time. Results are displayed on the Scheduled Trace Results cards.

Scheduled Trace Results Card Workflow Summary

The small Scheduled Trace Results card displays:

Item

Description

Indicates a scheduled trace result

 

Source and destination of the trace, identified by their address or hostname. Source is listed on left with arrow pointing to destination.

Results

Summary of trace results: a successful result implies all paths were successful without any warnings or failures; a failure result implies there was at least one path with warnings or errors.

  • Number of trace runs completed in the designated time period

  • Number of runs with warnings

  • Number of runs with errors

The medium Scheduled Trace Results card displays:

Item

Description

Time period

Range of time in which the displayed data was collected; applies to all card sizes

Indicates a scheduled trace result

Title

Scheduled Trace Result

Summary

Name of scheduled validation and summary of trace results: a successful result implies all paths were successful without any warnings or failures; a failure result implies there was at least one path with warnings or errors.

  • Number of trace runs completed in the designated time period

  • Number of runs with warnings

  • Number of runs with errors

Charts

Heat map: A time segmented view of the results. For each time segment, the color represents the percentage of warning and failed results. Refer to Granularity of Data Shown Based on Time Period for details on how to interpret the results.

Unique Bad Nodes: Distribution of unique nodes that generated the indicated warnings and/or failures

The large Scheduled Trace Results card contains two tabs:

The Results tab displays:

Item

Description

Time period

Range of time in which the displayed data was collected; applies to all card sizes

Indicates a scheduled trace result

Title

Scheduled Trace Result

Summary

Name of scheduled validation and summary of trace results: a successful result implies all paths were successful without any warnings or failures; a failure result implies there was at least one path with warnings or errors.

  • Number of trace runs completed in the designated time period

  • Number of runs with warnings

  • Number of runs with errors

Charts

Heat map: A time segmented view of the results. For each time segment, the color represents the percentage of warning and failed results. Refer to Granularity of Data Shown Based on Time Period for details on how to interpret the results.

Small charts: Display counts for each item during the same time period, for the purpose of correlating with the warnings and errors shown in the heat map.

Table/Filter options

When the Failures filter option is selected, the table displays the failure messages received for each run.

When the Paths filter option is selected, the table displays all of the paths tried during each run.

When the Warning filter option is selected, the table displays the warning messages received for each run.

The Configuration tab displays:

Item

Description

Time period

Range of time in which the displayed data was collected; applies to all card sizes

Indicates a scheduled trace configuration

Title

Scheduled Trace Configuration (Scheduled Trace Result)

Source

Address or hostname of the device where the trace was started

Destination

Address of the device where the trace was stopped

Schedule

The frequency and starting date and time to run the trace

VRF

Virtual Route Forwarding interface, when defined

VLAN

Virtual LAN identifier, when defined

Name

User-defined name of the scheduled trace

Run Now

Start the trace now

Edit

Modify the trace. Opens Trace Request card with this information pre-populated.

The full screen Scheduled Trace Results card displays:

Item

Description

Title

Scheduled Trace Results

Closes full screen card and returns to workbench

Time period

Range of time in which the displayed data was collected; applies to all card sizes; select an alternate time period by clicking

Results

Number of results found for the selected tab

Scheduled Trace Results tab

Displays the basic information about the trace, including:

  • Resolution Time: Time that trace was run

  • SRC.IP: IP address of the source device

  • DST.IP: Address of the destination device

  • Max Hop Count: Maximum number of hops across all paths between the devices

  • Min Hop Count: Minimum number of hops across all paths between the devices

  • Total Paths: Number of available paths found between the devices

  • PMTU: Average of the maximum transmission units for all paths

  • Errors: Message provided for analysis if trace fails

Click on a result to open a detailed view of the results.

Export

Enables export of all or selected items in a CSV or JSON formatted file

Enables manipulation of table display; choose columns to display and reorder columns

Granularity of Data Shown Based on Time Period

On the medium and large Trace Result cards, the status of the runs is represented in heat maps stacked vertically; one for runs with warnings and one for runs with failures. Depending on the time period of data on the card, the number of smaller time blocks used to indicate the status varies. A vertical stack of time blocks, one from each map, includes the results from all checks during that time. The results are shown by how saturated the color is for each block. If all traces run during that time period pass, then both blocks are 100% gray. If there are only failures, the associated lower blocks are 100% saturated white and the warning blocks are 100% saturated gray. As warnings and failures increase, the blocks increase their white saturation. As warnings or failures decrease, the blocks increase their gray saturation. An example heat map for a time period of 24 hours is shown here with the most common time periods in the table showing the resulting time blocks.

Time Period Number of Runs Number Time Blocks Amount of Time in Each Block
6 hours 18 6 1 hour
12 hours 36 12 1 hour
24 hours 72 24 1 hour
1 week 504 7 1 day
1 month 2,086 30 1 day
1 quarter 7,000 13 1 week

View Detailed Scheduled Trace Results

Once a scheduled trace request has completed, the results are available in the corresponding Trace Result card.

To view the results:

  1. Open the full screen Trace Request card to view all scheduled traces that have been run.

  2. Select the scheduled trace you want to view results for by clicking in the first column of the result and clicking the check box.

  3. On the Edit Menu that appears at the bottom of the window, click (Open Cards). This opens the medium Scheduled Trace Results card(s) for the selected items.

  4. Note the distribution of results. Are there many failures? Are they concentrated together in time? Has the trace begun passing again?

  5. Hover over the heat maps to view the status numbers and what percentage of the total results that represents for a given region.

  6. Switch to the large Scheduled Trace Result card.

  7. If there are a large number of warnings or failures, view the associated messages by selecting Failures or Warning in the filter above the table. This might help narrow the failures down to a particular device or small set of devices that you can investigate further.

  8. Look for a consistent number of paths, MTU, hops in the small charts under the heat map. Changes over time here might correlate with the messages and give you a clue to any specific issues. Note if the number of bad nodes changes over time. Devices that become unreachable are often the cause of trace failures.

  9. View the available paths for each run, by selecting Paths in the filter above the table.

  10. You can view the configuration of the request that produced the results shown on this card workflow, by hovering over the card and clicking . If you want to change the configuration, click Edit to open the large Trace Request card, pre-populated with the current configuration. Follow the instructions in Create a Scheduled Trace Request to make your changes in the same way you created a new scheduled trace.

  11. To view a summary of all scheduled trace results, switch to the full screen card.

  12. Look for changes and patterns in the results for additional clues to isolate root causes of trace failures. Select and view related traces using the Edit menu.

  13. View the details of any specific trace result by clicking on the trace. A new window opens similar to the following:

    Scroll to the right to view the information for a given hop. Scroll down to view additional paths.
    This display shows each of the hosts and detailed steps the trace takes to validate a given path between two devices. Using Path 1 as an example, each path can be interpreted as follows:
    Hop 1 is from the source device, server02 in this case. It exits this device at switch port bond0 with an MTU of 9000 and over the default VRF too get to leaf02. The trace goes in to swp2 with an MTU of 9216 over the vrf1 interface. It exits leaf02 through switch port 52 and so on.

  14. Export this data using the Export button or click to return to the results list to view another trace in detail.