It is helpful to verify that communications are freely flowing between the various devices in your network. You can verify the connectivity between two devices in both an adhoc fashion and by defining connectivity checks to occur on a scheduled basis. There are three card workflows which enable you to view connectivity, the Trace Request, On-demand Trace Results, and Scheduled Trace Results.

Contents

 This topic describes how to...

Create a Trace Request

Two types of connectivity checks can be run–an immediate trace and a scheduled trace. The Trace Request card workflow is used to configure and run both of these trace types.

Trace Request Card Workflow Summary

The small Trace Request card displays:

ItemDescription
Indicates a trace request
Select Trace listSelect a scheduled trace request from the list
GoClick to start the trace now

The medium Trace Request card displays:

ItemDescription
Indicates a trace request
TitleNew Trace Request
New Trace RequestCreate a new request layer 3 trace request. Use the large Trace Request card to create a new layer 2 or 3 request.
Source

(Required) Hostname or IP address of device where to begin the trace

Destination(Required) IP address of device where to end the trace
Run NowStart the trace now

The large Trace Request card displays:

ItemDescription
Indicates a trace request
TitleNew Trace Request
Trace selectionChoose New Trace Request to create a new request, or select a scheduled request from the list.
Source(Required) Hostname or IP address of device where to begin the trace.
Destination(Required) Ending point for the trace. For layer 2 traces, value must be a MAC address. For layer 3 traces, value must be an IP address.
VRFOptional for layer 3 traces. Virtual Route Forwarding interface to be used as part of the trace path.
VLAN IDRequired for layer 2 traces. Virtual LAN to be used as part of the trace path.
ScheduleSets the frequency with which to run a new trace (Run every) and when to start the trace for the first time (Starting).
Run NowStart the trace now
UpdateUpdate is available when a scheduled trace request is selected from the dropdown list and you make a change to its configuration. Clicking Update saves the changes to the existing scheduled trace.
Save As New

Save As New is available in two instances:

  • When you enter a source, destination, and schedule for a new trace. Clicking Save As New in this instance saves the new scheduled trace.
  • When changes are made to a selected scheduled trace request. Clicking Save As New in this instance saves the modified scheduled trace without changing the original trace on which it was based.

The full screen Trace Request card displays:

ItemDescription
TitleTrace Request
Closes full screen card and returns to workbench
Time periodRange of time in which the displayed data was collected; applies to all card sizes; select an alternate time period by clicking 
ResultsNumber of results found for the selected tab
Schedule Preview tabDisplays all scheduled trace requests for the given user. By default, the listing is sorted by Start Time, with the most recently started traces listed at the top. The tab provides the following additional data about each event:
  • Action: Indicates latest action taken on the trace job. Values include Add, Deleted, Update.
  • Frequency: How often the trace is scheduled to run
  • Active: Indicates if trace is actively running (true), or stopped from running (false)
  • ID: Internal system identifier for the trace job
  • Trace Name: User-defined name for a trace
  • Trace Params: Indicates source and destination, optional VLAN or VRF specified, and whether to alert on failure
ExportEnables export of all or selected items in a CSV or JSON formatted file
Enables manipulation of table display; choose columns to display and reorder columns

Create a Layer 3 On-demand Trace Request

It is helpful to verify the connectivity between two devices when you suspect an issue is preventing proper communication between them. It you cannot find a path through a layer 3 path, you might also try checking connectivity through a layer 2 path.

To create a layer 3 trace request:

  1. Open the medium Trace Request card.
  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.
  3. In the Destination field, enter the IP address of the device where you want to end the trace.

    In this example, we are starting our trace at server02 and ending it at 10.1.3.103.

    If you mistype an address, you must double-click it, or backspace over the error, and retype the address. You cannot select the address by dragging over it as this action attempts to move the card to another location.
  4. Click Run Now. A corresponding Trace Results card is opened on your workbench. Refer to View Layer 3 Trace Results for details.

Create a Layer 3 Trace Through a Given VRF

If you want to guide a trace through a particular VRF interface, you can do so using the large New Trace Request card.

To create the trace request:

  1. Open the large Trace Request card.
  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.
  3. In the Destination field, enter the IP address of the device where you want to end the trace.
  4. In the VRF field, enter the identifier for the VRF interface you want to use.

    In this example, we are starting our trace at leaf01 and ending it at 10.1.3.103 using VRF vrf1.
  5. Click Run Now. A corresponding Trace Results card is opened on your workbench. Refer to View Layer 3 Trace Results for details.

Create a Layer 2 Trace

It is helpful to verify the connectivity between two devices when you suspect an issue is preventing proper communication between them. It you cannot find a path through a layer 2 path, you might also try checking connectivity through a layer 3 path.

To create a layer 2 trace request:

  1. Open the large Trace Request card.
  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.
  3. In the Destination field, enter the MAC address for where you want to end the trace.
  4. In the VLAN ID field, enter the identifier for the VLAN you want to use.

    In this example, we are starting our trace at leaf01 and ending it at 00:03:00:33:33:01 using VLAN 13.
  5. Click Run Now. A corresponding Trace Results card is opened on your workbench. Refer to View Layer 2 Trace Results for details.

Create a Trace to Run on a Regular Basis (Scheduled Trace)

There may be paths through your network that you consider critical to your everyday or particularly important operations. In that case, it might be useful to create one or more traces to periodically confirm that at least one path is available between the relevant two devices. Scheduling a trace request can be performed from the large Trace Request card. 

To schedule a trace:

  1. Open the large Trace Request card.
  2. In the Source field, enter the hostname or IP address of the device where you want to start the trace.
  3. In the Destination field, enter the MAC address (layer 2) or IP address (layer 3) of the device where you want to end the trace.
  4. Optionally, enter a VLAN ID (layer 2) or VRF interface (layer 3).
  5. Click Timeframe under Schedule to specify how often you want to run the trace.
  6. Click Date/Time to specify the day you want the trace to run for the first time.
  7. Click Next.
  8. Click the time you want the trace to run for the first time.
  9. Click OK.
  10. Click Save As New.
  11. Provide a name for the trace. Note: This name must be unique for a given user.
  12. Click Save. You can now run this trace on demand or wait for it to run on its defined schedule.

Run a Scheduled Trace on Demand

You may find that, although you have a schedule for a particular trace, you want to have visibility into the connectivity data now. You can run a scheduled trace on demand from the small, medium and large Trace Request cards.

To run a scheduled trace now:

  1. Open the small or medium or large Trace Request card.

       
  2. Select the scheduled trace from the Select Trace or New Trace Request list.
  3. Click Go or Run Now. A corresponding Trace Results card is opened on your workbench. 

View On-demand Trace Results

Once you have started an on-demand trace, the results are displayed in the medium Trace Results card by default. You may view the results in more or less detail by switching to the large or small Trace Results card, respectively.

On-demand Trace Results Card Workflow Summary

The small On-demand Trace Results card displays:

ItemDescription
Indicates an on-demand trace result
 Source and destination of the trace, identified by their address or hostname. Source is listed on top with arrow pointing to destination.
Indicates success or failure of the trace request. A successful result implies all paths were successful without any warnings or failures. A failure result implies there was at least one path with warnings or errors.

The medium On-demand Trace Results card displays:

ItemDescription
Indicates an on-demand trace result
TitleOn-demand Trace Result
 Source and destination of the trace, identified by their address or hostname. Source is listed on top with arrow pointing to destination.
Indicates success or failure of the trace request. A successful result implies all paths were successful without any warnings or failures. A failure result implies there was at least one path with warnings or errors.
Total Paths FoundNumber of paths found between the two devices
MTU OverallAverage size of the maximum transmission unit for all paths
Minimum HopsSmallest number of hops along a path between the devices
Maximum HopsLargest number of hops along a path between the devices

The large On-demand Trace Results card displays:

ItemDescription
Indicates an on-demand trace result
TitleOn-demand Trace Result
Indicates success or failure of the trace request. A successful result implies all paths were successful without any warnings or failures. A failure result implies there was at least one path with warnings or errors.
 Source and destination of the trace, identified by their address or hostname. Source is listed on top with arrow pointing to destination.
Distribution by Hops chartDisplays the distributions of various hop counts for the available paths
Distribution by MTU chartDisplays the distribution of MTUs used on the interfaces used in the available paths
Table

Provides detailed path information, sorted by the route identifier, including:

  • Route ID: Identifier of each path
  • MTU: Average speed of the interfaces used
  • Hops: Number of hops to get from the source to the destination device
  • Warnings: Number of warnings encountered during the trace on a given path
  • Errors: Number of errors encountered during the trace on a given path
Total Paths FoundNumber of paths found between the two devices
MTU OverallAverage size of the maximum transmission unit for all paths
Minimum HopsSmallest number of hops along a path between the devices

The full screen On-demand Trace Results card displays:

ItemDescription
TitleOn-demand Trace Results
Trace Results tab

Provides detailed path information, sorted by the Resolution Time (date and time results completed), including:

  • SCR.IP: Source IP address
  • DST.IP: Destination IP address
  • Max Hop Count: Largest number of hops along a path between the devices
  • Min Hop Count: Smallest number of hops along a path between the devices
  • Total Paths: Number of paths found between the two devices
  • PMTU: Average size of the maximum transmission unit for all interfaces along the paths
  • Errors: Message provided for analysis when a trace fails
ExportEnables export of all or selected items in a CSV or JSON formatted file
Enables manipulation of table display; choose columns to display and reorder columns

View Layer 2 Trace Results

When you start the trace, the corresponding results card is opened on your workbench. While it is working on the trace, a notice is shown on the card indicating it is running.

Once the job is completed, the results are displayed.

In this example, we see that the trace was successful. Four paths were found between the devices, each with four hops and with an overall MTU of 1500. If there was a difference between the minimum and maximum number of hops or other failures, viewing the results on the large card would provide additional information.

In our example, we can verify that every path option had four hops since the distribution chart only shows one hop count and the table indicates each path had a value of four hops. Similarly, you can view the MTU data. If there had been any warnings, the count would have been visible in the table.

View Layer 3 Trace Results

When you start the trace, the corresponding results card is opened on your workbench. While it is working on the trace, a notice is shown on the card indicating it is running.

Once results are obtained, it displays them. Using our example from earlier, the following results are shown:

In this example, we see that the trace was successful. Eight paths were found between the devices, each with five hops and with an overall MTU of 1500. If there was a difference between the minimum and maximum number of hops or other failures, viewing the results on the large card would provide additional information.

In our example, we can verify that every path option had five hops since the distribution chart only shows one hop count and the table indicates each path had a value of five hops. Similarly, you can view the MTU data. If there had been any warnings, the count would have been visible in the table.

View All On-demand Trace Results

If you have run multiple on-demand traces, you may find it easier to view the results all together in a single view. The full screen Trace Results card provides this information.

To view all on-demand trace results, open the full screen On-demand Trace Results card.

Ordered by most recent trace, you can now view all recent traces together.

View Scheduled Trace Results

You can view the results of scheduled traces at any time. Results are displayed on the Scheduled Trace Results cards.

Scheduled Trace Results Card Workflow Summary

The small Scheduled Trace Results card displays:

ItemDescription
Indicates a scheduled trace result
 Source and destination of the trace, identified by their address or hostname. Source is listed on left with arrow pointing to destination.
Results

Summary of trace results: a successful result implies all paths were successful without any warnings or failures; a failure result implies there was at least one path with warnings or errors.

  •  Number of trace runs completed in the designated time period
  •  Number of runs with warnings
  •  Number of runs with errors

The medium Scheduled Trace Results card displays:

ItemDescription
Time periodRange of time in which the displayed data was collected; applies to all card sizes
Indicates a scheduled trace result
TitleScheduled Trace Result
Summary

Name of scheduled validation and summary of trace results: a successful result implies all paths were successful without any warnings or failures; a failure result implies there was at least one path with warnings or errors.

  •  Number of trace runs completed in the designated time period
  •  Number of runs with warnings
  •  Number of runs with errors
Charts

Heat map: A time segmented view of the results. For each time segment, the color represents the percentage of warning and failed results. Refer to Granularity of Data Shown Based on Time Period for details on how to interpret the results.

Unique Bad Nodes: Distribution of unique nodes that generated the indicated warnings and/or failures

The large Scheduled Trace Results card contains two tabs:

The Results tab displays:

ItemDescription
Time periodRange of time in which the displayed data was collected; applies to all card sizes
Indicates a scheduled trace result
TitleScheduled Trace Result
Summary

Name of scheduled validation and summary of trace results: a successful result implies all paths were successful without any warnings or failures; a failure result implies there was at least one path with warnings or errors.

  •  Number of trace runs completed in the designated time period
  •  Number of runs with warnings
  •  Number of runs with errors
Charts

Heat map: A time segmented view of the results. For each time segment, the color represents the percentage of warning and failed results. Refer to Granularity of Data Shown Based on Time Period for details on how to interpret the results.

Small charts: Display counts for each item during the same time period, for the purpose of correlating with the warnings and errors shown in the heat map.

Table/Filter options

When the Failures filter option is selected, the table displays the failure messages received for each run.

When the Paths filter option is selected, the table displays all of the paths tried during each run.

When the Warning filter option is selected, the table displays the warning messages received for each run.

The Configuration tab displays:

ItemDescription
Time periodRange of time in which the displayed data was collected; applies to all card sizes
Indicates a scheduled trace result
TitleScheduled Trace Configuration (Scheduled Trace Result)
Source

Address or hostname of the device where the trace was started

Destination

Address of the device where the trace was stopped

Schedule

The frequency and starting date and time to run the trace

VRFVirtual Route Forwarding interface, when defined
VLANVirtual LAN identifier, when defined
NameUser-defined name of the scheduled trace
Run NowStart the trace now
EditModify the trace. Opens Trace Request card with this information pre-populated.

The full screen Scheduled Trace Results card displays:

ItemDescription
TitleScheduled Trace Results
Closes full screen card and returns to workbench
Time periodRange of time in which the displayed data was collected; applies to all card sizes; select an alternate time period by clicking 
ResultsNumber of results found for the selected tab
Scheduled Trace Results tab

Displays the basic information about the trace, including:

  • Resolution Time: Time that trace was run
  • SRC.IP: IP address of the source device
  • DST.IP: Address of the destination device
  • Max Hop Count: Maximum number of hops across all paths between the devices
  • Min Hop Count: Minimum number of hops across all paths between the devices
  • Total Paths: Number of available paths found between the devices
  • PMTU: Average of the maximum transmission units for all paths
  • Errors: Message provided for analysis if trace fails

Click on a result to open a detailed view of the results.

ExportEnables export of all or selected items in a CSV or JSON formatted file
Enables manipulation of table display; choose columns to display and reorder columns

Granularity of Data Shown Based on Time Period

On the medium and large Trace Result cards, the status of the runs is represented in heat maps stacked vertically; one for runs with warnings and one for runs with failures. Depending on the time period of data on the card, the number of smaller time blocks used to indicate the status varies.  A vertical stack of time blocks, one from each map, includes the results from all checks during that time. The results are shown by how saturated the color is for each block. If all traces run during that time period pass, then both blocks are 100% gray. If there are only failures, the associated lower blocks are 100% saturated white and the warning blocks are 100% saturated gray. As warnings and failures increase, the blocks increase their white saturation. As warnings or failures decrease, the blocks increase their gray saturation. An example heat map for a time period of 24 hours is shown here with the most common time periods in the table showing the resulting time blocks.

Time Period
Number of Runs
Number Time Blocks
Amount of Time in Each Block
6 hours1861 hour
12 hours36121 hour
24 hours72241 hour
1 week50471 day
1 month2,086301 day
1 quarter7,000131 week

View Scheduled Trace Results

Once a scheduled trace request has completed, the results are available in the corresponding Trace Result card.

To view the results:

  1. Open the full screen Trace Request card to view all scheduled traces that have been run. 
  2. Select the scheduled trace you want to view results for by clicking in the first column of the result and clicking the check box.
  3. On the Edit Menu that appears at the bottom of the window, click (Open Cards). This opens the medium Scheduled Validation Results card(s) for the selected items.
  4. Note the distribution of results. Are there many failures? Are they concentrated together in time? Has the trace begun passing again?
  5. Hover over the heat maps to view the status numbers and what percentage of the total results that represents for a given region. 
  6. Switch to the large Scheduled Trace Result card.
  7. If there are a large number of warnings or failures, view the associated messages by selecting Failures or Warning in the filter above the table. This might help narrow the failures down to a particular device or small set of devices that you can investigate further.
  8. Look for a consistent number of paths, MTU, hops in the small charts under the heat map. Changes over time here might correlate with the messages and give you a clue to any specific issues. Note if the number of bad nodes changes over time. Devices that become unreachable are often the cause of trace failures.
  9. View the available paths for each run, by selecting Paths in the filter above the table.
  10. You can view the configuration of the request that produced the results shown on this card workflow, by hovering over the card and clicking . If you want to change the configuration, click Edit to open the large Trace Request card, pre-populated with the current configuration. Follow the instructions in Create a Scheduled Trace Request to make your changes in the same way you created a new scheduled trace.
  11. To view a summary of all scheduled trace results, switch to the full screen card.
  12. Look for changes and patterns in the results for additional clues to isolate root causes of trace failures. Select and view related traces using the Edit menu.
  13. View the details of any specific trace result by clicking on the trace. A new window opens similar to the following:

    Scroll to the right to view the information for a given hop. Scroll down to view additional paths.
    This display shows each of the hosts and detailed steps the trace takes to validate a given path between two devices. Using Path 1 as an example, each path can be interpreted as follows:
    Hop 1 is from the source device, spine-1 in this case. It exits this device at switch port 7 with an MTU of 9202 and over the default VRF too get to tor-1. The trace goes in to swp 3 with an MTU of 9202 over its default VRF interface. It exits tor-1 through switch port 5 and so on.
  14. Export this data using the Export button or click  to return to the results list to view another trace in detail.