Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Platform Support

Restrictions

Restrictions on C9400 SVL

Restrictions on C9500 SVL

Restrictions on C9500H SVL

Restrictions on C9600 SVL

Troubleshoot

Check the StackWise-virtual Configuration

Check Switch Status

Check SVL Link Status

Check DAD Link Status

Check ASIC Core IFM Map

Check FED Channel Health

Check LMP Health

Shut/Unshut SVL Port

Introduction

This document describes how to identify, collect useful logs, and solve problems that occur with StackWise-virtual (SVL) on Catalyst 9000 switches.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

- StackWise-virtual (SVL)

- Catalyst 9000 Switches

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document also provides support matrix, restrictions, commands, and common issues seen with SVL.

When you troubleshoot SVL, there are basic criteria that you need to understand and adhere to when you deploy SVL with the Cisco Catalyst 9000 switches. These are criteria that need to be met:

- Ensure that SVL is supported for a given switch, platform, and software release.

- Ensure that SVL is configured as per the guidelines provided in the configuration guide and strictly keeps to the provided restrictions.

- Ensure that SVL links are physically connected between the switches.

Platform Support

Currently supported platforms listed together with software train that introduced support.

Software train

SVL support introduced on

Comments

16.3

C3850-48XS

16.6

C9500-24Q

16.8

C9500-12Q, C9500-24Q, C9500-16X, C9500-40X

C3850-12XS, C3850-24XS, C3850-48XS

SVL/DAD link not supported on uplink module on C9500-16X, C9500-40X, C3850-12XS, C3850-24XS

16.9

C9404R, C9407R

- SVL/DAD link supported on supervisor ports only- SVL supported only with SUP-1 or SUP-1XL

16.10

C9500-32C, C9500-32QC, C9500-24Y4C, C9500-48Y4C

StackWise virtual support first introduced on C9500 High-performance models

16.11

C9500-NM-2Q, C9500-NM-8X, C9500-16X, C9500-40X

C9410R, C9400-SUP-1XL-Y

- SVL/DAD link now supported on uplink modules on C9500-16X, C9500-40X

-SVL supported on all C9400 chassis with SUP-1 or SUP-1XL or SUP-1XL-Y

- SUP XL-25G SVL SUPPORT

16.12

C9606R

- StackWise virtual support first introduced on C9600 chassis with C9600-LC-48YL and C9600-LC-24C

- SVL FIP support - UPLINKS FIPS SUPPORT ON SVL

- Secure SVL support

17.1

C9606R

New LC C9600-LC-48TX support on C9600 HA & SVL

17.2

C9606R

- Quad-Supervisor with Route Processor Redundancy (RPR) support first introduced on C9606R chassis

- Support for DAD links on C9600-LC-48S

Restrictions

Most of the restrictions can be found in the Cisco StackWise-virtual Configuration guide, next are some of the other restrictions which are more platform-specific and can not be explicitly called out in the configuration guide.

Common Restriction

Dual-Active Detection (DAD) and SVL configuration must be performed manually and the devices must be rebooted for configuration changes to take effect.

Restrictions on C9400 SVL

- SVL connections can be established through 10G, 40G or 25G (available only on C9400-SUP-1XL-Y) uplink ports on the supervisor modules and 10G downlink ports on the line cards

- SVL configuration on 1G interfaces is not supported.

- SVL and DAD links in 16.9.1 are supported on supervisor ports. DAD ePAgP can be configured on line card, as well as supervisor ports.

- SVL and DAD on certain line card is controlled availability in 16.11.1

For a complete list of restrictions on C9400, refer to the High Availability Configuration Guide, Catalyst 9400 Switches.

Restrictions on C9500 SVL

- When SVL is configured on Cisco Catalyst 9500 Series Switches with C9500-NM-2Q (2x40G), you cannot use a combination of fixed downlink and modular uplink ports. SVLs must have the same speed on each member.

- The 40G ports on a C9500-NM-2Q cannot be combined with the downlink ports on a switch as they have different speeds.

- In a Cisco StackWise Virtual solution, interfaces that support 4X10G breakout cables and QSA can be used for data/control traffic but cannot be used to configure SVLs or DAD links.

Restrictions on C9500H SVL

- On C9500-32C switches, you can configure SVL and DAD only on interfaces numbered 1-16 on the front panel of the switch.

- On C9500-32QC, you can configure SVL and DAD only on native 100G and 40G interfaces (default configuration ports). You cannot configure SVL and DAD on converted 100G and 40G interfaces.

- SVL/DAD links cannot be configured on the 4X10G and 4X25G breakout interfaces on C9500-32C SVL. However, breakout interfaces can be used for data/control traffic when C9500 Series High-Performance Switches are configured in StackWise Virtual mode.

- On Cisco Catalyst C9500 Series High-Performance Switches, SVL link configurations on 1G interfaces are not supported.

- 9500H does not support the SVL feature on any 16.9 release. (Cisco removed this configuration option via Cisco bug ID CSCvt46115 in 16.9.6). Please verify your 9500H is running 16.10 or later.

Restrictions on C9600 SVL

- On Cisco Catalyst C9600R Switches, SVL/DAD links cannot be configured on the 4X10 and 4X25G breakout interfaces. However, breakout interfaces can be used for data traffic when C9600R Switches are configured in StackWise Virtual mode.

- On Cisco Catalyst C9600R Switches, SVL link configurations on 1G interfaces are not supported.

Troubleshoot

Check the StackWise-virtual Configuration

Step 1. Use show running configuration to ensure StackWise-virtual configuration exists and correct in running-config.

Note: Only StackWise-virtual configuration is listed in the output.

Switch#show running-config

Building configuration...

Current configuration : 9047 bytes

!

! Last configuration change at 09:36:41 UTC Fri Nov 13 2020

!

version 16.11

[....]

stackwise-virtual

domain 1

!

[.....]

!

license boot level network-advantage addon dna-advantage

!

[....]

interface GigabitEthernet1/1/0/43

stackwise-virtual dual-active-detection

!

interface GigabitEthernet1/1/0/44

!

interface TenGigabitEthernet1/3/0/1

stackwise-virtual link 1

!

interface TenGigabitEthernet1/3/0/2

stackwise-virtual link 1

!

interface TenGigabitEthernet1/3/0/3

stackwise-virtual link 1

!

interface TenGigabitEthernet1/3/0/4

stackwise-virtual link 1

!

interface TenGigabitEthernet1/3/0/5

!

interface TenGigabitEthernet1/3/0/6

[....]

interface TenGigabitEthernet2/3/0/1

stackwise-virtual link 1

!

interface TenGigabitEthernet2/3/0/2

stackwise-virtual link 1

!

interface TenGigabitEthernet2/3/0/3

stackwise-virtual link 1

!

interface TenGigabitEthernet2/3/0/4

stackwise-virtual link 1

!

interface TenGigabitEthernet2/3/0/5

!

interface TenGigabitEthernet2/3/0/6

!

interface GigabitEthernet2/5/0/43

stackwise-virtual dual-active-detection

!

interface GigabitEthernet2/5/0/44

!

Step 2. Show romvar can be used to verify that the rommon variables display the correct values which correspond to the configured StackWise-virtual configuration

Switch#show romvar

Switch 1

ROMMON variables:

SWITCH_NUMBER="1"

MODEL_NUM="C9400-SUP-1XL"

LICENSE_BOOT_LEVEL="network-advantage+dna-advantage,all:MACALLAN-CHASSIS;"

D_STACK_DISTR_STACK_LINK2=""

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="1"

D_STACK_DISTR_STACK_LINK1="Te1/3/0/1,Te1/3/0/2,Te1/3/0/3,Te1/3/0/4,"

D_STACK_DAD="Gi1/1/0/43,"

Switch 2

ROMMON variables:

LICENSE_BOOT_LEVEL="network-advantage+dna-advantage,all:MACALLAN-CHASSIS;"

D_STACK_DISTR_STACK_LINK2=""

SWITCH_NUMBER="2"

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="1"

D_STACK_DISTR_STACK_LINK1="Te2/3/0/1,Te2/3/0/2,Te2/3/0/3,Te2/3/0/4,"

D_STACK_DAD="Gi2/5/0/43,"

Note: The previous output displays the rommon variable created and updated from a C9400/C9600 switch configured in SVL mode. They use 4 tuple interface conventions when configured in SVL mode.

D_STACK_DISTR_STACK_LINK1 represents the rommon variable for SVL links from both switch1 & switch2

D_STACK_DAD represents the rommon variable for DAD links

D_STACK_DOMAIN_NUM represents the SVL domain number, ensure they are the same on both switch1 & switch2

D_STACK_MODE represents that Cat9k switch is configured in SVL/distributed stack mode

SVL Domain number, as well as SVL and DAD link configuration, is not only stored as part of running-config/startup-config but also as rommon variables

The rommon variables and the related values of these rommon variables can be verified. Use CLI as shown previously

The next output displays the rommon variable created and updated from a C9500H/C9500 switch configured in SVL mode where the interface name tracks 3 tuple model when configured in SVL mode.

Switch#show romvar | include D_STACK

D_STACK_DISTR_STACK_LINK2=""

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="100"

D_STACK_DISTR_STACK_LINK1="Te1/0/3,Te1/0/5,"

D_STACK_DAD="Te1/0/4,"

D_STACK_DISTR_STACK_LINK2=""

D_STACK_MODE="aggregation"

D_STACK_DOMAIN_NUM="100"

D_STACK_DISTR_STACK_LINK1="Te1/0/3,Te1/0/5,"

D_STACK_DAD="Te1/0/4,"

Check Switch Status

The switch status of both the chassis is expected to be in Ready state. Also, check the show module command output to ensure all LCs are in OK state.

Switch#show switch

Switch/Stack Mac Address : 00a7.42d7.4620 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role Mac Address Priority Version State

-------------------------------------------------------------------------------------

*1 Active 00a7.42d7.3680 1 V02 Ready

2 Standby 00a7.42d7.4620 1 V02 Ready

Switch States

SVL switch states

Description

Ready

Stack/switch is ready for use

mismatch

Image version mismatched with the peer switch

Initializing

The system is not ready for use, still under initialize phase

HA sync in progress

The system is not ready for use, it waits for config sync to complete

The next is the show module output from a 9400 switch

Switch#show module

Chassis Type: C9407R

Switch Number 1

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 48 48-Port UPOE 10/100/1000 (RJ-45) WS-XL48U JAE201900TY

3 10 Supervisor 1 XL Module C9400-SUP-1XL JAE222805NB

7 24 24-Port 10 Gigabit Ethernet (SFP+) C9400-LC-24XS JAE22170EAG

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

1 0035.1A8D.2DEC to 0035.1A8D.2E1B 0.4 17.1.1r 16.11.01c ok

3 2C5A.0F1C.626C to 2C5A.0F1C.6275 2.0 17.1.1r 16.11.01c ok

7 780C.F02B.4D50 to 780C.F02B.4D67 1.0 17.1.1r 16.11.01c ok

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Active sso sso

Switch Number 2

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

3 10 Supervisor 1 XL Module C9400-SUP-1XL JAE222805QB

5 48 48-Port 10/100/1000 (RJ-45) C9400-LC-48T JAE213207ZX

Mod MAC addresses Hw Fw Sw Status

---+--------------------------------+----+------------+------------------+--------

3 2C5A.0F1C.622C to 2C5A.0F1C.6235 2.0 17.1.1r 16.11.01c ok

5 E4AA.5D59.FB48 to E4AA.5D59.FB77 1.0 17.1.1r 16.11.01c ok

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3 Standby sso sso

Chassis MAC address range: 44 addresses from 2c5a.0f1c.6240 to 2c5a.0f1c.626b

Switch#

Check SVL Link Status

The Link-Status of SVL ports are expected to be in "U" (Up) state and the Protocol-Status has to be "R" (Ready)

Note: These outputs are applicable on C9500/C9500H platforms which use 3 tuples for interface convention when configured in SVL mode

Switch#show stackwise-virtual link

Stackwise Virtual Link(SVL) Information:

----------------------------------------

Flags:

------

Link Status

-----------

U-Up D-Down

Protocol Status

---------------

S-Suspended P-Pending E-Error T-Timeout R-Ready

-----------------------------------------------

Switch SVL Ports                     Link-Status Protocol-Status

------ --- -----                     ----------- ---------------

1     1 FortyGigabitEthernet1/0/1  U           R

FortyGigabitEthernet1/0/2  U           R

2     1 FortyGigabitEthernet2/0/1  U           R

FortyGigabitEthernet2/0/2  U           R

SVL Protocol status

Description

Suspended

Protocol is in a suspended state, could be seen when SVL link is down

Check link status and ensure link is up

Pending

Protocol is pending, could be seen when link is not bundled yet

Check the remote side of the link, if both sides are in pend check LMP health

Error

Protocol is in error state, could be seen when LMP packets are exchanged with value errors

Timeout

Protocol has timed out, could be seen when LMP messages are not transmitted or received over 16 second time window

Ready

Protocol is in ready state, this is the desired state in normal operation

Both LMP messages as well as SDP messages are exchanged successfully

Note: These outputs are applicable on C9400/C9600 platforms which use 4 tuple for interface convention when configured in SVL mode.

Switch#show stackwise-virtual link

Stackwise Virtual Link(SVL) Information:

----------------------------------------

Flags:

------

Link Status

-----------

U-Up D-Down

Protocol Status

---------------

S-Suspended P-Pending E-Error T-Timeout R-Ready

-----------------------------------------------

Switch SVL Ports                     Link-Status Protocol-Status

------ --- -----                     ----------- ---------------

1     1 FortyGigabitEthernet1/1/0/3  U           R

FortyGigabitEthernet1/1/0/5  U           R

2     1 FortyGigabitEthernet2/1/0/3  U           R

FortyGigabitEthernet2/1/0/5  U           R

Check DAD Link Status

Note: These outputs are applicable on C9500/C9500H platforms which use 3 tuple for interface convention when configured in SVL mode.

Switch#show stackwise-virtual dual-active-detection

Dual-Active-Detection Configuration:

-------------------------------------

Switch Dad port Status

------ ------------ ---------

1  FortyGigabitEthernet1/0/4  up

2  FortyGigabitEthernet2/0/4  up

Note: These outputs are applicable on C9400/C9600 platforms which use 4 tuple for interface convention when configured in SVL mode

Switch#show stackwise-virtual dual-active-detection

Dual-Active-Detection Configuration:

-------------------------------------

Switch Dad port Status

------ ------------ ---------

1  FortyGigabitEthernet1/1/0/4  up

2  FortyGigabitEthernet2/1/0/4  up

Check ASIC Core IFM Map

Note: This output is a typical output from C9500H SVL. The number of ASIC can differ based on the platform/SKU.

Switch#sh platform software fed sw active ifm mapp

Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active

TwentyFiveGigE1/0/1       0x3c      1    0    1    20    0    16     4     1    97   NIF  Y

TwentyFiveGigE1/0/2       0x3d      1    0    1    21    0    17     5     2    98   NIF  Y

Switch#sh platform software fed sw standby ifm mapp

Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active

TwentyFiveGigE1/0/1       0x8      1     0    1    20   0      16    4    1    1    NIF    Y

TwentyFiveGigE1/0/2       0x9      1     0    1    21   0      17    5    2    2    NIF    Y

Note: This output is a typical output from C9600 SVL. The number of ASIC can differ based on the platform/SKU.

Switch#sh platform software fed sw active ifm mapp

Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active

FortyGigabitEthernet1/1/0/3  0xb    0    0    0    16   0      16    0    3    2360  NIF   Y

FortyGigabitEthernet1/1/0/5  0xd    1    0    1    8    0      14    1    5    2361  NIF   Y

Switch#sh platform software fed sw standby ifm mapp

Interface                 IF_ID    Inst Asic Core Port SubPort Mac  Cntx LPN  GPN  Type Active

FortyGigabitEthernet2/1/0/3 0x6b     0   0   0    16    0      16    0    3   2361  NIF   Y

FortyGigabitEthernet2/1/0/5 0x6d     1   0   1     8    0      14    1    5   2360  NIF   Y

Note: When 16.3.x version is used on the switch, the command would be show platform software fed sw active ifm mapp lpn.

Check FED Channel Health

Switch#show platform software fed switch active fss counters

FSS Packet Counters

SDP                           LMP

TX       |     RX            TX       |    RX

---------------------        ---------------------

72651          72666         1157750      1154641

OOB1                            OOB2

TX       |      RX            TX        |        RX

---------------------         ---------------------

8            8              7740057        7590208

EMP

TX        |      RX               LOOPBACK

----------------------        -------------------

0                0                   79

Switch#show platform software fed switch active fss err-pkt-counters latency

Switch#show platform software fed switch active fss err-pkt-counters seqerr

Switch#show platform software fed switch active fss registers | i group

Note: Ensure the counters increment in the previous show commands. Verify this show commands 3-4 times repetitively.

Check LMP Health

Use this show commands to check the LMP health

Switch#show platform software fed sw active fss bundle

Stack Port (0-Based) 0

Control port 16

Next Probable Control port Unknown

Member Port LPN list

-------------------

LPN:Partner_LPN

1.16:1.16

Stack Port (0-Based) 1

Control port Unknown

Next Probable Control port Unknown

Member Port LPN list

-------------------

LPN:Partner_LPN

Switch#show platform software fed switch active fss sdp-packets

FED FSS SDP packets max 10:

------------------------

FED-> Nif Mgr

Timestamp Src Mac Dst Mac Seq Num

--------------------------------------------------------------------

Sun Nov 15 18:59:07 2020 bc26:c722:9ef8 ffff:ffff:ffff 51843

Sun Nov 15 18:59:11 2020 bc26:c722:9ef8 ffff:ffff:ffff 51844

Sun Nov 15 18:59:15 2020 bc26:c722:9ef8 ffff:ffff:ffff 51845

Sun Nov 15 18:59:19 2020 bc26:c722:9ef8 ffff:ffff:ffff 51846

Sun Nov 15 18:59:23 2020 bc26:c722:9ef8 ffff:ffff:ffff 51847

Sun Nov 15 18:59:27 2020 bc26:c722:9ef8 ffff:ffff:ffff 51848

Sun Nov 15 18:59:31 2020 bc26:c722:9ef8 ffff:ffff:ffff 51849

Sun Nov 15 18:59:35 2020 bc26:c722:9ef8 ffff:ffff:ffff 51850

Sun Nov 15 18:58:59 2020 bc26:c722:9ef8 ffff:ffff:ffff 51841

Sun Nov 15 18:59:03 2020 bc26:c722:9ef8 ffff:ffff:ffff 51842

Nif Mgr -> FED

Timestamp Src Mac Dst Mac Seq Num

--------------------------------------------------------------------

Sun Nov 15 18:59:29 2020 bc26:c722:dae0 ffff:ffff:ffff 51863

Sun Nov 15 18:59:33 2020 bc26:c722:dae0 ffff:ffff:ffff 51864

Sun Nov 15 18:59:37 2020 bc26:c722:dae0 ffff:ffff:ffff 51865

Sun Nov 15 18:59:01 2020 bc26:c722:dae0 ffff:ffff:ffff 51856

Sun Nov 15 18:59:05 2020 bc26:c722:dae0 ffff:ffff:ffff 51857

Sun Nov 15 18:59:09 2020 bc26:c722:dae0 ffff:ffff:ffff 51858

Sun Nov 15 18:59:13 2020 bc26:c722:dae0 ffff:ffff:ffff 51859

Sun Nov 15 18:59:17 2020 bc26:c722:dae0 ffff:ffff:ffff 51860

Sun Nov 15 18:59:21 2020 bc26:c722:dae0 ffff:ffff:ffff 51861

Sun Nov 15 18:59:25 2020 bc26:c722:dae0 ffff:ffff:ffff 51862

Switch#show platform software fed switch active fss lmp-packets

Interface:TenGigabitEthernet1/3/0/1 IFID:0x37

FED FSS LMP packets max 10:

FED --> Nif Mgr

Timestamp                          Local      Peer       Seq

LPN        LPN        Num

-------------------------------------------------------

Sun Nov 15 19:01:31 2020            1              1              206696

Sun Nov 15 19:01:32 2020            1              1              206697

Sun Nov 15 19:01:33 2020            1              1              206698

Sun Nov 15 19:01:34 2020            1              1              206699

Sun Nov 15 19:01:36 2020            1              1              206701

Sun Nov 15 19:01:37 2020            1              1              206702

Sun Nov 15 19:01:27 2020            1              1              206692

Sun Nov 15 19:01:28 2020            1              1              206693

Sun Nov 15 19:01:29 2020            1              1              206694

Sun Nov 15 19:01:30 2020            1              1              206695

Nif Mgr --> FED

Timestamp                           Local      Peer       Seq

LPN        LPN        Num

-------------------------------------------------------

Sun Nov 15 19:01:29 2020            1              1              206696

Sun Nov 15 19:01:30 2020            1              1              206697

Sun Nov 15 19:01:31 2020            1              1              206698

Sun Nov 15 19:01:32 2020            1              1              206699

Sun Nov 15 19:01:33 2020            1              1              20670

Sun Nov 15 19:01:34 2020            1              1              206701

Sun Nov 15 19:01:35 2020            1              1              206702

Sun Nov 15 19:01:36 2020            1              1              206703

Sun Nov 15 19:01:37 2020            1              1              206704

Sun Nov 15 19:01:28 2020            1              1              206695

Interface:TenGigabitEthernet1/3/0/2 IFID:0x38

FED FSS LMP packets max 10:

FED --> Nif Mgr

Timestamp                          Local      Peer       Seq

LPN        LPN        Num

-------------------------------------------------------

Sun Nov 15 19:01:32 2020            2              2              206697

Sun Nov 15 19:01:33 2020            2              2              206698

Sun Nov 15 19:01:34 2020            2              2              206699

Sun Nov 15 19:01:35 2020            2              2              206700

Sun Nov 15 19:01:36 2020            2              2              206701

Sun Nov 15 19:01:37 2020            2              2              206702

Sun Nov 15 19:01:28 2020            2              2              206693

Sun Nov 15 19:01:29 2020            2              2              206694

Sun Nov 15 19:01:30 2020            2              2              206695

Sun Nov 15 19:01:31 2020            2              2              206696

Nif Mgr --> FED

Timestamp                          Local      Peer       Seq

LPN        LPN        Num

-------------------------------------------------------

Sun Nov 15 19:01:33 2020            2              2              206700

Sun Nov 15 19:01:34 2020            2              2              206701

Sun Nov 15 19:01:35 2020            2              2              206702

Sun Nov 15 19:01:36 2020            2              2              206703

Sun Nov 15 19:01:37 2020            2              2              206704

Sun Nov 15 19:01:28 2020            2              2              206695

Sun Nov 15 19:01:29 2020            2              2              206696

Sun Nov 15 19:01:30 2020            2              2              206697

Sun Nov 15 19:01:31 2020            2              2              206698

Sun Nov 15 19:01:32 2020            2              2              206699

Switch#show platform software fed switch active fss interface-counters

Interface TenGigabitEthernet1/3/0/1 IFID: 0x37 Counters

LMP

TX    |    RX

---------------------

206125      204784

Interface TenGigabitEthernet1/3/0/2 IFID: 0x38 Counters

LMP

TX    |    RX

---------------------

207012      206710

Note: Ensure the counters increment in the previous show commands. Verify this show commands 3-4 times repetitively

Switch#test platform software nif_mgr lmp member-port 1

Member port LPN 1 details

----------------------------

Transmitting on LPN: 1

member_port idx: 0

Stack Port: 0

Connection Status: Ready

Port Link Status: Up

LMP HELLO disabled: FALSE

LMP Tx count: 3864

LMP Tx seq no: 3864

LMP Rx count: 3856

LMP Timeout Rx count: 0

LMP Partner Platform Information:

Blueshift Version:1

Distributed Stack Domain:100

Distributed Stack Mode:1

System Model String:C9500-24Q

System Product ID:FCW2144A3KF

System Version ID:V01

Stack MAC Address:0027:90be:1f00

System CMI Index:0

LMP Port LPN:1

System Switch Number:2

LMP PENDING Partner Platform Information:

Blueshift Version:1

Distributed Stack Domain:100

Distributed Stack Mode:1

System Model String:C9500-24Q

System Product ID:FCW2144A3KF

System Version ID:V01

Stack MAC Address:0027:90be:1f00

System CMI Index:0

LMP Port LPN:1

Switch#test platform software nif_mgr lmp member-port 2

Member port LPN 2 details

----------------------------

Transmitting on LPN: 2

member_port idx: 1

Stack Port: 0

Connection Status: Ready

Port Link Status: Up

LMP HELLO disabled: FALSE

LMP Tx count: 3873

LMP Tx seq no: 3873

LMP Rx count: 3870

LMP Timeout Rx count: 0

LMP Partner Platform Information:

Blueshift Version:1

Distributed Stack Domain:100

Distributed Stack Mode:1

System Model String:C9500-24Q

System Product ID:FCW2144A3KF

System Version ID:V01

Stack MAC Address:0027:90be:1f00

System CMI Index:0

LMP Port LPN:2

System Switch Number:2

LMP PENDING Partner Platform Information:

Blueshift Version:1

Distributed Stack Domain:100

Distributed Stack Mode:1

System Model String:C9500-24Q

System Product ID:FCW2144A3KF

System Version ID:V01

Stack MAC Address:0027:90be:1f00

System CMI Index:0

LMP Port LPN:2

Switch#test platform software nif_mgr lmp status

Switch#test platform software nif_mgr lmp stack-port 1

Stack port 1 details

----------------------------

stack_port idx:0

Stack Link status:Up

Number Member Ports:1

Member Port LPN List:

1/16,

Switch#test platform software nif_mgr lmp stack-port 2

Stack port 2 details

----------------------------

stack_port idx:1

Stack Link status:Down

Number Member Ports:0

Member Port LPN List:

Note: The previous are test commands to verify the LMP packet counters and status of the LMP ports on Cat9k SVL

Shut/Unshut SVL Port

Shut and no shut comands are disabled on SVL and DAD ports when the system configured in StackWise-virtual mode. For test purposes, if there is a need to shut/unshut SVL ports, try as shown here:

Switch#test platform software nif_mgr port ?

disable  shutdown port

enable   unshut port

Alternatively, use the next approach which can simulate SFP/QSFP OIR from software (for example, on Cat9400/Cat9500H/Cat9600). These are hidden commands and requires 'service internal' to be configured:

Switch#test idprom interface <...> ?

fake-insert  Fake insert

fake-remove  Fake remove

Extract Trace Archive from the System

When SVL Active switch can communicate with the SVL Standby switch then a trace archive for both switches can be generated. Use this CLI.

Switch#request platform software trace rotate all

Switch#request platform software trace archive

Unable to archive /tmp/udev_ng4k.vbd.log : file does not exist

Unable to archive /tmp/vbd_app_init.log : file does not exist

excuting cmd on chassis 1 ...

sending cmd to chassis 2 ...

Creating archive file [flash:Switch_1_RP_0_trace_archive-20190807-073924.tar.gz]

Done with creation of the archive file: [flash:Switch_1_RP_0_trace_archive-20190807-073924.tar.gz]

Switch#

Switch#dir flash: | in tar

180238 -rw- 7189863 Aug 7 2019 07:39:34 +00:00 Switch_1_RP_0_trace_archive-20190807-073924.tar.gz

Switch#

In the case of Quad SUP, you need to collect a trace archive for all the SUP separately.

request platform software trace slot switch active R0 archive

request platform software trace slot switch active R1 archive

request platform software trace slot switch standby R0 archive

request platform software trace slot switch standby R1 archive

In situations where SVL Active switch cannot communicate with the SVL Standby switch such as SVL Split, ensure to generate a trace archive on both units.

Note: In case of a crash, these trace logs are available on the system report.

Note: New CLI command to decode zipped binary trace files introduced in 16.7(1)

show log file crashinfo:tracelogs/<filename>.bin.gz internal

Note: New show tech-support CLI is available since 16.11.1

show tech-support stackwise-virtual switch [active|standby|all|#]

New LMP and SDP Counters

Support added from release 16.10.1

SDP - Stack Discovery Protocol - traffic for role negotiation and topology. The

The stack manager component is responsible for Role negotiation between stack members and elects the Active and Standby role. Stack Manager sends and receives the SDP packets over the SVL and gets the view of all the switches which are part of the Stackwise Virtual.

LMP - Link Management Protocol -

L2 traffic to maintain the SVL. The link Management protocol is a software component, which runs a hello between ends and decides if the physical link is eligible to be part of the StackWise Virtual. LMP also monitors each configured physical link while they are part of the SVL. LMP is a part of the Network Interface Manager (Nif Mgr) software process.

FSS - Front Side Stack - another name for StackWise Virtual

From a software (Nif-mgr = Network Interface Manager) perspective:

show platform software nif-mgr switch active <R0 | R1> switch-info

show platform software nif-mgr switch active <R0 | R1> counters

show platform software nif-mgr switch active <R0 | R1> counters lpn 1

show platform software nif-mgr switch active <R0 | R1> packets

set platform software nif_mgr switch active <R0 | R1> pak-cache 40 —> set the packet cache count per SVL port to 40 (default = 10)

From a hardware (FED = Forward Engine Driver) perspective:

show platform software fed switch active fss lmp-packets interface <type & #>

show platform software fed switch active fss sdp-packets

show platform software fed switch active fss counters

show platform software fed switch active fss interface-counters <interface type & #>

set platform software fed switch active F1 active fss pak-cache 40 —> set the packet cache count per SVL port to 40 (default = 10)

Check Quad SUP Redundancy Status

Quad Sup system has a max of 4 Supervisors with 2 Supervisors present per chassis. Show module provides a complete view of both the chassis and the SUP present in the System.

You can see the Slots in which SUP are present and the Supervisor which is marked as In chassis Standby update with status as provisioned.

SG_SVL_QuadSup#show module

Chassis Type: C9606R

Switch Number 1

Mod Ports Card Type                              Model          Serial No.

---+-----+--------------------------------------+--------------+--------------

1   48    48-Port 10GE / 25GE                    C9600-LC-48YL CAT2310L58W

2   24    24-Port 40GE/12-Port 100GE             C9600-LC-24C  CAT2310L4CP

3   0     Supervisor 1 Module                    C9600-SUP-1   CAT2319L302

4   0     Supervisor 1 Module                    C9600-SUP-1   CAT2319L301

5   48    48-Port 10GE / 25GE                    C9600-LC-48YL CAT2312L2G7

6   24    24-Port 40GE/12-Port 100GE             C9600-LC-24C  CAT2310L4D6

Mod MAC addresses                    Hw   Fw           Sw                 Status

---+--------------------------------+----+------------+------------------+--------

1 DC8C.371D.2300 to DC8C.371D.237F   1.0  17.3.1r[FC2] 2020-11-05_13.02_s   ok

2 DC8C.371D.2080 to DC8C.371D.20FF   1.0  17.3.1r[FC2] 2020-11-05_13.02_s   ok

3 DC8C.37CA.6500 to DC8C.37CA.657F   1.0  17.3.1r[FC2] 2020-11-05_13.02_s   ok

4 --                                 --   N/A          --                   Provisioned

5 DC8C.37A0.6880 to DC8C.37A0.68FF   1.0  17.3.1r[FC2] 2020-11-05_13.02_s   ok

6 DC8C.371D.1A80 to DC8C.371D.1AFF   1.0  17.3.1r[FC2] 2020-11-05_13.02_s   ok

Mod Redundancy Role     Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3     Standby             sso                      sso

4     InChassis-Standby   rpr                      rpr

Switch Number 2

Mod Ports Card Type Model Serial No.

---+-----+--------------------------------------+--------------+--------------

1 24 24-Port 40GE/12-Port 100GE C9600-LC-24C CAT2313L2WE

3 0 Supervisor 1 Module C9600-SUP-1 CAT2321L553

4 0 Supervisor 1 Module C9600-SUP-1 CAT2319L309

5 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2312L2C5

6 48 48-Port 10GE / 25GE C9600-LC-48YL CAT2312L2DW

Mod MAC addresses                                    Hw        Fw                Sw                            Status

---+--------------------------------+----+------------+------------------+--------

1 DC8C.37A0.C480 to DC8C.37A0.C4FF    1.0     17.3.1r[FC2]   2020-11-05_13.02_s   ok

3 DC8C.37CA.6D00 to DC8C.37CA.6D7F   1.0     17.3.1r[FC2]   2020-11-05_13.02_s   ok

4 -- -- N/A --                                                                                                              Provisioned

5 DC8C.37A0.5F80 to DC8C.37A0.5FFF     1.0     17.3.1r[FC2]   2020-11-05_13.02_s ok

6 DC8C.37A0.5C80 to DC8C.37A0.5CFF    1.0     17.3.1r[FC2]   2020-11-05_13.02_s ok

Mod Redundancy Role Operating Redundancy Mode Configured Redundancy Mode

---+-------------------+-------------------------+---------------------------

3      Active                               sso                                       sso

4      InChassis-Standby            rpr                                        rpr

Chassis 1 MAC address range: 64 addresses from 2c4f.5204.c080 to 2c4f.5204.c0bf

Chassis 2 MAC address range: 64 addresses from 2c4f.5204.bec0 to 2c4f.5204.beff

To observe per sup detailed redundancy state the next show CLI can be used ... Details provided includes Current up time and image details.

SG_SVL_QuadSup#sh redundancy rpr

My Switch Id = 2

Peer Switch Id = 1

Last switchover reason = none

Configured Redundancy Mode = sso

Operating Redundancy Mode = sso

Switch 2 Slot 3 Processor Information:

---------------------------------------------

Current Software State = ACTIVE

Uptime in current state = 18 minutes

Image Version = Cisco IOS Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Experimental Version 17.5.20201101:045441 [s2c/polaris_dev-/nobackup/suhs/latest/polaris 104]

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Sun 01-Nov-20 10:28 by suhs

BOOT = bootflash:packages.conf;

Switch 2 Slot 4 Processor Information:

---------------------------------------------

Current Software State = InChassis-Standby (Ready)

Uptime in current state = 18 minutes

Image Version =

BOOT = bootflash:packages.conf;

Switch 1 Slot 3 Processor Information:

---------------------------------------------

Current Software State = STANDBY HOT

Uptime in current state = 18 minutes

Image Version = Cisco IOS Software [Bengaluru], Catalyst L3 Switch Software (CAT9K_IOSXE), Experimental Version 17.5.20201101:045441 [s2c/polaris_dev-/nobackup/suhs/latest/polaris 104]

Copyright (c) 1986-2020 by Cisco Systems, Inc.

Compiled Sun 01-Nov-20 10:28 by suhs

BOOT = bootflash:packages.conf;

Switch 1 Slot 4 Processor Information:

---------------------------------------------

Current Software State = InChassis-Standby (Ready)

Uptime in current state = 18 minutes

Image Version =

BOOT = bootflash:packages.conf;

Other Major Enabler for Quad SUP system is to make Global Active Sup have an access to all the SUP File system , as well as reachability.

This provides the means to replicare the Configuration, Images, License, and other configured parameters across all the SUP.

Use this CLI to validate the File system availability:

SG_SVL_QuadSup#dir ?

/all List all files

/recursive List files recursively

all-filesystems List files on all filesystems

bootflash-1-0: Directory or file name

bootflash-1-1: Directory or file name

bootflash-2-1: Directory or file name

bootflash: Directory or file name

cns: Directory or file name

crashinfo-1-0: Directory or file name

crashinfo-1-1: Directory or file name

crashinfo-2-1: Directory or file name

crashinfo: Directory or file name

disk0-1-0: Directory or file name

disk0-1-1: Directory or file name

disk0-2-1: Directory or file name

disk0: Directory or file name

flash: Directory or file name

null: Directory or file name

nvram: Directory or file name

revrcsf: Directory or file name

stby-bootflash: Directory or file name

stby-crashinfo: Directory or file name

stby-disk0: Directory or file name

stby-nvram: Directory or file name

stby-rcsf: Directory or file name

system: Directory or file name

tar: Directory or file name

tmpsys: Directory or file name

usbflash0: Directory or file name

webui: Directory or file name

| Output modifiers

<cr> <cr>

SG_SVL_QuadSup#sh file systems

File Systems:

Size(b) Free(b) Type Flags Prefixes

- - opaque rw system:

- - opaque rw tmpsys:

* 11250098176 8731799552 disk rw bootflash: flash:

11250171904 7888437248 disk rw bootflash-1-0:

1651314688 0 disk rw crashinfo:

1651507200 0 disk rw crashinfo-1-0:

944993665024 896891006976 disk rw disk0:

944994516992 896892141568 disk rw disk0-1-0:

15988776960 15988768768 disk rw usbflash0:

7663022080 7542669312 disk ro webui:

- - opaque rw null:

- - opaque ro tar:

- - network rw tftp:

33554432 33483313 nvram rw nvram:

- - opaque wo syslog:

- - network rw rcp:

- - network rw http:

- - network rw ftp:

- - network rw scp:

- - network rw sftp:

- - network rw https:

- - opaque ro cns:

11250171904 6551502848 disk rw bootflash-2-1:

1651507200 0 disk rw crashinfo-2-1:

944994516992 896136118272 disk rw disk0-2-1:

11250171904 6074400768 disk rw bootflash-1-1:

1651507200 0 disk rw crashinfo-1-1:

945128734720 896416088064 disk rw disk0-1-1:

33554432 33479217 nvram rw stby-nvram:

- - nvram rw stby-rcsf:

11250098176 7888244736 disk rw stby-bootflash:

1651314688 0 disk rw stby-crashinfo:

944993665024 896891629568 disk rw stby-disk0:

- - opaque rw revrcsf:

Migrate 2 Sup SVL System to Quad SUP SVL System

Please refer to this link for the Migration steps

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9600-series-switches/215627-catalyst-9600-migration-to-quad-superv.html

Quad SUP Switchover Behavior

Quad SUP with all the four SUP populated in a system failure of Active SUP tracks a Z model for transition to New Active Supposition.

This image depicts the New Active SUP transition on the failure of the current Active SUP.

Please use CLI mentioned as part of Section 9 to check for the current SUP status and health of each sup at any point of Switchover. In addition show redundancy CLI can be used to track the Global Active / Standby redundancy transition details along with the history of switchover.

Quad SUP ISSU Behavior

Quad SUP with all the four Sup populated in a system, ISSU  to new V2 image tracks a minimal data path impact model.

One shot ISSU progression is captured in this image which starts with:

1. Copy the new V2  image to all 4 SUP and activate both the ICS with New V2 image. This results in a reload of both the ICS to boot up with New V2 image.

2. Active the Global Standby with V2, which result in the switchover to In-chassis Standby Sup associated with the Global standby followed by Old Global standby come up as ICS with V2.

3. Active the Global Active with V2 image which results in Switchover to In-chassis Standby associated with the Global Active. On successful boot up of Old Global Active as ICS with V2, completes the commit operation.

Troubleshoot Common Scenarios/Issues

StackWise Commands Do not Work

StackWise virtual commands are visible/available to configure only when the license level is configured as Network Advantage in case of cat9k switches and IPBase or IPServices in case of Cat3k switches.

Possible Cause

These issues are seen mainly when network essentials are not configured properly.

- On Cat3k, check if a license is IPBase or IPServices, use show version command.

- On Cat9K, check if a license is Network Advantage, use show version command.

- Check if MODEL_NUM rommon variable is populated correctly and matches actual device model.

Tips to Troubleshoot

If the license is right and still commands not available, could be a config sync issue in which case use these commands to collect the information that helps to debug this issue:

1. show version

2. show license summary

3. show romvar | i MODEL_NUM

Possible Solution

1. Configure the right license level based on the platform and retry.

2. If the Model number is different the mostly the SUP can be old protoboard, you can correct the model number through ROMMON.

StackWise Configuration is Missed

When the system comes up, you can see StackWise configuration missed from the running configuration.

Possible Cause

Check if the desired configuration exists in the running-config database.  Most likely it is an iosd config sync issue or the License level is configured back to non network-advantage

Tips to Troubleshoot

1. Check LICENSE LEVEL, to see if SVL is supported for that license level. Ensure license level is set to network-advantage or the previous show license summary.

2. Check running-config and startup-config show running-config and show startup-config.

If the issue is immediately after ISSU and there is a mismatch between output of startup-config and running-config then it could be an ISSU issue, please refer to the next step.

3. Check the output for the ISSU impacted CLI. show redundancy config-sync failure mcl.

If the interested configuration is missed in both running and startup-config navigate to this next step.

4. Check rommon variables, use show romvar | inc D_S.

If the interested configuration is missed or present in romvar do the next step.

5. Check show issu state detail.

If you are unable to troubleshoot then please collect the next command outputs to investigate further

1. show platform software fed switch active fss counters (3-4 Increments)

2. test platform software nif_mgr lmp member-port LPN

3. collect show tech-support stackwise-virtual

4. Collect trace archive file

Possible Solution

Configure the right license level based on the platform and retry. If the issue occurs after ISSU then mostly you have to clean up or reconfigured the configuration that is listed in MCL failures.

StackWise links/Dual Active Detection Links are Down

Possible Cause

There are many reasons for SVL/DAD links to go down. It is hard to tell just with a look at show command outputs, it needs some specific btrace logs along with show command output to carefully analyze and find the root cause.

Some of the most common ones that can cause Link down are bad SFP or bad cable or it could be a simple display issue.

Tips to Troubleshoot

1. Be clear on whats the status of the link, use command show stackwise-virtual link

2. Ensure the physical connectivity of the SVL ports and their configuration is good. Check show stackwise-virtual.

If you are unable to troubleshoot then please provide the next command outputs to investigate.

1. show platform software fed switch active fss counters (3-4 Increments)

2. test platform software nif_mgr lmp member-port LPN

3. collect show tech-support stackwise-virtual

4. Collect trace archive file.

SVL Protocol Status Does not show R

Be clear on whats the status of the link shown in the show stackwise-virtual command

Possible Cause:

1. If Protocol is in S-Suspended state then most of the times means that the link is Down.

2. If Protocol is in T-Timeout state then it means that it is not able to receive/send LMP packets.

Note: If other link is in P - Pending state then the link in T state is able to transmit LMP packets but not receive any.

3. For Timeout/Pending state important to capture Nif_mgr, Fed and also just in case stack_mgr btrace logs.

4. Check if switch recovers with shut/no shut of SVL Links ( just a data pointer and not a solution )

Tips to Troubleshoot

If you are unable to find the root cause please provide these next command outputs

1. Check SVL/DAD link status. Use show stackwise-virtual link

2. Check show platform software fed switch active/standby fss counters (3-4 Increments)

3. test platform software nif_mgr lmp member-port LPN

4. Check show platform software fed switch active/standby ifm map

7. There could be a possible issue with a specific type of xcvr used on SVL/DAD link. Verify show interface <SVL/DAD link> status & show idprom <SVL/DAD link>

8. Check show platform to ensure SUP/LC that has SVL/DAD links are in OK state.

If you are unable to troubleshoot then please provide these next command outputs for the Engineering team to investigate

1. Collect trace archive file

2. Collect show tech-support stackwise-virtual

3. show platform software fed switch active/standby fss counters (3-4 Increments)

Possible Solution:

1. Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN

2 Physical OIR of the transceiver on SVL/DAD links

SVL Link Flapping

Possible Cause

The encounter of link flap on an SV Testbed can be in multiple domains. The most common and basic information to be collected irrespective of the domain/area of the flap are listed in the next section

Tips to Troubleshoot

1. Generate trace archive.

2. Check FED channel Health.

3. Check LMP health.

4. identify ASIC/Core map which corresponds to SVL link.

5. collect show tech-support stackwise-virtual

If you are unable to find the root cause please provide these command outputs for further troubleshooting from the Engineering team.

1. test platform software nif_mgr lmp member-port <slot> <lpn port>

2. show platform hardware iomd switch 1/2 1/0 lc-portmap brief | detail

3. show tech-support stackwise-virtual

4. Collect trace archive file.

Possible Solution

- Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN

- Physical OIR of the transceiver or try to change the transceiver or cable on SVL/DAD links.

Error-disable of the SVL/DAD Links

Possible Cause

The transceiver/SFP/QSFP used on the SVL/DAD links can be faulty at the hardware layer and the xcvrs can itself generate frequent link flaps and thereby error-disable the SVL/DAD links.

Tips to Troubleshoot

1. Verify show idprom interface <SVL/DAD link>

2. Verify if any of the interfaces are indeed err-disabled on the switch, use show interfaces status err-disabled.

If you are unable to find the root cause please provide these command outputs

1. show errdisable flap-values

2. show errdisable recovery

3. show errdisable detect

4. show tech-support stackwise-virtual

5. Collect trace archive file

Possible Solution

1. Shut/ No Shut SVL LINKS: test platform software nif_mgr port disable / enable LPN

2. Physical OIR of the transceiver or try to change the transceiver or cable on SVL/DAD links.

Stack Does not Get Discovered

Expected output upon system come up would be

Waiting for 120 seconds for other switches to boot

#####################################################

Switch number is 2

All switches in the stack have been discovered. Accelerating discovery

Note: The timeout mentioned previously would differ based on the Cat9K platform/SKU

Possible Cause

Stack discovery failures can be because of multiple reasons. Listed some of them here:

1. SVL links status can NOT be UP or protocol status can not be Ready.

2. SVL links flap.

3. One of the stack switches can have crashed or gone down.

4. Accidentally pull off cable or transceiver or shut down of SVL links.

Tips to Troubleshoot

In an accelerate discovery, if the stack does not get discovered you can need to check the link status of all the SVL. It can be that none of the SVL Links were up. You can check this with the command show stackwise-virtual link

If you are unable to find the root cause please provide these command outputs along with the trace archive

1. Collect trace archive file

2. Collect show tech-support stackwise-virtual

3. show platform software fed switch active/standby fss counters (3-4 Increments)

Switch is Not in Recovery Mode

When all the SVL links are down and DAD links are configured on the SVL, the active switch must be in recovery-mode.

Switch(recovery-mode)#

Possible Cause

DAD link can have gone down because of either physical removal or shut down of the port.

Tips to Troubleshoot

1. Check if the DAD links are UP, use show stackwise-virtual dual-active-detection [pagp]

2. If DAD Links are down, check why the link is down.

5. If DAD links are up and the still switch does not move to recovery mode, you must check the IleMstStateTable bits for the particular DAD ports as a single DAD packet is sent out and it can be missed.

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. test platform software nif_mgr lmp member-port <slot> <lpn port>

2. show interface status <SVL>

3. Collect trace archive file

4. Collect show tech-support stackwise-virtual

5. show platform software fed switch active/standby fss counters (3-4 Increments)

Switch Gets to Recovery Mode before removal of all SVL Links

Possible Cause

Active switch gets into recovery mode could be several reasons. Listed next are a few of the possible causes.

1. SVL links can experience unexpected link flaps.

2. SVL links can experience unexpected remote/local link faults.

3. There could be a problem in LMP/SDP packets exchange over the SVL links.

Tips to Troubleshoot

1. If the switch goes to recovery mode just by the removal of half / less than complete SVL links  -> use the test command to shut/no shut SVL -> check if it is possible to reproduce the issue with physical OIR.

2. If it still goes to recovery mode without shut command, then first probably the root cause → Control port can not be a shift.

3. SVL links can be time out, LMP packets can not get exchanged correctly between active/standby switches.

5. Fed channel health is important to check if other packets do go through.

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show stackwise-virtual link

2. show stackwise-virtual dual-active-detection

3. show platform software fed switch active fss counters (3-4 Increments)

4. test platform software nif_mgr lmp member-port <slot> <lpn port>

5. show platform software fed switch active fss bundle

6. Collect trace archive file

7. Collect show tech-support stackwise-virtual

Ports do not get Err-Disabled in Recovery Mode

Tips to Troubleshoot

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show interface status

2. show stackwise-virtual dual-active-detection

3. show stackwise-virtual dual-active-detection pagp

4. test platform software nif_mgr lmp member-port LPN

Direct Ping from a Device Connected to Active, no MEC

Tips to Troubleshoot

If ping over active SVL does not work, can not be SVL issue, increase size of ping packets for debuggability purpose, track packet drop path (request/response) and also break stack and try the same.

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show stackwise-virtual link

2. show platform software fed switch active fss counters (3-4 Increments)

3.  test platform software nif_mgr lmp member-port LPN

4.  show platform software fed switch active ifm map

5. show tech-support stackwise-virtual

TRAFFIC DROP CHECK :

7. clear controllers ethernet-controller

8. show controllers ethernet-controller <Te/Fo interfaces if traffic path>

Direct Ping from a Device Connected to Standby, no MEC

Tips to Troubleshoot

1. if ping over standby does not work, increase the size of ping packets for debuggability purpose, track packet drop path (request/response).

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show stackwise-virtual link

2. show platform software fed switch active fss counters (3-4 Increments)

3.  test platform software nif_mgr lmp member-port LPN

4.  show platform software fed switch active fss ifm map

5. show tech-support stackwise-virtual

TRAFFIC DROP CHECK :

7.clear controllers ethernet-controller

8. show controllers ethernet-controller <Te/Fo interfaces if traffic path>

9. show interfaces counters errors

Ping over port-channel

Tips to Troubleshoot

1. To diagnose issue isolate problem to either active port-channel port or standby port-channel port.

2. Once the previous isolation is established, repeat actions for either active or standby as listed previously.

3. Ensure port-channel member ports are up on each end of port-channel

If you are unable to find the root cause or troubleshoot, please provide the previous command outputs along with the trace archive:

1. show stackwise-virtual link

2. show platform software fed switch active fss counters (3-4 Increments)

3. test platform software nif_mgr lmp member-port LPN

4. show platform software fed switch active fss ifm map

5. show tech-support stackwise-virtual

TRAFFIC DROP CHECK:

7. clear controllers ethernet-controller

8. show controllers ethernet-controller <Te/Fo interfaces if traffic path

9. show interfaces counters errors

FED crash / IOSd crash/ Stack_mgr

Tips to Troubleshoot

Please collect and provide this information:

1. Collect console logs.

2. system report and core file decode if any.

Chassis lost both active and standby

Tips to Troubleshoot

Please collect and provide this information:

1. Complete system report.

2. Links state and Protocol State of SVLs.

Active removed before stdby became ready

Tips to Troubleshoot

Please collect and provide this information:

1. Complete system report mainly

2. Links state and Protocol State of SVLs.

Traffic

Tips to Troubleshoot

1. Ensure source and destination mac addresses are set accordingly in traffic streams.

2. Ensure traffic path is in same vlan domain or trunk mode.

3. If drop is on active and traffic stream is not expected to traverse over SVL, can not be SVL issue, break stack and try.

4. if drop is on standby and packet is expected to traverse over SVL collect csv dump, ifm map.

5. Identify drop, use show controller ethernet-controller <interface>.

6. if port-channel is involved, try to isolate drop by shut down one member of port-channel to determine if drop is on active or standby, repeat same steps listed previously.

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show stackwise-virtual link

2. show platform software fed switch active fss counters (3-4 Increments)

3.  test platform software nif_mgr lmp member-port LPN

4.  show platform software fed switch active ifm map

5. show tech-support stackwise-virtual

TRAFFIC DROP CHECK :

clear controllers ethernet-controller

show controllers ethernet-controller <Te/Fo interfaces if traffic path>

show interfaces counters errors

L3 Traffic

Tips to Troubleshoot

1. Ensure arp entries are resolved and necessary routes have been added.

2. Ensure source and destination IP addresses are set correctly.

3. If drop is on active and traffic stream is not expected to traverse over SVL, can not be SVL issue, break stack and try.

4. if drop is on standby and packet is expected to traverse over SVL collect csv dump, ifm map

5. Identify drop, use show controller ethernet-controller <interface>.

6. If port-channel is involved, try to isolate drop by shut down one member of port-channel to determine if the drop is on active or standby, repeat the same steps listed previously.

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show stackwise-virtual link

2. show platform software fed switch active fss counters (3-4 Increments)

3. test platform software nif_mgr lmp member-port LPN

4. show platform software fed switch active ifm map

5. show tech-support stackwise-virtual

TRAFFIC DROP CHECK :

clear controllers ethernet-controller

show controllers ethernet-controller <Te/Fo interfaces if traffic path>

show interfaces counters errors

Traffic drop on SVL

Tips to Troubleshoot

1. Identify where traffic is dropped such as

- ingress switch on ingress interface [not related to SVL]

- ingress switch on egress SVL interface

- egress switch on ingress SVL interface

- egress switch on egress interface [not related to SVL]

2. Generate trace archive.

3. Check FED channel Health.

4. Check LMP health.

5. identify ASIC/Core map which corresponds to the SVL link.

6. Capture packets that get dropped on SVL.

FIPS on SVL

Possible Cause

There is always a chance that FIPS on the SVL key can not have been configured individually on each switch. The same FIPS key must be configured individually on both switches which are part of the SVL.

1.The FIPS_KEY stored in rommon can differ even though you configured the same auth key on both sw-1 and sw-2. This is expected behavior.

2. Verify show fips status and ensure the switch is configured for fips mode.

Tips to Troubleshoot

1. Verify FIPS mode, use show fips status.

2. Verify show fips authorization-key.

3. Verify show romvar

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show fips status

2. show fips authorization-key

3. show platform software fed switch <active|standby> fss sesa-counters

4. show stackwise-virtual link

5. show stackwise-virtual

6. Collect show tech-support stackwise-virtual

7. Collect and provide trace archive.

Secure SVL

Possible Cause

There is always a chance that a secure SVL key was not configured individually on each switch. Same Secure SVL key must be configured individually on both switches which are part of the SVL

1. The SSVL_KEY stored in rommon can differ even though you configured the same auth key on both sw-1 and sw-2. This is expected behavior.

2. Verify show secure-stackwise-virtual status and ensure its configured in SECURE-SVL mode.

Tips to Troubleshoot

1. The SSVL_KEY stored in rommon can differ even though you configured the same auth key on both sw-1 and sw-2. This is expected behavior. Use show romvar CLI

2. If there is mismatch between Switch-1 and Switch-2 in the Secure SVL Key configuration, when you bring up SVL, you could potentially see an error messages like:

"stack_mgr: Reload due to reason SESA: SESA AER Request not received, Err: 31"

3. Verify show secure-stackwise-virtual status and ensure it configured in SECURE-SVL mode.

If you are unable to find the root cause or troubleshoot please provide these command outputs along with the trace archive

1. show secure-stackwise-virtual status

2. show secure-stackwise-virtual authorization-key

3. show secure-stackwise-virtual interface <SVL_LINK>

4. Collect show tech-support stackwise-virtual

5. debug secure-stackwise-virtual

6. Collect and provide trace archive

V-Mistmatch in the Switch

Possible Cause

A different version of the software booted on active & standby chassis leads to V-Mismatch in SVL

Tips to Troubleshoot

Verify show switch and check if there is any V-Mismatch

Possible Solution

There is a feature called software auto-upgrade which is enabled by default on Cat9k platforms. This feature when enabled and detects software mismatch and pushes the software package which is present on the active to standby switch and the standby switch gets auto-upgraded to the same software version which is on the active switch.

If software auto-upgrade is disabled, enable it and then reboot the standby switch so that software auto-upgrade kicks in and both active & standby switches have the same software version used in SVL.

Software auto-upgrade works only in install mode on Cat9k platforms which is the recommended boot mode.

1. Verify software auto-upgrade is enabled on the SVL. Use the show CLI:

switch#show run all | i software

software auto-upgrade enable

Stack split with the removal of just one SVL link out of multiple SVL links

Possible Cause

This type of problem can happen when transceivers are half/partially inserted on the front panel ports, there could be a chance that IOMD process can get into busy states when read idprom of these xcvrs. This can lead to slowness in execution & completion of some of the typical show commands like show idprom <interface>, show interface status, etc on active or standby switches in SVL.

1. Verify show idprom interface <intf>. Verify on both active & standby switch interfaces idprom dump and check if there is slowness/delay in completion of this CLI.

2. Verify show interfaces <intf>. Verify on both active & standby switches interfaces and check if there is slowness/delay in completion of this CLI.

3. Also check if the CPU usage is high on the SVL, use show processes cpu sorted.

Tips to Troubleshoot

If you are unable to find the root cause please provide these command outputs along with the trace archive.

1. Check if CPU is high, use show processes cpu sorted.

2. Check if TDL subslot ping iworks fine, use test platform software tdl ping subslot <>.

Possible Solution

One of the possible causes for this issue could be half or loosely inserted transceivers on the front panel port of the switches in SVL.

Ensure that transceivers are seated/inserted properly on the front panel ports by physical inspection of the transceivers/cables.

Once that is done, verify the next commands to ensure things function well.

1. Verify show idprom interface <intf> . Verify on both active & standby switch interfaces idprom dump and check CLI gets executed without any delay in completion of this CLI.

2. Verify show interfaces <intf>. Verify on both active & standby switches interfaces and check CLI gets executed without any delay in completion of this CLI.

3. Also check if the CPU usage is not high on the SVL, use show processes cpu sorted.

If the problem still persists, please collect these CLI outputs and logs.

1. Collect show tech-support stackwise-virtual

2. Collect and provide trace archive