Problem
Cisco 3750 switches drops or goes offline. Monitoring the switch shows random disconnects and log shows errors such as:
*Mar 1 00:01:21.528: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack
*Mar 1 00:01:23.063: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Mar 1 00:01:24.414: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
00:01:29: %SYS-5-CONFIG_I: Configured from memory by console
00:01:29: %STACKMGR-5-SWITCH_READY: Switch 1 is READY
00:01:29: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN
00:01:29: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN
00:01:29: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY
00:01:30: %SYS-5-RESTART: System restarted —
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE9, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Mar-14 22:44 by prod_rel_team
00:01:30: %SNMP-5-COLDSTART: SNMP agent on host switch-xyz is undergoing a cold start
00:01:30: %SSH-5-ENABLED: SSH 2.0 has been enabled
00:01:31: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
00:01:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
00:02:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
00:02:30: %PLATFORM-1-CRASHED: System previously crashed with the following message:
00:02:30: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE9, RELEASE SOFTWARE (fc1)
00:02:30: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport
00:02:30: %PLATFORM-1-CRASHED: Copyright (c) 1986-2014 by Cisco Systems, Inc.
00:02:30: %PLATFORM-1-CRASHED: Compiled Mon 03-Mar-14 22:44 by prod_rel_team
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: SRR0 = 0x0294E300 SRR1 = 0x00029230 SRR2 = 0x01BD0DF8 SRR3 = 0x00029230
00:02:30: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x10000000
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: CPU Register Context:
00:02:30: %PLATFORM-1-CRASHED: Vector = 0x00002000 PC = 0x01BD0DF8 MSR = 0x00029230 CR = 0x20000008
00:02:30: %PLATFORM-1-CRASHED: LR = 0x01BD0DF8 CTR = 0x02B486D8 XER = 0x6000005E
00:02:30: %PLATFORM-1-CRASHED: R0 = 0x01BD0DF8 R1 = 0x040B4528 R2 = 0x00000000 R3 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R4 = 0x0294E300 R5 = 0x00029230 R6 = 0x040A810C R7 = 0xBEEFCAFE
00:02:30: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x03C10000 R10 = 0x03B38F98 R11 = 0x03B38EC8
00:02:30: %PLATFORM-1-CRASHED: R12 = 0x001AD7B9 R13 = 0x00110000 R14 = 0x02941C28 R15 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R28 = 0x00000000 R29 = 0x02BE87AC R30 = 0xD9C2CDD9 R31 = 0x00000003
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: Stack trace:
00:02:30: %PLATFORM-1-CRASHED: PC = 0x01BD0DF8, SP = 0x040B4528
00:02:30: %PLATFORM-1-CRASHED: Frame 00: SP = 0x040B4538 PC = 0x01BD0DF8
00:02:30: %PLATFORM-1-CRASHED: Frame 01: SP = 0x040B4558 PC = 0x01BD40F4
00:02:30: %PLATFORM-1-CRASHED: Frame 02: SP = 0x040B4570 PC = 0x02941CE4
00:02:30: %PLATFORM-1-CRASHED: Frame 03: SP = 0x040B4578 PC = 0x01BD2DC8
00:02:30: %PLATFORM-1-CRASHED: Frame 04: SP = 0x00000000 PC = 0x01BC9840
Cause
Multiple suggested causes:
- slow memory leak in stacks of 3750 switches that
have IP routing disabled - Possible SNMP over polling of some OID’s that are not supported.
Solution
There are a number of recommended solutions such as those below
- IOS upgrade
- Enable ip routing and reload switch stack
- Disable SNMP polling
The solution that seemed to work for me was #3 disabling SNMP polling from our monitoring application to the affected switch.
Tested Platform
Cisco 3750 Switches
Cisco IOS C3750-IPBASEK9-M Version 12.2(55)SE9
Hits: 1607