3850 Switch Loses Config After Reload

Thanks to my mentor James Bond for solving this conundrum.

The Problem:

Switch has running-config saved to startup-config, but the switch seems to "lose" its config when it is reloaded, displaying the Automatic Switch Configuration option and coming up to the generic Switch> prompt.  Issuing the copy start run command successfully restores the running-configuration.

The switch was essentially booting into password recovery mode after each reload, where the running-configuration is blank.

#show version confirmed at the bottom that the configuration register was set to be 0x102

The Solution:

Boot into rommon (rommon is when you interrupt the boot loader and get to the Switch prompt and run the flash_init command) and remove the boot variable that is set to ignore the startup config.

1. Boot into ROMMON mode
1. Connect to the console of the switch
2. Power off the switch
3. While holding the MODE button down, restore power to the switch
4. Keep pressing the MODE button and watch the console output
5. After approx. 10-15 seconds, the switch should end up at the ROMMON/boot loader switch: prompt
6. Release the MODE button
2. Type command flash_init
3. Type command set
4. Look for if SWITCH_IGNORE_STARTUP_CFG bit is set to 1
5. Type command unset SWITCH_IGNORE_STARTUP_CFG
6. Reload the switch

Example set output:

ABNORMAL_RESET_COUNT=0

ASIC_PCI_RESET=1

BOOT=flash:packages.conf

BOOT_LOADER_UPGRADE_DISABLE=1

BSI=0

CFG_MODEL_NUM=WS-C3850-48P-S

CLEI_CODE_NUMBER=IPM8E00ARB

CSR_PCIERST_DISCONNECTED=yes

DC_COPY=yes

D_STACK_DOMAIN_NUM=1

ECI_CODE_NUMBER=468919

LICENSE_BOOT_LEVEL=ipbasek9,all:ngwc;

MAC_ADDR=00:EB:D5:94:72:80

MANUAL_BOOT=no

MODEL_NUM=WS-C3850-48P

MODEL_REVISION_NUM=AA0

MOTHERBOARD_ASSEMBLY_NUM=73-15800-07

MOTHERBOARD_REVISION_NUM=B0

MOTHERBOARD_SERIAL_NUM=FOC202114XB

POE1_ASSEMBLY_NUM=73-16439-01

POE1_REVISION_NUM=A0

POE1_SERIAL_NUM=FOC202071F2

POE2_ASSEMBLY_NUM=73-16439-01

POE2_REVISION_NUM=A0

POE2_SERIAL_NUM=FOC202071CT

RANDOM_NUM=966545832

RECOVERY_BUNDLE=sda9:cat3k_caa-recovery.bin

STKPWR_ASSEMBLY_NUM=73-11956-08

STKPWR_REVISION_NUM=B0

STKPWR_SERIAL_NUM=FOC20216XC4

SWITCH_IGNORE_STARTUP_CFG=1                   <<<<<<<   This Boot variable means every time the 

SWITCH_NUMBER=1                                                                switch reboots to ignore the config in NVRAM

SYSTEM_SERIAL_NUM=FCW2021C1US

TAN_NUM=800-43041-02

TAN_REVISION_NUMBER=C0

TEMPLATE=advanced

TERMLINES=0

USB_ASSEMBLY_NUM=73-16576-01

USB_REVISION_NUM=A0

USB_SERIAL_NUM=FOC20212NWT

VERSION_ID=V06

unset  SWITCH_IGNORE_STARTUP_CFG=1               <<<<<<<<<<  unset the bootvar and reload

Alternatively, according to Cisco password recovery documentation, you may also be able to run the command SWITCH_IGNORE_STARTUP_CFG=0

Sources:

How to Boot 3850 in ROMMON Mode:

https://supportforums.cisco.com/t5/lan-switching-and-routing/how-to-boot-3850-in-rommon-mode/td-p/2698416

Why SW-3850 loses config after reload even though config is saved?

https://supportforums.cisco.com/t5/lan-switching-and-routing/why-sw-3850-loses-config-after-reload-even-though-config-is/m-p/2471643/highlight/true#M293620

Lost or Forgotten Password on a Switch

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/system_management/configuration_guide/b_sm_3se_3850_cg/b_sm_3se_3850_cg_chapter_01110.html#task_1021182

Recovering from a Lost or Forgotten Password

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/system_management/configuration_guide/b_sm_3se_3850_cg/b_sm_3se_3850_cg_chapter_01110.html#task_1021182

Cannot Ping Device on Different Subnet

While working on deploying the network at a new operations center in Texas, we ran into some issues regarding both security cameras and credit card reader device installations.  The security cameras and card readers are each installed and managed by other departments (ie Security dept., Help Desk, etc.)

These devices were initially plugged into ports configured by default as end-user data ports and received DHCP addresses.  However, due to the general security reasons and architectural design of the VLANs, their corresponding subnets, and network segmentation of the campus, the devices needed static addresses on purpose-specific subnets configured.  This required our network team to change switchport configs off the default DHCP vlan accordingly to allow for connectivity.

The problem here was many times the device (after static address assigned and port config updated) was only pinging from the local subnet.  Even on the Layer 3 switch, a ping was only successful when sourced from that subnet's SVI (eg #ping 192.168.100.27 source vl 200)

Turns out, when the device is only pinging on the local subnet, it was usually because of one (or both) of two simple reasons.

Reason 1 - Wrong Statically Assigned Subnet Mask

Our network was utilizing a mix of /26 & /27 networks for the same VLAN ID and IP address 3rd octet spread across various layer 3-separated closets.  The folks assigning the static address would configure either a /24 mask by habit or default, or mix up the /26 and the /27 masks, etc.  Once the mask was fixed to match the mask of the assigned subnet's local gateway, the issue was resolved.

Reason 2 - Wrong Statically Assigned Default Gateway

Again because our network was utilizing a mix of /26 & /27 networks for the same VLAN ID and 3rd octet spread across various layer 3-separated closets, the gateway was not always the same host address; so the first three octets were the same for all of these subnets, but the gateway differed in the fourth octet.  The folks assigning the static address would again, either do the standard gateway IP for a /24 network by default or habit, or confuse the correct gateway for that /26 or /27 subnet.  Once the gateway IP address was fixed, the device became reachable from outside its own subnet.