Recently I have had to recover different Cisco Catalyst 3850 switches due to failed software installs or failed auto-upgrades in the stack, or unintentionally installing the wrong code on the wrong switch model, or having corrupted or missing packages.conf directory files :-/
Some of the switches were seemingly bricked from bad code, while others were stuck at the ROMMON
switch: prompt, and others still cycling in a (re)boot loop.
I was able to recover these using a USB thumb drive and following the Cisco 3850 Recovery Techniques documentation (see source references at the bottom of this post for links)
The steps I completed are as follows:
1) First I needed a USB flash drive onto which I could store a copy of known working code version, so that I could boot the switch off the USB, and later copy the image from the USB to the switch's internal flash. I ended up using a 128GB SanDisk Cruzer USB drive that I had in my bag, and I downloaded & copied the appropriate .bin file from Cisco to the USB.
2) Second, I needed the switch to be at the
switch: prompt. If it wasn't already, I pressed the Mode button for about 10 seconds after powering on the switch to get it to the ROMMON mode.
3) Next, I looked for the USB drive in the switch directory, and verified the .bin file was there.
switch: dir
usbflash0:
Directory of usbflash0:/
1 drw- 0 SanDiskSecureAccess
4 -rw- 16024600 RunSanDiskSecureAccess_Win.exe
9 -rw- 328157104 cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin
125081878528 bytes available (347471872 bytes used)
4) Then I booted the switch to the image on the USB drive, which loads into Bundle mode.
switch: boot
usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin
Reading full image into
memory..................................................
................................................................................
................................................................................
................................................................................
..........................done
[Output Omitted]
Press RETURN to get started!
5) Once booted in Bundle mode, I looked for the packages.conf file in the flash directory.
Switch-C3850-24XS#dir flash:
Directory of flash:/
39842 -rw- 2097152 May 1 2018 11:39:33 -05:00 nvram_config
39843 -rw- 81992100 Jul 11 2017 00:13:21 -05:00
cat3k_caa-base.SPA.03.07.04E.pkg
39844 -rw- 5667516 Jul 11 2017 00:13:22 -05:00
cat3k_caa-drivers.SPA.03.07.04E.pkg
39845 -rw- 36656136 Jul 11 2017 00:13:21 -05:00
cat3k_caa-infra.SPA.03.07.04E.pkg
39846 -rw- 46349632 Jul 11 2017 00:13:22 -05:00
cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg
39847 -rw- 28904332 Jul 11 2017 00:13:21 -05:00
cat3k_caa-platform.SPA.03.07.04E.pkg
39848 -rw- 128582464 Jul 11 2017 00:13:22 -05:00
cat3k_caa-wcm.SPA.10.3.141.0.pkg
39854 -rw- 1237 Apr 30 2018 17:09:15 -05:00
packages.conf
39850 drwx 4096 Jul 11 2017 00:26:36 -05:00
virtual-instance
71713 drwx 4096 May 1 2018 11:39:05 -05:00 dc_profile_dir
71715 drwx 4096 Jul 11 2017 00:29:39 -05:00
wnweb_store
39851 -rw- 1096 May 1 2018 11:39:28 -05:00 vlan.dat
39852 -rw- 305292424 Apr 30 2018 15:14:04 -05:00
cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
39853 -rw- 1236 Jul 11 2017 00:13:31 -05:00
packages.conf.00-
39855 -rw- 83298720 Apr 30 2018 17:09:11 -05:00
cat3k_caa-base.SPA.03.06.08E.pkg
39856 -rw- 3949504 Apr 30 2018 17:09:11 -05:00
cat3k_caa-drivers.SPA.03.06.08E.pkg
39857 -rw- 33811696 Apr 30 2018 17:09:11 -05:00
cat3k_caa-infra.SPA.03.06.08E.pkg
39858 -rw- 43109956 Apr 30 2018 17:09:11 -05:00
cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
39859 -rw- 28447888 Apr 30 2018 17:09:11 -05:00
cat3k_caa-platform.SPA.03.06.08E.pkg
39860 -rw- 112669764 Apr 30 2018 17:09:11 -05:00
cat3k_caa-wcm.SPA.10.2.180.0.pkg
3463024640 bytes total (2514513920 bytes free)
6) I went ahead and copied and renamed the current packages.conf file in flash to
packages.conf.badop
Switch-3850-24XS#copy flash:packages.conf flash:packages.conf.badop
Destination filename [packages.conf.badop]?
Copy in progress...C
1237 bytes copied in 0.010
secs (123700 bytes/sec)
7) Checked flash directory again to confirm the two packages files were present in flash, then deleted the original
packages.conf file (also saw corrupt
packages.conf.00-)
Switch-3850-24XS#dir flash:
Directory of flash:/
39842 -rw- 2097152 May 1 2018 11:39:33 -05:00 nvram_config
39843 -rw- 81992100 Jul 11 2017 00:13:21 -05:00
cat3k_caa-base.SPA.03.07.04E.pkg
39844 -rw- 5667516 Jul 11 2017 00:13:22 -05:00
cat3k_caa-drivers.SPA.03.07.04E.pkg
39845 -rw- 36656136 Jul 11 2017 00:13:21 -05:00
cat3k_caa-infra.SPA.03.07.04E.pkg
39846 -rw- 46349632 Jul 11 2017 00:13:22 -05:00
cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg
39847 -rw- 28904332 Jul 11 2017 00:13:21 -05:00
cat3k_caa-platform.SPA.03.07.04E.pkg
39848 -rw- 128582464 Jul 11 2017 00:13:22 -05:00
cat3k_caa-wcm.SPA.10.3.141.0.pkg
39854 -rw- 1237 Apr 30 2018 17:09:15 -05:00
packages.conf
39850 drwx 4096 Jul 11 2017 00:26:36 -05:00
virtual-instance
71713 drwx 4096 May 1 2018 11:39:05 -05:00 dc_profile_dir
71715 drwx 4096 Jul 11 2017 00:29:39 -05:00
wnweb_store
39851 -rw- 1096 May 1 2018 11:39:28 -05:00 vlan.dat
39852 -rw- 305292424 Apr 30 2018 15:14:04 -05:00
cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
39853 -rw- 1236 Jul 11 2017 00:13:31 -05:00
packages.conf.00-
39855 -rw- 83298720 Apr 30 2018 17:09:11 -05:00
cat3k_caa-base.SPA.03.06.08E.pkg
39856 -rw- 3949504 Apr 30 2018 17:09:11 -05:00
cat3k_caa-drivers.SPA.03.06.08E.pkg
39857 -rw- 33811696 Apr 30 2018 17:09:11 -05:00
cat3k_caa-infra.SPA.03.06.08E.pkg
39858 -rw- 43109956 Apr 30 2018 17:09:11 -05:00
cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
39859 -rw- 28447888 Apr 30 2018 17:09:11 -05:00
cat3k_caa-platform.SPA.03.06.08E.pkg
39860 -rw- 112669764 Apr 30 2018 17:09:11 -05:00
cat3k_caa-wcm.SPA.10.2.180.0.pkg
39849 -rw- 1237 May 1 2018 11:43:04 -05:00 packages.conf.badop
3463024640 bytes total (2514509824 bytes free)
Switch-3850-24XS#del flash:packages.conf
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
Switch-3850-24XS#
8) Next, I expanded the bundle running on the switch to create a new
packages.conf file in flash.
Switch-3850-24XS#software expand running switch 1 to flash:
Preparing expand operation ...
[1]:
Expanding the running bundle
[1]:
Copying package files
[1]:
Package files copied
[1]:
Finished expanding the running bundle
9) After that I verified the boot variables.
Switch-3850-24XS#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist
Boot Variables on next reload:
BOOT variable = flash:packages.conf
Allow Dev Key = yes
Manual Boot = no
Enable Break = no
10) Lastly, initiated a reload.
Switch-3850-24XS#reload
Reload command is being issued on Active unit, this
will reload the whole stack
Proceed with reload? [confirm]
11) After the switch reloaded, it came up as normal, and I was able to verify the version and mode of the software on the switch, followed by a software clean to remove old files of code from flash.
Switch-3850-24XS#show version
[Output Omitted]
Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
*
1 34
WS-C3850-24XS
03.07.04E cat3k_caa-universalk9 INSTALL
Configuration register is 0x102
Switch-3850-24XS#
Switch-3850-24XS#software clean
SOURCES:
Catalyst 3850 Series Switch Upgrade, Management & Recovery Techniques: