Click on Pictures to View

To view a larger version of an image within a post, just click on the picture you want to view :)

Friday, August 24, 2018

Cisco 3850 Software Recovery via USB

Recently I have had to recover different Cisco Catalyst 3850 switches due to failed software installs or failed auto-upgrades in the stack, or unintentionally installing the wrong code on the wrong switch model, or having corrupted or missing packages.conf directory files :-/

Some of the switches were seemingly bricked from bad code, while others were stuck at the ROMMON switch: prompt, and others still cycling in a (re)boot loop.

I was able to recover these using a USB thumb drive and following the Cisco 3850 Recovery Techniques documentation (see source references at the bottom of this post for links)

The steps I completed are as follows:

1)  First I needed a USB flash drive onto which I could store a copy of known working code version, so that I could boot the switch off the USB, and later copy the image from the USB to the switch's internal flash.  I ended up using a 128GB SanDisk Cruzer USB drive that I had in my bag, and I downloaded & copied the appropriate .bin file from Cisco to the USB.

2)  Second, I needed the switch to be at the switch: prompt.  If it wasn't already, I pressed the Mode button for about 10 seconds after powering on the switch to get it to the ROMMON mode.

3)  Next, I looked for the USB drive in the switch directory, and verified the .bin file was there.

switch: dir     usbflash0:
Directory of usbflash0:/

          1  drw-  0          SanDiskSecureAccess
          4  -rw-  16024600   RunSanDiskSecureAccess_Win.exe
          9  -rw-  328157104  cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin


125081878528 bytes available (347471872 bytes used)



4)  Then I booted the switch to the image on the USB drive, which loads into Bundle mode.

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin
Reading full image into memory..................................................
................................................................................
................................................................................
................................................................................
..........................done

    [Output Omitted]

Press RETURN to get started!

5)  Once booted in Bundle mode, I looked for the packages.conf file in the flash directory.

Switch-C3850-24XS#dir flash:
Directory of flash:/

39842  -rw-     2097152   May 1 2018 11:39:33 -05:00  nvram_config
39843  -rw-    81992100  Jul 11 2017 00:13:21 -05:00  cat3k_caa-base.SPA.03.07.04E.pkg
39844  -rw-     5667516  Jul 11 2017 00:13:22 -05:00  cat3k_caa-drivers.SPA.03.07.04E.pkg
39845  -rw-    36656136  Jul 11 2017 00:13:21 -05:00  cat3k_caa-infra.SPA.03.07.04E.pkg
39846  -rw-    46349632  Jul 11 2017 00:13:22 -05:00  cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg
39847  -rw-    28904332  Jul 11 2017 00:13:21 -05:00  cat3k_caa-platform.SPA.03.07.04E.pkg
39848  -rw-   128582464  Jul 11 2017 00:13:22 -05:00  cat3k_caa-wcm.SPA.10.3.141.0.pkg
39854  -rw-        1237  Apr 30 2018 17:09:15 -05:00  packages.conf
39850  drwx        4096  Jul 11 2017 00:26:36 -05:00  virtual-instance
71713  drwx        4096   May 1 2018 11:39:05 -05:00  dc_profile_dir
71715  drwx        4096  Jul 11 2017 00:29:39 -05:00  wnweb_store
39851  -rw-        1096   May 1 2018 11:39:28 -05:00  vlan.dat
39852  -rw-   305292424  Apr 30 2018 15:14:04 -05:00  cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
39853  -rw-        1236  Jul 11 2017 00:13:31 -05:00  packages.conf.00-
39855  -rw-    83298720  Apr 30 2018 17:09:11 -05:00  cat3k_caa-base.SPA.03.06.08E.pkg
39856  -rw-     3949504  Apr 30 2018 17:09:11 -05:00  cat3k_caa-drivers.SPA.03.06.08E.pkg
39857  -rw-    33811696  Apr 30 2018 17:09:11 -05:00  cat3k_caa-infra.SPA.03.06.08E.pkg
39858  -rw-    43109956  Apr 30 2018 17:09:11 -05:00  cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
39859  -rw-    28447888  Apr 30 2018 17:09:11 -05:00  cat3k_caa-platform.SPA.03.06.08E.pkg
39860  -rw-   112669764  Apr 30 2018 17:09:11 -05:00  cat3k_caa-wcm.SPA.10.2.180.0.pkg


3463024640 bytes total (2514513920 bytes free)

6)  I went ahead and copied and renamed the current packages.conf file in flash to packages.conf.badop
Switch-3850-24XS#copy flash:packages.conf flash:packages.conf.badop
Destination filename [packages.conf.badop]?
Copy in progress...C

1237 bytes copied in 0.010 secs (123700 bytes/sec)

 7)  Checked flash directory again to confirm the two packages files were present in flash, then deleted the original packages.conf file (also saw corrupt packages.conf.00-)

Switch-3850-24XS#dir flash:
Directory of flash:/

39842  -rw-     2097152   May 1 2018 11:39:33 -05:00  nvram_config
39843  -rw-    81992100  Jul 11 2017 00:13:21 -05:00  cat3k_caa-base.SPA.03.07.04E.pkg
39844  -rw-     5667516  Jul 11 2017 00:13:22 -05:00  cat3k_caa-drivers.SPA.03.07.04E.pkg
39845  -rw-    36656136  Jul 11 2017 00:13:21 -05:00  cat3k_caa-infra.SPA.03.07.04E.pkg
39846  -rw-    46349632  Jul 11 2017 00:13:22 -05:00  cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg
39847  -rw-    28904332  Jul 11 2017 00:13:21 -05:00  cat3k_caa-platform.SPA.03.07.04E.pkg
39848  -rw-   128582464  Jul 11 2017 00:13:22 -05:00  cat3k_caa-wcm.SPA.10.3.141.0.pkg
39854  -rw-        1237  Apr 30 2018 17:09:15 -05:00  packages.conf
39850  drwx        4096  Jul 11 2017 00:26:36 -05:00  virtual-instance
71713  drwx        4096   May 1 2018 11:39:05 -05:00  dc_profile_dir
71715  drwx        4096  Jul 11 2017 00:29:39 -05:00  wnweb_store
39851  -rw-        1096   May 1 2018 11:39:28 -05:00  vlan.dat
39852  -rw-   305292424  Apr 30 2018 15:14:04 -05:00  cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin
39853  -rw-        1236  Jul 11 2017 00:13:31 -05:00  packages.conf.00-
39855  -rw-    83298720  Apr 30 2018 17:09:11 -05:00  cat3k_caa-base.SPA.03.06.08E.pkg
39856  -rw-     3949504  Apr 30 2018 17:09:11 -05:00  cat3k_caa-drivers.SPA.03.06.08E.pkg
39857  -rw-    33811696  Apr 30 2018 17:09:11 -05:00  cat3k_caa-infra.SPA.03.06.08E.pkg
39858  -rw-    43109956  Apr 30 2018 17:09:11 -05:00  cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg
39859  -rw-    28447888  Apr 30 2018 17:09:11 -05:00  cat3k_caa-platform.SPA.03.06.08E.pkg
39860  -rw-   112669764  Apr 30 2018 17:09:11 -05:00  cat3k_caa-wcm.SPA.10.2.180.0.pkg
39849  -rw-        1237   May 1 2018 11:43:04 -05:00  packages.conf.badop


3463024640 bytes total (2514509824 bytes free)

Switch-3850-24XS#del flash:packages.conf
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
Switch-3850-24XS#

8)  Next, I expanded the bundle running on the switch to create a new packages.conf file in flash.


Switch-3850-24XS#software expand running switch 1 to flash:
Preparing expand operation ...
[1]: Expanding the running bundle
[1]: Copying package files
[1]: Package files copied

[1]: Finished expanding the running bundle



9)  After that I verified the boot variables.

Switch-3850-24XS#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Allow Dev Key = yes
Manual Boot = no

Enable Break = no

10)  Lastly, initiated a reload.

Switch-3850-24XS#reload
Reload command is being issued on Active unit, this will reload the whole stack

Proceed with reload? [confirm]

11)  After the switch reloaded, it came up as normal, and I was able to verify the version and mode of the software on the switch, followed by a software clean to remove old files of code from flash.

Switch-3850-24XS#show version

    [Output Omitted]

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 34    WS-C3850-24XS      03.07.04E         cat3k_caa-universalk9 INSTALL


Configuration register is 0x102
Switch-3850-24XS#
Switch-3850-24XS#software clean





SOURCES:

Catalyst 3850 Series Switch Upgrade, Management & Recovery Techniques:


Posted by at No comments:
Labels: , , , , , , , , ,
Subscribe to: Posts (Atom)