Cisco 9120 Access Points Not Joining Catalyst WLC 9800

 So a migration from good old Cisco AirOS 5508 redundant wireless LAN controllers to the brand spanking new (yikes!) Catalyst 9800-40 chassis in an SSO design (HA basically).

In running Amsterdam 17.3.4, we were rapidly replacing our nearly EOL 2602s, 2702s, 3702s, 1572s and even 2802s as these will lose support on the 9800 controller with the next major release version of firmware.

However we noticed that some 9120s were unexpectedly not joining the WLC as we got new ones in.  There were all different versions from the factory - this version (VID) can be confirmed on the label on the access point itself, will be small print in the upper right corner - V01, V03, V05, V07 and the like.  You can also check with show inventory command if you have CLI access.

We had gotten a box of V07 and after hanging all of them was when we realized none of them would join which was totally unexpected, hence finding the field notice 72424.

It turned out there was a bug reported and in the Cisco tool where certain versions of APs and V07 APs would not join if they joined after a certain date in December 2022 or January 2023.

Of course, however, after doing an emergency code upgrade to the recommended 17.3.6 version, all of the 9120s then happily joined, and we lost all remaining 3702s and 1572s to stuck in Downloading and or failing to join due to a DTLS - another bug with a workaround.

Workaround (listed to be performed before upgrading to 17.3.6...lol) is as follows:

disable NTP on the controller > change date on WLC to recent enough date before Dec. 4, 2022 > wait for APs to finish downloading and able to join the controller > reboot any APs failing to join after upgrade completes > enable NTP again once all APs are joined again.

I am still in the process of testing if this workaround, listed to be done before you upgrade, on a notice for issues after upgrade, applies or not.  Otherwise looks like in the second Field Notice there might be additional AP service packs available for 17.3.6 (or relevant/related others with similar problem) to resolve that issue if already installed.

You'll need a Cisco account to view in the bug tool links I believe, FYI.

Sources:

https://community.cisco.com/t5/wireless/9130axi-access-point-not-able-to-join-9800-40-wlc/td-p/4698344

https://community.cisco.com/t5/wireless/new-aps-manufactured-after-sept-2022-can-t-join-older-codes/td-p/4675090

https://www.cisco.com/c/en/us/support/docs/field-notices/724/fn72424.html

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwd80290

https://www.cisco.com/c/en/us/support/docs/wireless/aironet-700-series-access-points/218447-ios-ap-image-download-fails-due-to-expir.html

Cisco 3850 Software Recovery via USB

Recently I have had to recover different Cisco Catalyst 3850 switches due to failed software installs or failed auto-upgrades in the stack, or unintentionally installing the wrong code on the wrong switch model, or having corrupted or missing packages.conf directory files :-/

Some of the switches were seemingly bricked from bad code, while others were stuck at the ROMMON switch: prompt, and others still cycling in a (re)boot loop.

I was able to recover these using a USB thumb drive and following the Cisco 3850 Recovery Techniques documentation (see source references at the bottom of this post for links)

The steps I completed are as follows:

1)  First I needed a USB flash drive onto which I could store a copy of known working code version, so that I could boot the switch off the USB, and later copy the image from the USB to the switch's internal flash.  I ended up using a 128GB SanDisk Cruzer USB drive that I had in my bag, and I downloaded & copied the appropriate .bin file from Cisco to the USB.

2)  Second, I needed the switch to be at the switch: prompt.  If it wasn't already, I pressed the Mode button for about 10 seconds after powering on the switch to get it to the ROMMON mode.

3)  Next, I looked for the USB drive in the switch directory, and verified the .bin file was there.

switch: dir     usbflash0:

Directory of usbflash0:/

          1  drw-  0          SanDiskSecureAccess

          4  -rw-  16024600   RunSanDiskSecureAccess_Win.exe

          9  -rw-  328157104  cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin

125081878528 bytes available (347471872 bytes used)

4)  Then I booted the switch to the image on the USB drive, which loads into Bundle mode.

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.07.04.E.152-3.E4.bin

Reading full image into memory..................................................

................................................................................

................................................................................

................................................................................

..........................done

    [Output Omitted]

Press RETURN to get started!

5)  Once booted in Bundle mode, I looked for the packages.conf file in the flash directory.

Switch-C3850-24XS#dir flash:

Directory of flash:/

39842  -rw-     2097152   May 1 2018 11:39:33 -05:00  nvram_config

39843  -rw-    81992100  Jul 11 2017 00:13:21 -05:00  cat3k_caa-base.SPA.03.07.04E.pkg

39844  -rw-     5667516  Jul 11 2017 00:13:22 -05:00  cat3k_caa-drivers.SPA.03.07.04E.pkg

39845  -rw-    36656136  Jul 11 2017 00:13:21 -05:00  cat3k_caa-infra.SPA.03.07.04E.pkg

39846  -rw-    46349632  Jul 11 2017 00:13:22 -05:00  cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg

39847  -rw-    28904332  Jul 11 2017 00:13:21 -05:00  cat3k_caa-platform.SPA.03.07.04E.pkg

39848  -rw-   128582464  Jul 11 2017 00:13:22 -05:00  cat3k_caa-wcm.SPA.10.3.141.0.pkg

39854  -rw-        1237  Apr 30 2018 17:09:15 -05:00  packages.conf

39850  drwx        4096  Jul 11 2017 00:26:36 -05:00  virtual-instance

71713  drwx        4096   May 1 2018 11:39:05 -05:00  dc_profile_dir

71715  drwx        4096  Jul 11 2017 00:29:39 -05:00  wnweb_store

39851  -rw-        1096   May 1 2018 11:39:28 -05:00  vlan.dat

39852  -rw-   305292424  Apr 30 2018 15:14:04 -05:00  cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin

39853  -rw-        1236  Jul 11 2017 00:13:31 -05:00  packages.conf.00-

39855  -rw-    83298720  Apr 30 2018 17:09:11 -05:00  cat3k_caa-base.SPA.03.06.08E.pkg

39856  -rw-     3949504  Apr 30 2018 17:09:11 -05:00  cat3k_caa-drivers.SPA.03.06.08E.pkg

39857  -rw-    33811696  Apr 30 2018 17:09:11 -05:00  cat3k_caa-infra.SPA.03.06.08E.pkg

39858  -rw-    43109956  Apr 30 2018 17:09:11 -05:00  cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg

39859  -rw-    28447888  Apr 30 2018 17:09:11 -05:00  cat3k_caa-platform.SPA.03.06.08E.pkg

39860  -rw-   112669764  Apr 30 2018 17:09:11 -05:00  cat3k_caa-wcm.SPA.10.2.180.0.pkg

3463024640 bytes total (2514513920 bytes free)

6)  I went ahead and copied and renamed the current packages.conf file in flash to packages.conf.badop

Switch-3850-24XS#copy flash:packages.conf flash:packages.conf.badop

Destination filename [packages.conf.badop]?

Copy in progress...C

1237 bytes copied in 0.010 secs (123700 bytes/sec)

7)  Checked flash directory again to confirm the two packages files were present in flash, then deleted the original packages.conf file (also saw corrupt packages.conf.00-)

Switch-3850-24XS#dir flash:

Directory of flash:/

39842  -rw-     2097152   May 1 2018 11:39:33 -05:00  nvram_config

39843  -rw-    81992100  Jul 11 2017 00:13:21 -05:00  cat3k_caa-base.SPA.03.07.04E.pkg

39844  -rw-     5667516  Jul 11 2017 00:13:22 -05:00  cat3k_caa-drivers.SPA.03.07.04E.pkg

39845  -rw-    36656136  Jul 11 2017 00:13:21 -05:00  cat3k_caa-infra.SPA.03.07.04E.pkg

39846  -rw-    46349632  Jul 11 2017 00:13:22 -05:00  cat3k_caa-iosd-universalk9.SPA.152-3.E4.pkg

39847  -rw-    28904332  Jul 11 2017 00:13:21 -05:00  cat3k_caa-platform.SPA.03.07.04E.pkg

39848  -rw-   128582464  Jul 11 2017 00:13:22 -05:00  cat3k_caa-wcm.SPA.10.3.141.0.pkg

39854  -rw-        1237  Apr 30 2018 17:09:15 -05:00  packages.conf

39850  drwx        4096  Jul 11 2017 00:26:36 -05:00  virtual-instance

71713  drwx        4096   May 1 2018 11:39:05 -05:00  dc_profile_dir

71715  drwx        4096  Jul 11 2017 00:29:39 -05:00  wnweb_store

39851  -rw-        1096   May 1 2018 11:39:28 -05:00  vlan.dat

39852  -rw-   305292424  Apr 30 2018 15:14:04 -05:00  cat3k_caa-universalk9.SPA.03.06.08.E.152-2.E8.bin

39853  -rw-        1236  Jul 11 2017 00:13:31 -05:00  packages.conf.00-

39855  -rw-    83298720  Apr 30 2018 17:09:11 -05:00  cat3k_caa-base.SPA.03.06.08E.pkg

39856  -rw-     3949504  Apr 30 2018 17:09:11 -05:00  cat3k_caa-drivers.SPA.03.06.08E.pkg

39857  -rw-    33811696  Apr 30 2018 17:09:11 -05:00  cat3k_caa-infra.SPA.03.06.08E.pkg

39858  -rw-    43109956  Apr 30 2018 17:09:11 -05:00  cat3k_caa-iosd-universalk9.SPA.152-2.E8.pkg

39859  -rw-    28447888  Apr 30 2018 17:09:11 -05:00  cat3k_caa-platform.SPA.03.06.08E.pkg

39860  -rw-   112669764  Apr 30 2018 17:09:11 -05:00  cat3k_caa-wcm.SPA.10.2.180.0.pkg

39849  -rw-        1237   May 1 2018 11:43:04 -05:00  packages.conf.badop

3463024640 bytes total (2514509824 bytes free)

Switch-3850-24XS#del flash:packages.conf

Delete filename [packages.conf]?

Delete flash:/packages.conf? [confirm]

Switch-3850-24XS#

8)  Next, I expanded the bundle running on the switch to create a new packages.conf file in flash.

Switch-3850-24XS#software expand running switch 1 to flash:

Preparing expand operation ...

[1]: Expanding the running bundle

[1]: Copying package files

[1]: Package files copied

[1]: Finished expanding the running bundle

9)  After that I verified the boot variables.

Switch-3850-24XS#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable does not exist

Boot Variables on next reload:

BOOT variable = flash:packages.conf

Allow Dev Key = yes

Manual Boot = no

Enable Break = no

10)  Lastly, initiated a reload.

Switch-3850-24XS#reload

Reload command is being issued on Active unit, this will reload the whole stack

Proceed with reload? [confirm]

11)  After the switch reloaded, it came up as normal, and I was able to verify the version and mode of the software on the switch, followed by a software clean to remove old files of code from flash.

Switch-3850-24XS#show version

    [Output Omitted]

Switch Ports Model              SW Version        SW Image              Mode

------ ----- -----              ----------        ----------            ----

*    1 34    WS-C3850-24XS      03.07.04E         cat3k_caa-universalk9 INSTALL

Configuration register is 0x102

Switch-3850-24XS#

Switch-3850-24XS#software clean

SOURCES:

Catalyst 3850 Series Switch Upgrade, Management & Recovery Techniques:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html#anc16



How to Boot 3850 in ROMMON Mode:
https://supportforums.cisco.com/t5/lan-switching-and-routing/how-to-boot-3850-in-rommon-mode/td-p/2698416