So the issue was several network admins were no longer able to login to our Cisco WLC with their network credentials. One user was able to log in with their network credentials successfully.
Upon logging into the WLC using a local account, and after some research online, I verified we had the correct RADIUS server entered into the WLC's RADIUS configuration tab, and that it had the setting enabled for management.
I did notice in the logs that looked like this right after an attempted and failed login by network account "austec" via network creds:
*emWeb: Oct 12 15:02:58.126: %EMWEB-3-LOGIN_FAILED: ews_auth.c:2175 Login failed for the user:austec. Service-Type is not present or it doesn't allow READ/WRITE permission..
*emWeb: Oct 12 15:02:58.126: %AAA-5-AAA_AUTH_NETWORK_USER: aaa.c:2752 Authentication failed for network user 'austec'
I also confirmed the Network Policy Server security events in Event Viewer on the RADIUS server actually showed the user login as successful. However when trying to log into the WLC it still would continually prompt for username and password for most folks.
Researched the "Service Type is Not Present" error, and debugging/terminal monitor equivalent on the WLC in an SSH session and came across a Cisco article called "Cisco Radius Server Authentication of Management Users on WLC Configuration Example" (see bottom of post for link) that described the issue, and explained debugging in relation to it:
There are certain circumstances when a controller authenticates management users via the ACS, the authentication finishes successfully (access-accept), and you do not see any authorization error on the controller. But, the user is prompted again for authentication.
In such cases, you cannot interpret what is wrong and why the user cannot log into the WLC by just using the debug aaa events enable command. Instead, the controller displays another prompt for authentication.
One possible reason for this is that the ACS is not configured to transmit the Service-Type attribute for that particular user or group even though the username and password are correctly configured on the ACS.
The output of the debug aaa events enable command does not indicate that a user does not have the required attributes (for this example, the Service-Type attribute) even though an access-accept is sent back from the AAA server. This example debug aaa events enable command output shows an example.
(Cisco Controller) >debug aaa events enable
During troubleshooting, I discovered I could still log into the new virtual Cisco WLC at one of our new remote sites. I compared the RADIUS settings, and saw they were using different servers as the default/top server.
RDP'd into the two different RADIUS servers and followed these instructions to help me compare the settings:
On the NPM server, in Server Manager, click Tools, and then click Network Policy Server. The NPM console opens.
Double-click Policies, click Network Policies, and then in the details pane double-click the policy that you want to configure.
In the policy Properties dialog box, click the Settings tab.
In policy Properties, in Settings, in RADIUS Attributes, ensure that Standard is selected.
In the details pane, in Attributes, the Service-Type attribute is configured with a default value of Framed.
I noticed that our Service-Type was set to "Administrative" on both boxes. But what I did notice was that the conditions for the relevant Network Policy was different. On the server to which the virtual WLC was authenticating successfully, the Policy conditions were specified to two different Active Directory user groups, using an "OR" condition.
On the server where authentications were breaking, it was listed one group below the other, which I inferred indicated it must be a member of both groups.
To test my theory, I added myself and another user who was unable to log into the WLC to the Telecomadmin security group in AD and waited a minute. We then tried logging into the WLC and were both able to get in. To confirm, I removed us from the group, and the logins broke again.
So to resolve the issue, I logged onto the RADIUS server with the implied "AND" network policy conditions. In the NPS window, I went into the appropriate policy's properties, clicked the "Conditions" tab, edited the first entry (Domain\Network Admins Group) and added the Domain\Telecomadmin group. It defaulted to an OR statement after Applying & OK. Last, I removed the standalone Telecomadmin group entry.
We then tested logging into the WLC, and it worked. So if you are having issues with this, make sure your RADIUS network policy conditions or groups didn't change.
Sources:
Cisco Forums - Watching Debug Via SSH Session to the WLC:
https://supportforums.cisco.com/t5/getting-started-with-wireless/watching-debug-via-ssh-session-to-the-wlc/td-p/2123603
Cisco WLC Debug & Show Commands:
https://www.cisco.com/c/en/us/support/docs/wireless/4400-series-wireless-lan-controllers/112064-wlc-commands.html#control
Cisco Forums - Service Type is Not Present Error When Attempting to Authenticate WLC Management Users:
https://supportforums.cisco.com/t5/aaa-identity-and-nac/service-type-is-not-present-error-when-attempting-to/td-p/1733782
Document - Generic TACACS+ for WLC Configuration:
https://supportforums.cisco.com/t5/aaa-identity-and-nac/service-type-is-not-present-error-when-attempting-to/td-p/1733782?attachment-id=15929
Cisco Cheat Sheet Common Wireless Issues:
https://www.cisco.com/c/en/us/support/docs/wireless/5508-wireless-controller/200072-Cheat-Sheet-Common-Wireless-issues.html#anc6
**Cisco Radius Server Authentication of Management Users on WLC Configuration Example:
https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wlan-security/71989-manage-wlc-users-radius.html
**Microsoft Configure Network Policies:
https://docs.microsoft.com/en-us/windows-server/networking/technologies/nps/nps-np-configure#configure-nps-for-vlans
Microsoft Connection Request Policies:
https://msdn.microsoft.com/en-us/library/cc753603(v=ws.11).aspx
RADIUS Attribute - Service-Type:
https://www.dialogic.com/webhelp/BorderNet2020/2.2.0/WebHelp/radatt_service_type.htm
WLC Authentication by ISE Server:
https://rscciew.wordpress.com/2014/06/19/wlc-authentication-by-ise-server/
