RADIUS Clients Disappearing from NPS

An issue at hand recently was that newly added RADIUS clients to an NPS server were disappearing for no apparent reason.  This was a Windows 2008 Server running the Network Policy Server.

I had added several RADIUS client entries back on February 8th, 2018, and we noticed them missing from the server on Wednesday the 22nd.  While I recognized this was unusual, I added them all back in due to not enough time to investigate the cause.
However, they were missing again by March 2nd.

I checked event logs on the NPS server, and saw login/authentication errors for networked equipment starting Tuesday the 21st, and again Tuesday the 26th after the re-adds from the 22nd.  I went ahead and enabled auditing for NPS events for successess and failures.

After some research and comparison, and having restored the NPS server to a couple different previous versions to no avail in restoring the entries, I finally found an online thread that described the issue accurately, and proposed a cause as resolution.

"...we basically discovered that when people made changes to the NPS server, they disconnected and never logged out - so...  the changes were held open in a disconnected session.  The next person to log in, makes changes, logs out as they should, then the person with the disconnected session logs back in, makes changes to the file that is already open (but not the latest one), saves the changes, thus over-writing the last changes..."

I tested this theory with a coworker, doing disconnects and logins with edits by one user, versus logoffs, logins and edits by another user.

I happened to notice during this that a teammate was in a disconnected session from Tuesday 2-26-2018 at 10:55 on the main NPS server - the last time the RADIUS ias.xml file (C:\Windows\System32\ias\ias.xml) was updated and the day the RADIUS clients disappeared (or when NPS errors started).  I assumed this person had had a disconnected session from the week before on Tuesday as well, the day the RADIUS clients first disappeared.
There were also two other RADIUS client entries added to the main server, but NOT to the secondary server, suggesting this person made modifications those days at least to the main server.

There was also a disconnected session from this teammate on the secondary server.  The secondary server RADIUS ias.xml file was last edited 2-22-18, ending with a particular entry name edit I personally made that day.  Therefore, it was likely that the next time that user would've logged on, it would've reestablished their disconnected session, and any RADIUS edits made by them would then have been made to the suspended/open file in their account, essentially overriding the actual file made by people who log off...needless to say, I ended their disconnected session to avoid this :)




SOURCES:

Windows Server 2008 R2 NPS RADIUS clients disappearing -
https://social.technet.microsoft.com/Forums/windowsserver/en-US/824b462c-515d-471a-8c46-cc1f23460386/windows-server-2008-r2-nps-radius-clients-disappearing?forum=winserverNIS

NPS Events & Audit Policy -
https://social.technet.microsoft.com/Forums/windowsserver/en-US/064f3e68-42fa-4669-aede-838e7cc7df92/nps-events-and-audit-policy?forum=winserverNAP

Cisco WLC Continually Prompts for Network Login Credentials

So the issue was several network admins were no longer able to login to our Cisco WLC with their network credentials.  One user was able to log in with their network credentials successfully.

Upon logging into the WLC using a local account, and after some research online, I verified we had the correct RADIUS server entered into the WLC's RADIUS configuration tab, and that it had the setting enabled for management.

I did notice in the logs that looked like this right after an attempted and failed login by network account "austec" via network creds:

*emWeb: Oct 12 15:02:58.126: %EMWEB-3-LOGIN_FAILED: ews_auth.c:2175 Login failed for the user:austec. Service-Type is not present or it doesn't allow READ/WRITE permission..
*emWeb: Oct 12 15:02:58.126: %AAA-5-AAA_AUTH_NETWORK_USER: aaa.c:2752 Authentication failed for network user 'austec'


I also confirmed the Network Policy Server security events in Event Viewer on the RADIUS server actually showed the user login as successful.  However when trying to log into the WLC it still would continually prompt for username and password for most folks.

Researched the "Service Type is Not Present" error, and debugging/terminal monitor equivalent on the WLC in an SSH session and came across a Cisco article called "Cisco Radius Server Authentication of Management Users on WLC Configuration Example" (see bottom of post for link) that described the issue, and explained debugging in relation to it:

Troubleshoot

There are certain circumstances when a controller authenticates management users via the ACS, the authentication finishes successfully (access-accept), and you do not see any authorization error on the controller. But, the user is prompted again for authentication.

In such cases, you cannot interpret what is wrong and why the user cannot log into the WLC by just using the debug aaa events enable command. Instead, the controller displays another prompt for authentication.

One possible reason for this is that the ACS is not configured to transmit the Service-Type attribute for that particular user or group even though the username and password are correctly configured on the ACS.

The output of the debug aaa events enable command does not indicate that a user does not have the required attributes (for this example, the Service-Type attribute) even though an access-accept is sent back from the AAA server. This example debug aaa events enable command output shows an example.

 (Cisco Controller) >debug aaa events enable

During troubleshooting, I discovered I could still log into the new virtual Cisco WLC at one of our new remote sites.  I compared the RADIUS settings, and saw they were using different servers as the default/top server.

RDP'd into the two different RADIUS servers and followed these instructions to help me compare the settings:

1. On the NPM server, in Server Manager, click Tools, and then click Network Policy Server. The NPM console opens.
2. Double-click Policies, click Network Policies, and then in the details pane double-click the policy that you want to configure.
3. In the policy Properties dialog box, click the Settings tab.
4. In policy Properties, in Settings, in RADIUS Attributes, ensure that Standard is selected.
5. In the details pane, in Attributes, the Service-Type attribute is configured with a default value of Framed.

I noticed that our Service-Type was set to "Administrative" on both boxes.  But what I did notice was that the conditions for the relevant Network Policy was different.  On the server to which the virtual WLC was authenticating successfully, the Policy conditions were specified to two different Active Directory user groups, using an "OR" condition.

On the server where authentications were breaking, it was listed one group below the other, which I inferred indicated it must be a member of both groups.

To test my theory, I added myself and another user who was unable to log into the WLC to the Telecomadmin security group in AD and waited a minute.  We then tried logging into the WLC and were both able to get in.  To confirm, I removed us from the group, and the logins broke again.

So to resolve the issue, I logged onto the RADIUS server with the implied "AND" network policy conditions.  In the NPS window, I went into the appropriate policy's properties, clicked the "Conditions" tab, edited the first entry (Domain\Network Admins Group) and added the Domain\Telecomadmin group.  It defaulted to an OR statement after Applying & OK.  Last, I removed the standalone Telecomadmin group entry.

We then tested logging into the WLC, and it worked.  So if you are having issues with this, make sure your RADIUS network policy conditions or groups didn't change.















Sources:

Cisco Forums - Watching Debug Via SSH Session to the WLC:
https://supportforums.cisco.com/t5/getting-started-with-wireless/watching-debug-via-ssh-session-to-the-wlc/td-p/2123603

Cisco WLC Debug & Show Commands:
https://www.cisco.com/c/en/us/support/docs/wireless/4400-series-wireless-lan-controllers/112064-wlc-commands.html#control

Cisco Forums - Service Type is Not Present Error When Attempting to Authenticate WLC Management Users:
https://supportforums.cisco.com/t5/aaa-identity-and-nac/service-type-is-not-present-error-when-attempting-to/td-p/1733782

Document - Generic TACACS+ for WLC Configuration:
https://supportforums.cisco.com/t5/aaa-identity-and-nac/service-type-is-not-present-error-when-attempting-to/td-p/1733782?attachment-id=15929

Cisco Cheat Sheet Common Wireless Issues:
https://www.cisco.com/c/en/us/support/docs/wireless/5508-wireless-controller/200072-Cheat-Sheet-Common-Wireless-issues.html#anc6

**Cisco Radius Server Authentication of Management Users on WLC Configuration Example:
https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wlan-security/71989-manage-wlc-users-radius.html

**Microsoft Configure Network Policies:
https://docs.microsoft.com/en-us/windows-server/networking/technologies/nps/nps-np-configure#configure-nps-for-vlans

Microsoft Connection Request Policies:
https://msdn.microsoft.com/en-us/library/cc753603(v=ws.11).aspx

RADIUS Attribute - Service-Type:
https://www.dialogic.com/webhelp/BorderNet2020/2.2.0/WebHelp/radatt_service_type.htm

WLC Authentication by ISE Server:
https://rscciew.wordpress.com/2014/06/19/wlc-authentication-by-ise-server/