SMA1000: Troubleshooting RDP Connection Failures Through VPN Tunnel

This article addresses Remote Desktop Protocol (RDP) connection failures that occur when a user attempts to connect to a Windows Server through a SonicWall SMA 1000 series VPN tunnel. The troubleshooting steps apply to Windows Server 2022 and Windows Server 2025, with clients running the built-in Remote Desktop Connection (mstsc.exe) or the Windows App on Windows 10 or Windows 11.

Symptoms of RDP connection failures could be:

• RDP connection times out or is refused when connected to the VPN tunnel.
• RDP works when connected directly on the LAN but fails over the VPN.
• Error messages may include "Remote Desktop can't connect to the remote computer" or "An authentication error has occurred."

RDP failures through a VPN tunnel are typically caused by one of the following: 

• Split tunneling misconfiguration — The RDP traffic (TCP 3389) is not routed through the VPN tunnel.
• SMA1000 access policy — The access policy does not permit RDP traffic to the target server or subnet.
• Windows Firewall on the server — The server firewall blocks RDP connections from the VPN subnet.
• Network Level Authentication (NLA) mismatch — The server requires NLA but the client cannot complete CredSSP negotiation over the tunnel.
• DNS resolution — The client cannot resolve the server hostname through the VPN DNS configuration.

Work through the following steps in order. Each addresses one of the root causes listed above.

1: Verify VPN Tunnel Routing

Confirm that the RDP target server's IP address or subnet is included in the Connect Tunnel or WorkPlace routing policy. In the SMA1000 AMC, navigate to Services > Settings and review the tunnel routes. The server's IP or subnet must be listed as a tunneled resource. On the client machine, run "route print" from a command prompt while connected to the VPN and confirm a route exists for the server's subnet via the VPN adapter.

2: Check SMA1000 Access Policy

In the AMC, navigate to Security Administration > Access Control. Verify that the user's realm and community have an access policy that permits TCP port 3389 (or the custom RDP port, if changed) to the target server or subnet. If no matching rule exists, add one and test again.

3: Verify Windows Firewall on the Server

On the Windows Server, open Windows Defender Firewall with Advanced Security. Confirm that the inbound rule "Remote Desktop - User Mode (TCP-In)" is enabled. Verify the rule's Scope tab includes the VPN client subnet (the SMA1000 tunnel pool range) in the "Remote IP address" list. If the firewall restricts RDP to specific subnets, the VPN pool range must be added.

4: Verify Remote Desktop Is Enabled on the Server

On Windows Server 2022/2025, open Settings > System > Remote Desktop and confirm the toggle is set to On. Alternatively, open Server Manager > Local Server and confirm Remote Desktop is set to Enabled. If Network Level Authentication (NLA) is required, ensure the client machine supports CredSSP. NLA is the recommended setting — do not disable it unless there is a specific compatibility requirement.

5: Troubleshoot NLA / CredSSP Errors

If the client receives an authentication error referencing CredSSP or NLA, confirm the following: (a) the client OS is fully patched with the latest Windows updates, (b) the Group Policy setting "Encryption Oracle Remediation" (Computer Configuration > Administrative Templates > System > Credentials Delegation) is not set to "Force Updated Clients," which can block connections to unpatched servers. If the server and client are both fully patched, this setting should be left at the default. If there is a version mismatch, update both endpoints rather than weakening the CredSSP policy.

6: Verify DNS Resolution Over the Tunnel

If connecting by hostname, confirm DNS resolution works over the VPN tunnel. Run "nslookup <server hostname>" from the client while connected. If resolution fails, check the SMA1000 DNS settings under System Configuration > DNS. Ensure the DNS server configured in the tunnel profile can resolve the internal hostname. As a test, try connecting by IP address to rule out DNS as the issue.

Important: If all steps above are verified and RDP still fails, collect a packet capture on the SMA1000 appliance and the target server to identify where the TCP 3389 connection is being dropped. Contact SonicWall Technical Support with the captures and TSR (Tech Support Report) for further analysis.