Basic L2TP/IPSec Troubleshooting in Windows

Basic L2TP/IPSec Troubleshooting in Windows

The information in this article applies to:

Microsoft Windows 2000 Server

Microsoft Windows 2000 Advanced Server

Microsoft Windows 2000 Professional

This article was previously published under Q259335

SUMMARY

This article provides information that helps troubleshoot Layer-Two Tunneling Protocol (L2TP) and Internet Protocol Security (IPSec) in Windows.

L2TP is a standard that allows the transfer of Point to Point Protocol (PPP) traffic between different networks (described in Request for Comments [RFC] #2661). L2TP is combined with IPSec to provide both tunneling and security for Internet Protocol (IP), Internetwork Packet Exchange (IPX), and other protocol packets across any IP network.

L2TP encapsulates original packets inside a PPP frame (performing compression when possible), as well as inside a User Datagram Protocol (UDP)-type packet assigned to port 1701. Because the UDP packet format is an IP packet, L2TP automatically uses IPSec to secure the tunnel (based on the security settings in the user configuration of the L2TP tunnel). The IPSec Internet Key Exchange (IKE) protocol negotiates security for the L2TP tunnel using certificate-based authentication by default. This authentication process uses computer certificates, not user certificates, to verify that both source and destination computers trust each other. If IPSec transport security is successfully established, L2TP negotiates the tunnel (including compression and user authentication options) and performs access control based on the user identity.

The L2TP/IPSec packet structure looks like the following example (the PPP Payload contains the original IP datagram, and the italicized text represents what is encrypted with IPSec):

Microsoft Point-to-Point Encryption Protocol (MPPE), which can be used to secure the PPP payload when the Extensible Authentication Protocol Transport Layer Security (EAP-TLS) or Microsoft Challenge Handshake Authorization Protocol (MS-CHAP) is used, is negotiated by Windows when the L2TP peer (client or server) requests it.

MPPE uses the Rivest-Shamir-Adleman (RSA) RC4 stream cipher and either 40-bit, 56-bit, or 128-bit secret keys. MPPE keys are generated from the Microsoft Challenge Handshake Authentication Protocol (MS-CHAP) and EAP-TLS user-authentication process. The remote access server can be configured to require data encryption. If the remote access client cannot perform the required encryption, the connection attempt is rejected and the following error message (#742) may appear:

The remote computer does not support the required data encryption type.

IPSEC is negotiated before PPP starts; MPPE is negotiated after PPP starts. PPP runs over L2TP using IPSec. During the PPP authentication phase, a user name is sent to the Remote Access Server (RAS) component of the Virtual Private Network (VPN) server by using the configured authentication protocol (CHAP, for example). The RAS server then matches the user name and other call properties to a Remote Access Policy. Each policy has a profile, and RAS compares the conditions of the incoming call with the profile to determine whether to accept the connection request.

Considerations

If the Virtual Private Network (VPN) client is behind any network device performing Network Address Translation (NAT), the L2TP session fails because encrypted IPSec Encapsulating Security Payload (ESP) packets become corrupted. If the VPN client is on the same computer as Windows Internet Connection Sharing/Network Address Translation, the client is most likely able to establish an L2TP session because NAT does not perform any IP address or Port translation when packets originate from its own node.

You need a computer certificate with a private key, which can be found in the Local Computer Personal Certificate store.

For additional information about installing a certificate, click the article number below to view the article in the Microsoft Knowledge Base:

253498 How to Install a Certificate for Use with IP Security

If a computer certificate is not found, L2TP issues a warning that you do not have a certificate, but it does not know whether the certificate has a properly installed and associated private key for the existing certificate. Internet Key Exchange (IKE) determines this during negotiation. Start the Local Computer Certificates snap-in, double-click Certificate, and verify that General indicates "You have a private key that corresponds with this certificate." Also verify that the certificate path is complete, and that the certificate is valid.

The client must have a machine certificate whose root certificate authority is the same as the certificate on the gateway certificate. The reason for the certificate failure is noted by IKE in the security log event entry. For additional information, click the article number below to view the article in the Microsoft Knowledge Base:

257225 Basic IPSec Troubleshooting in Windows 2000

For additional information about what kind of machine certificate works properly, click the article number below to view the article in the Microsoft Knowledge Base:

249125 Using Certificates for Windows 2000 and Cisco IOS Interoperation

Both sides must be able to process the certificate validation successfully. If certificate authentication is successful, an entry in the security log notes the successful event of an IPSec main mode SA establish (logon/logoff).

IKE failure to negotiate: You can verify whether IPSec is succeeding by running Ipsecmon.exe (as local admin) with options set to refresh at one-second intervals. If you see the IPSec SA appear, it indicates that IPSec succeeded, and you may conclude that L2TP is the source of the problem. Use the netdiag /test:ipsec /v /debug command to see the details of IPSec policy (you cannot see the whole policy if a domain administrator has set policy on your local computer).

For additional information about Ipsecmon.exe and the Netdiag command, click the article number below to view the article in the Microsoft Knowledge Base:

257225 Basic IPSec Troubleshooting in Windows 2000

Also note the following:

IKE timeout: IKE may time out during the initial negotiation request if routers in front of the VPN server do not allow UDP port 500 through. It also times out if the VPN server does not have appropriate IPSec policy configured, which usually means that the RRAS server does not have L2TP ports enabled, or that a manual IPSec policy setting is misconfigured. When IKE times out, the audit log shows that peer failed to reply, and that a network capture trace shows ISAKMP UDP packets initiating only from your client. If configured specifically for L2TP, the VPN client responds with the following error message:

The security negotiation timed out.

If configured with Automatic, it attempts the task again using the next protocol on its list, namely PPTP.

IKE Failure: Failure to negotiate IKE and the reasons for the failure are recorded in the security log if you enable Security Settings, Logon/Logoff failure audits. IKE can fail because certificate credentials do not work, or because there is an IPSec policy configuration problem if they set manual IPSec policy.

For additional information about automatic policy, click the article number below to view the article in the Microsoft Knowledge Base:

248750 Description of IPSec Policy created for L2TP/IPSec

Success of IKE negotiation is noted in the audit log. For the entire IPSec security negotiation to be successful, you need both a main mode SA establish and a quick mode SA establish for UDP port 1701.

If one of the following symptoms occurs, IPSec is not causing the problem:

The audit log shows successful main mode SA establish and successful quick mode SA establish.

The network capture trace shows ESP traffic originating from the client or server.

Ipsecmon.exe shows an IPSec SA.

NOTE: There are always two IPSec SAs established: one for each direction, each with its own Security Parameter Index (SPI); however, Ipsecmon.exe shows only the outbound SA.

ESP blocked: When NAT is in front of the client or the routers are in front of the VPN, the server may not allow protocol 50, ESP, through. Outbound ESP traffic with the SPI number appear, but inbound ESP packets from the gateway with a different SPI number do not appear.

ESP modified: If NAT, or perhaps a faulty switch or other network node, is modifying or corrupting packets anywhere on the path, they are dropped by the IPSec driver with the Event 4285 "Failed to authenticate hash" appearing in the system log of the receiving system. Packets may also be corrupted by a Network Interface with IPSec Offload capabilities. To determine whether an interface has this capability, use the following command:

netsh int ip show offload

If the IPSec offload capability of a NIC is the suspected cause, start a Network Monitor capture and Ipsecmon.exe to analyze each connection attempt and verify the "Confidential Bytes Received" counter in Ipsecmon to determine whether packets are being lost on receive. You can also set the HKLM\System\CurrentControlSet\Services\IPSEC\EnableOffload DWORD registry value to 0. If the connection then succeeds, the issue is offload-related. Another troubleshooting alternative is to disable the IPSec automatic policy.For additional information about disabling the automatic IPSec policy, click the article number below to view the article in the Microsoft Knowledge Base:

240262 How to Configure a L2TP/IPSec Connection Using a Pre-shared Key

If the IPSec Policy Agent was stopped by using the Services snap-in or the net stop policyagent command, the L2TP automatic IPSec policy configuration is lost. For VPN clients, the policy is automatically plumbed when the client connectoid is started. Make sure that the IPSec policy agent service is started and running before you launch the client connectoid. After you click Connect and the connection attempt is in progress, you can use the netdiag /test:ipsec /v /debug command to see IPSec statistics and active filters (if you do not have domain admin privilege you cannot use the /debug switch). For additional information about other possible issues, click the article number below to view the article in the Microsoft Knowledge Base:

247231 Event ID 20111 When Establishing a L2TP/IPSec Connection

MORE INFORMATION

For more information about VPN, see "Microsoft Privacy Protected Network Access: Virtual Private Networking and Intranet Security" at the following location:

http://www.microsoft.com/windows2000/techinfo/howitworks/communications/remoteaccess/nwpriv.asp

For additional information about IPSec troubleshooting, click the article number below to view the article in the Microsoft Knowledge Base:

257225 Basic IPSec Troubleshooting in Windows 2000