How to Configure L2TP/IPSec VPNs with Certificates while Using OS X

10/14/2021 1,075 People found this article helpful 104,306 Views

Description

This article details how to setup an IPSec or L2TP Connection to the SonicWall while using Certificates as an Authentication Method. This configuration is specifically done using OS X but many of the steps are similar regardless of Operating System.

Resolution

Resolution for SonicOS 6.5

This release includes significant user interface changes and many new features that are different from the SonicOS 6.2 and earlier firmware. The below resolution is for customers using SonicOS 6.5 firmware.

Feature/Application

Using digital certificates for authentication instead of preshared keys in a VPN configuration is considered more secure. In SonicWall UTM devices, digital certificates are one way of authenticating two peer devices to establish an IPsec VPN tunnel. The other is IKE using preshared key.

This KB article describes the method to configure SonicWall WAN GroupVPN and Mac OS X L2TP/IPsec clients to use digital certificates for authentication before establishing an L2TP/IPsec VPN tunnel.

Note

The Client configuration described here is for a Mac OS X 10.8.2 (Mountain Lion). However, the configuration would be similar in other Mac OS X versions.
The Names Server and Client Certificates are used in order to distinguish between the Certificates used in the SonicWall (Server) and the Mac OS X L2TP/IPsec Client (Client).
The Client Certificate used here has a Key Size of 1024 Bits. Although there is no official confirmation of a limitation from Apple, a Certificate with a larger Key Size failed during our testing.

Procedure

Before we begin the configuration process, the following requirements must be fulfilled:

The Certificates used in this configuration must be either obtained from a third party Certificate Authority (CA), like Verisign, or from a private CA like Microsoft CA or OpenSSL. Self-signed certificates are not supported.
The Certificates' Key Usage section must contain at least Digital Signature. If the certificates are to be obtained from a Microsoft CA, select either of these templates: IPSec (Offline Request), User, Administrator. If using OpenSSL, make sure the CA config file has at least Digital Signature under the Key Usage section.
The Server Certificate (certificate imported into the SonicWall) must have a Subject Alternative Name and must contain DNS: <FQDN> or <IP address>.

SonicWall Configuration

To obtain the Client and Server Certificates described in this configuration, refer this KB article: UTM: How to obtain certificates for VPN connections (Site to Site, GVC, L2TP) from a Windows Certificate server

Once the certificate has been imported, either in the method described in the above link or by importing a PKCS#12 (.pfx or .p12) format certificate, the Manage | Appliance | Certificates page must be similar to the following.

NOTE: Note the Subject Alternative Name field set with Domain Name.

WAN GroupVPN Configuration

Login to the SonicWall management GUI
Navigate to the Manage | VPN | Base Settings page.
Click the Enable VPN check box at the top of the page and the Enable check box of WAN GroupVPN.
Click on Accept at the top to save the changes.
Click on the configure button under WAN GroupVPN to open the VPN Policy window.
Select Authentication Method as IKE using 3rd Party Certificates.
Select the signed certificate, imported into the SonicWall earlier, under Gateway Certificate.

Peer Certificates: When setting Peer ID Type, the administrator may choose from Distinguished Name, Email Address or Domain Name. The IDs must be of the certificate to be used (later in this article) in the L2TP clients.
- Peer ID Type: Domain Name - In this example, this is the attribute set at the time of obtaining the user certificate and found under the Subject Alternative Name (SAN) field of the certificate. Note: The server must be configured to accept SANs. For Microsoft CAs, refer this article: How to configure a CA to accept a SAN attribute from a certificate request
- Peer ID Type: Email ID - This must be attribute of san:email=<email@address.com> set at the time of obtaining the user certificate and found under the Subject Alternative Name field of the certificate. For example, the string *@hal-2010.local would allow anyone with an email address ending in hal-2010.local to have access.
- Peer ID Type: Distinguished Name - DN is a specific reference to a particular certificate. DN can be found in the Subject field of a certificate. The DN can also be found using the following OpenSSL command, openssl x509 -in l2tp.crt -noout -subject, where l2tp.crt is the name of the certificate. A portion of a DN (RDN) can also be used. This would be helpful in certificates without a SAN of domain name or email address. For example, a certificate with a DN of, /C=IN/ST=KA/O=SonicWall Inc./CN=l2tp-mac.kb-soniclab.local/emailAddress=admin@kb-soniclab.local, can be referenced in the DN field with the values of, CN=l2tp-mac.kb-soniclab.local or using wildcards, CN=*.kb-soniclab.local.

L2TP Server Configuration

For a Step-by-Step Walkthrough of Configuring the L2TP Server Role on SonicOS follow this KB.

Mac OS X Configuration

1. Login to the Mac.

2. Navigate to the Applications | Utilities Folder.

3. Click on Keychain Access.

4. In the Keychains window, navigate to System.

5. From the File Menu, click on Import Items.

6. Select the Client Certificate obtained earlier.

7. Enter your computer's Administrator Username and Password

8. Enter the Certificate Password when prompted and finish importing the Certificate.

Now that the client certificate has been imported, you must allow applications access to its Private Key. It is recommended to allow just Racoon, the IKE (ISAKMP/Oakley) key management daemon, but for simplicity we have shown allowing all applications access to the private key.

1. In the Keychains window, under Category on the left, navigate to Certificates.

2. Click on the triangle to the left of the certificate to display the Private Key of this Certificate.

3. Double click on the Private Key.

4. In the pop-up window, click Access Control at the top.

5. Click Allow all applications to access this item.

6. Click Save Changes

7. Enter your computer admin password when prompted

8. When prompted with Do you want to allow access to this item? click Allow.

9. In the Certificates category, click on the triangle to hide the private key. Double click on the certificate.

10. In the new pop-up window, click the triangle to the left of Trust to expand it.

11. Select Always Trust in the drop down menu labelled When using this Certificate.

12. Close the pop-up window.

Now that you have successfully imported the client certificate and allowed access to its Private Key, you must now import its CA certificate. The client certificate will not be validated until you import its CA certificate.

1. From the File menu, click on Import Items.

2. Select the CA Certificate of the Client Certificate.

3. Select Always Trust when prompted with Do you want to trust......

4. Enter your computer's administrator username and password to finish importing the certificate.

5. Close the Keychain Access application.

L2TP/IPsec Client Configuration

1. Navigate to System Preferences | Network.

2. Click on the plus (+) symbol in the lower left.

3. In the pop-up window, select VPN under Interface and enter a friendly name under Service Name.

4. Click on Create.

5. Select the newly created interface.

6. Set Configuration to Default.

7 . Under Server Address, enter the FQDN of the SonicWall IP address. This must match the Subject Alternative Name of the Server Certificate in the SonicWall.

8. Under Account Name, enter a Username of a local or LDAP User who is authorized to establish L2TP/IPsec VPN connections.

9. Click on Authentication Settings.

10. Enter the password of the user under Password

11. Under Machine Authentication, select radio button Certificate.

12. Click on Select to bring up the Choose An Identity window.

13. Select the certificate imported earlier from the list.

14. Click on Continue and then click OK.

15. Click on Apply to save the settings.

16. Click on Connect.

Troubleshooting

In such configuration, the most common cause of failure could be due to Certificates. Please check the following:

If the certificates are signed by different CAs, make sure the CA Certificate/s of the Peer is imported into the Client and Server.
Make sure the Subject Alternative Name (SAN) in the Server Certificate matches the Server Address in the L2TP/IPSec Client Configuration. If the SAN is FQDN, and does not have a corresponding DNS A Record, edit the hosts file under /private/etc and map the FQDN to the WAN IP Address of the SonicWall.
Make sure the Key Usage section of the Certificates has the appropriate entry authorized to establish IPsec VPN connections.
In our testing a Certificate with a Key Size larger than 1024 bits failed at IKE Main Mode Message # 5 from the client. During IKE using Main Mode with Certificate authentication, Message # 5 contains both a Signature and the Certificate. In other words, if a certificate with a public key size above 1024 bits is used, with extraneous info in the Subject field, the resulting packet would also be large. Therefore, we recommend creating user certificates with keys of 1024 bits and with minimal information in the Subject field.

For further troubleshooting, refer the Logs of the SonicWall and the Mac.

SonicWall L2TP/IPsec connection logs can be found under the following categories:

VPN IKE (Priority - Debug )
L2TP Server (Priority - Info)

In the Mac, L2TP/IPsec related logs can be found in the following files:

/var/log/racoon.log
/var/log/system.log
/var/log/ppp.log

Resolution for SonicOS 6.2 and Below

The below resolution is for customers using SonicOS 6.2 and earlier firmware. For firewalls that are generation 6 and newer we suggest to upgrade to the latest general release of SonicOS 6.5 firmware.

Feature/Application

Using digital certificates for authentication instead of preshared keys in a VPN configuration is considered more secure. In SonicWall UTM devices, digital certificates are one way of authenticating two peer devices to establish an IPsec VPN tunnel. The other is IKE using preshared key.

This KB article describes the method to configure SonicWall WAN GroupVPN and Mac OS X L2TP/IPsec clients to use digital certificates for authentication before establishing an L2TP/IPsec VPN tunnel.

Note

The Client configuration described here is for a Mac OS X 10.8.2 (Mountain Lion). However, the configuration would be similar in other Mac OS X versions.
The Names Server and Client Certificates are used in order to distinguish between the Certificates used in the SonicWall (Server) and the Mac OS X L2TP/IPsec Client (Client).
The Client Certificate used here has a Key Size of 1024 Bits. Although there is no official confirmation of a limitation from Apple, a Certificate with a larger Key Size failed during our testing.

Procedure

Before we begin the configuration process, the following requirements must be fulfilled:

The Certificates used in this configuration must be either obtained from a third party Certificate Authority (CA), like Verisign, or from a private CA like Microsoft CA or OpenSSL. Self-signed certificates are not supported.
The Certificates' Key Usage section must contain at least Digital Signature. If the certificates are to be obtained from a Microsoft CA, select either of these templates: IPSec (Offline Request), User, Administrator. If using OpenSSL, make sure the CA config file has at least Digital Signature under the Key Usage section.
The Server Certificate (certificate imported into the SonicWall) must have a Subject Alternative Name and must contain DNS: <FQDN> or <IP address>.

SonicWall Configuration

To obtain the Client and Server Certificates described in this configuration, refer this KB article: UTM: How to obtain certificates for VPN connections (Site to Site, GVC, L2TP) from a Windows Certificate server

Once the certificate has been imported, either in the method described in the above link or by importing a PKCS#12 (.pfx or .p12) format certificate, the System | Certificates page must be similar to the following.

NOTE: Note the Subject Alternative Name field set with Domain Name.

WAN GroupVPN Configuration

Login to the SonicWall management GUI
Navigate to the VPN | Settings page.
Click the Enable VPN check box at the top of the page and the Enable check box of WAN GroupVPN.
Click on Accept at the top to save the changes.
Click on the configure button under WAN GroupVPN to open the VPN Policy window.
Select Authentication Method as IKE using 3rd Party Certificates.
Select the signed certificate, imported into the SonicWall earlier, under Gateway Certificate.

Peer Certificates: When setting Peer ID Type, the administrator may choose from Distinguished Name, Email Address or Domain Name. The IDs must be of the certificate to be used (later in this article) in the L2TP clients.
- Peer ID Type: Domain Name - In this example, this is the attribute set at the time of obtaining the user certificate and found under the Subject Alternative Name (SAN) field of the certificate. Note: The server must be configured to accept SANs. For Microsoft CAs, refer this article: How to configure a CA to accept a SAN attribute from a certificate request
- Peer ID Type: Email ID - This must be attribute of san:email=<email@address.com> set at the time of obtaining the user certificate and found under the Subject Alternative Name field of the certificate. For example, the string *@hal-2010.local would allow anyone with an email address ending in hal-2010.local to have access.
- Peer ID Type: Distinguished Name - DN is a specific reference to a particular certificate. DN can be found in the Subject field of a certificate. The DN can also be found using the following OpenSSL command, openssl x509 -in l2tp.crt -noout -subject, where l2tp.crt is the name of the certificate. A portion of a DN (RDN) can also be used. This would be helpful in certificates without a SAN of domain name or email address. For example, a certificate with a DN of, /C=IN/ST=KA/O=SonicWall Inc./CN=l2tp-mac.kb-soniclab.local/emailAddress=admin@kb-soniclab.local, can be referenced in the DN field with the values of, CN=l2tp-mac.kb-soniclab.local or using wildcards, CN=*.kb-soniclab.local.

L2TP Server Configuration

For a Step-by-Step Walkthrough of Configuring the L2TP Server Role on SonicOS follow this KB.

Mac OS X Configuration

1. Login to the Mac.

2. Navigate to the Applications | Utilities Folder.

3. Click on Keychain Access.

4. In the Keychains window, navigate to System.

5. From the File Menu, click on Import Items.

6. Select the Client Certificate obtained earlier.

7. Enter your computer's Administrator Username and Password

8. Enter the Certificate Password when prompted and finish importing the Certificate.

Now that the client certificate has been imported, you must allow applications access to its Private Key. It is recommended to allow just Racoon, the IKE (ISAKMP/Oakley) key management daemon, but for simplicity we have shown allowing all applications access to the private key.

1. In the Keychains window, under Category on the left, navigate to Certificates.

2. Click on the triangle to the left of the certificate to display the Private Key of this Certificate.

3. Double click on the Private Key.

4. In the pop-up window, click Access Control at the top.

5. Click Allow all applications to access this item.

6. Click Save Changes

7. Enter your computer admin password when prompted

8. When prompted with Do you want to allow access to this item? click Allow.

9. In the Certificates category, click on the triangle to hide the private key. Double click on the certificate.

10. In the new pop-up window, click the triangle to the left of Trust to expand it.

11. Select Always Trust in the drop down menu labelled When using this Certificate.

12. Close the pop-up window.

Now that you have successfully imported the client certificate and allowed access to its Private Key, you must now import its CA certificate. The client certificate will not be validated until you import its CA certificate.

1. From the File menu, click on Import Items.

2. Select the CA Certificate of the Client Certificate.

3. Select Always Trust when prompted with Do you want to trust......

4. Enter your computer's administrator username and password to finish importing the certificate.

5. Close the Keychain Access application.

L2TP/IPsec Client Configuration

1. Navigate to System Preferences | Network.

2. Click on the plus (+) symbol in the lower left.

3. In the pop-up window, select VPN under Interface and enter a friendly name under Service Name.

4. Click on Create.

5. Select the newly created interface.

6. Set Configuration to Default.

7 . Under Server Address, enter the FQDN of the SonicWall IP address. This must match the Subject Alternative Name of the Server Certificate in the SonicWall.

8. Under Account Name, enter a Username of a local or LDAP User who is authorized to establish L2TP/IPsec VPN connections.

9. Click on Authentication Settings.

10. Enter the password of the user under Password

11. Under Machine Authentication, select radio button Certificate.

12. Click on Select to bring up the Choose An Identity window.

13. Select the certificate imported earlier from the list.

14. Click on Continue and then click OK.

15. Click on Apply to save the settings.

16. Click on Connect.

Troubleshooting

In such configuration, the most common cause of failure could be due to Certificates. Please check the following:

If the certificates are signed by different CAs, make sure the CA Certificate/s of the Peer is imported into the Client and Server.
Make sure the Subject Alternative Name (SAN) in the Server Certificate matches the Server Address in the L2TP/IPSec Client Configuration. If the SAN is FQDN, and does not have a corresponding DNS A Record, edit the hosts file under /private/etc and map the FQDN to the WAN IP Address of the SonicWall.
Make sure the Key Usage section of the Certificates has the appropriate entry authorized to establish IPsec VPN connections.
In our testing a Certificate with a Key Size larger than 1024 bits failed at IKE Main Mode Message # 5 from the client. During IKE using Main Mode with Certificate authentication, Message # 5 contains both a Signature and the Certificate. In other words, if a certificate with a public key size above 1024 bits is used, with extraneous info in the Subject field, the resulting packet would also be large. Therefore, we recommend creating user certificates with keys of 1024 bits and with minimal information in the Subject field.

For further troubleshooting, refer the Logs of the SonicWall and the Mac.

SonicWall L2TP/IPsec connection logs can be found under the following categories:

VPN IKE (Priority - Debug )
L2TP Server (Priority - Info)

In the Mac, L2TP/IPsec related logs can be found in the following files:

/var/log/racoon.log
/var/log/system.log
/var/log/ppp.log

Not Finding Your Answers?

ASK THE COMMUNITY

Was This Article Helpful?

YES NO

Capture Cloud Platform

Federal

Partner Portal

Support Portal

Capture Cloud Platform

Federal

Partner Portal

Support Portal

How to Configure L2TP/IPSec VPNs with Certificates while Using OS X

Description

Resolution

Resolution for SonicOS 6.5

Troubleshooting

Resolution for SonicOS 6.2 and Below

Troubleshooting

Related Articles

Categories

Not Finding Your Answers?

Was This Article Helpful?

Article Helpful Form

Article Not Helpful Form

Stay In Touch