How do I troubleshoot high data plane CPU usage on a firewall?

03/26/2020 145 7749

DESCRIPTION:

In some cases the firewall may exhibit high data plane (DP) CPU activity due to network congestion. To resolve the high DP CPU please first confirm that the following log messages appear in the firewall tech support report (TSR) and/or packet capture.

Log Event in Tech Support Report

Download a TSR by performing the following:

1. Login to the firewall and navigate to INVESTIGATE | System Diagnostics | Download Report.
   
   
2. Open the TSR file and search for "byte buffer count".
3. If you see an entry (as seen below) then continue to the resolution section of this article.
   1500 byte buffer count = 1 free (Max: 20000 Lowest: 0 Was 0 at: 11/15/2019 11:02:35.320)
LOW RESOURCE: 1500 byte buffer count = 1 free

Log Event in Packet Capture

While performing a packet capture under INVESTIGATE | Packet Monitor if the log message (as seen below) is displayed then continue to the resolution section of this article:

DROPPED, Drop Code: 140(IDP detection OOO Out of Buffers), Module Id: 25(network), (Ref.Id: _7130_uyHtJcpfngKrRmv) 3:3

CAUSE:

Low buffer memory allocation.

RESOLUTION:

If log messages are observed on the firewall (as seen above) please perform the following:

1. Login to the firewall.
2. Navigate to the diag.html page. In the browser URL replace main.html with diag.html.
3. Click Internal Settings.
   
4. Search for IDP Buffer Mempool 1500 Size.
5. Change the value from 0 to 100,000. Note: in some environments this value may need to be increased further. 
   
6. Click Accept.
7. Reboot the firewall for the changes to occur.

ISSUE ID:

DTS 209149

This issue has been resolved in SonicOS 6.5.3.0-36n and later.