Flow Reporting and Visualization FAQs (SonicOS 5.8 and Later)
03/26/2020 47 16962
What Is a Flow? A flow is identified as a unidirectional stream of packets between a given source and destination, uniquely defined by a network-layer IP address, transport-layer source and destination port numbers, layer 3 protocol type and input logical interface. SonicWall’s flow reporting also considers non connection related flows like user, URL, location, etc. Both visualization and flow reporting to an external collector using the IPFIX with extensions type use bidirectional streams as per RFC5103 - Bidirectional Flow Export Using IP Flow Information Export (IPFIX). Basically, IPFIX considers a data record for any table/ row used a flow.
What is considered visualization? The SonicWall Visualization Dashboard offers administrators an effective and efficient interface to visually monitor their network in real time, providing effective local flow reporting charts of real-time data, customizable rules, and flexible interface settings primarily through the App Flow and Real-Time Monitors.
What is considered Netflow reporting? NetFlow/ IPFIX is a network protocol and an industry standard for traffic monitoring and collecting IP traffic information. Starting from SonicOS v18.104.22.168 with 'Report to EXTERNAL flow collector' set and an 'External collector's IP address' defined, flow records are generated which are exported from the SonicWall in UDP packets and collected by an external flow collector.
How to enable the App Flow Monitor (AFM) and Real-Time Monitor (RTM)? Ensure that the TZ210 or higher SonicWall NGFW model running SonicOS 5.8 or higher is registered and licensed for App Visualization. Check and apply the 'Enable Flow Reporting and Visualization' flag in the Log > Flow Reporting section and manually restart the unit from the System section afterward. Provided flows are traversing the rebooted firewall, the AFM and RTM should start showing data visualizing these flows in a real-time fashion.
How to get the SonicWall licensed for App Visualization? The App Visualization license for the visualization and flow reporting services on units running SonicOS 22.214.171.124 and higher, is included with the existing GAV/IPS, CGSS and Total Secure licenses. An App Visualization 30 day trial is available through the management GUI of the unit in the System > Licenses section.
What is the reason for AFM mouse-over status dialog in the Dashboard > App Flow Monitor GUI section to indicate yellow? This signifies that NOT all appropriate signatures and databases are active and that some or all are still being downloaded, cannot be downloaded or could not be activated. It indicates the status of App Control-, GAV-, Anti-Spyware-, IPS- Signature and Country Databases, Content Filtering Service (CFS), Maximum Flows in the internal AFM database and if Flow Collection is enabled. NOTE: most likely the status will turn green after a couple of minutes. If not, check the relevant services and their status.
Why is the AFM not showing any data even when the AFM status indicator shows green and the RTM is showing data? The most likely reason for this is that the interface and/ or firewall/app rules based reporting flag(s) in the Log > Flow Reporting GUI section are checked, while the correlating flow reporting flags on the interface, FW access and/ or App rule level are NOT set. Since these advanced flow reporting global ‘filter’ options, which are disabled by default, only apply to the AFM and external flow collectors the RTM is NOT affected. NOTE: the first recommended step is to uncheck the interface and/ or firewall/app rules based reporting flag(s) in the Log > Flow Reporting GUI section, as per default.
Why does the AFM Users tab only show admin and/ or unknown? By design only the users that get authenticated by the firewall ULA mechanisms including Xauth, UTM SSL-VPN portal, NetExtender, L2TP or through Single Sign on (SSO, TSA/NTLM) will show up on the AFM. Active User Sessions other then admin in the Users > Status GUI section should be present and actively generating unfiltered flows to be visualized in the AFM. For verification purposes it is recommended to create a local user, setting the user login flag(s) on the relevant interface(s), manually authenticate from a downstream host with this user account and start generating traffic which traverses the SonicWall. From the admin management session, check the AFM Users tab for flows generated by user(s) other then admin.
Why are users who are member of the SonicWall Read-Only Admins group not able to see the RTM and AFM in the Dashboard? By default SonicOS only allows the admin to view the RTM and AFM to prevent unnecessary extra load on the internal web service due to concurrent monitoring sessions. The internal settings provide a way to override this limitation by checking the 'Enable Visualization UI for Non-Admin/Config users' flag under the Flow Reporting diag page section.
Why is my external flow collector not reporting any of the SonicWall specific types of flows? Most likely the external flow collector does not support IPFIX with extensions and therefore mainly the standard 7 network flow types are forwarded to the collector. For richer flow reporting capabilities, use a flow reporting application which supports IPFIX with extensions and set the firewall’s flow reporting type to the same.
Why is no AFM data showing for URLs while web traffic is traversing the firewall? The AFM URLs are dependent on CFS and therefore it might be an issue with CFS not being licensed/ activated, CFS is being bypassed for admin or other user sessions or the CFS policy assignment is set to ‘Via App Rules’ but no App Rule is present or active with a CFS category List match object.
What is the main difference between applications shown in the AFM which are prefixed with ‘General’ compared to the ones which are not? The applications shown in the AFM which are prefixed with General are based on protocols defined by service objects, where the ones which are not are mainly based on application signatures.
Will enabling Visualization and/ or External Flow Collector impact the overall performance of the firewall? Yes, both Visualization and External Flow Collector impact the maximum connections level of the unit with between 20 and 25% compared to when these are disabled. The System Information in the System > Status GUI sections shows the actual maximum connection numbers in a table. Visualization will have more impact on the unit’s memory compared to External Flow Collection since it utilizes an internal database. It’s affect on the internal web service is also greater, where obviously the amount of sessions using the AFM and RTM are relevant. Lastly the increased CPU utilization depends on the amount of flows which require visualization and/ or reporting, but some impact is expected. Use the Connection, Core and Web Server monitors together with the RTM to determine the actual impact.
How can I save an AFM filter view? Apply a specific AFM filter view, type a descriptive name in the 'Load Filter' drop down menu and hit the save button below it.
What is the reason for the AFM showing responders and/ or initiators from unexpected countries? These are likely tracking cookies or similar types of ‘unwanted’ traffic which might be an indication to further investigate and adjust security levels making policies more restrictive, etc.
Is the unit sending flows to the external collector? To confirm this, check the Flow Reporting Statistics in the Log > Flow Reporting GUI section where the values like 'NetFlow/IPFIX Packets Sent' and others should be incrementing over time. For visualization you reference the App Flow Reporting Statistics. For the external collector the Packet Monitor can be used, filtering on the collector’s IP and configured UDP port (e.g. 2055) or running a capture on the target host filtering on source address of the firewall’s interface.
Why is RTM memory usage not showing? RTM Memory Usage requires the unit to be running in diagnostic mode or a debug firmware build.
Why is the RTM Application chart inactive and showing [Chart Disabled]? The RTM Application chart is dependent on the Visualization state. To activate the RTM Application chart, check the ‘Enable Flow Reporting and Visualization’ flag in the Log > Flow reporting section and reboot the unit.
Why is NetExtender and L2TP client traffic not Visualized through AFM? Besides policy and route based site to site VPN, currently only Global VPN Client traffic is visualized on the terminating unit under the AFM’s VPN tab. Note that flows traversing the NextExtender and L2TP client tunnels are visualized and reported.
Why does interface based flow reporting still report flows for interfaces which have flow reporting disabled? Since a flow consists of two interfaces, initiator and responder, SonicWall's current flow reporting implementation reports flow if any of the two interfaces has “flow reporting” enabled.
Why are the AFM reported total packet values for a specific protocol/ application larger for the last 24 hours compared to the last 7 days? SonicOS will automatically purge 25% of the database used by the AFM when it overflows, flushing the oldest closed connections. This mechanism will affect visualization accuracy over time depending on the flow reporting volume and the allocated memory. NOTE: SonicWall recommends using an external collector for flow reporting over extended periods.
Why does, launching monitor pages in stand-alone browser frames, make the management GUI sluggish? Launching monitor pages on a standalone browser frame is not recommended for the TZ210(W) single core and NSA240 dual core units.
Why are, unlike physical and VLAN/ sub interfaces, VPN tunnel interfaces not visualized in the RTM and AFM? Traffic traversing route based VPN tunnels using tunnel interfaces is shown in the AFM VPN tab but grouping Initiators or Responders flows on Interface will not. The in- and egress bandwidth of the RTM also does not show VPN tunnel interfaces. This capability will be added in future versions.
How are the average rates for the RTM and AFM calculated? For RTM the average rate calculation is highly accurate and done instantaneously, considering the amount of initiator and responder bytes sent on a connection in the last second. For AFM it is a lifetime (rolling) average rate calculated per flow, where the Ave Rate (KBps) considers the sum of all active connections divided by the number of sessions, excluding the closed connections for accuracy reasons.
Considering the RTM and AFM, which browser does SonicWall recommend? SonicOS 5.8 with Visualization uses advanced browser technologies such as HTML5 which are only supported in the latest browsers. SonicWall therefore recommends using Google Chrome or Mozilla Firefox browsers for administration of SonicOS 5.8.
From an administrative perspective, what are the main goals of Visualization? Visualization primarily answers questions, often asked by administrators, like: ‘Who is doing what at this moment or in the recent past?’
What is the recommended sequence to identify which top user is doing what and when on my network? Use the AFM to firstly identify the top user adding a filter, secondly using and filtering the applications, URLs and other variables determining the nature of the activities to finally use the flow table sessions for time indication and other information.
How-to determine the current top bandwidth consumers? Use the RTM In- and Egress bandwidth and Application charts with legends to identify the interface(s) and applications which utilize the most. Use this information to filter the AFM, drilling down to the source hosts and users of the current top bandwidth consumers.
Can rules be created based on AFM visualized flows? SonicOS 5.8.1 and above firmware supports AFM rule creation, allowing actions like blocking, throttling and monitoring of non-General applications, users IP address, URLs, Domain Names, initiator and responder countries. This branch of firmware also introduces the Geo-IP & Botnet Filter service, which prevents connections to and from Botnet Command and Control Servers besides blocking all connections to and from specific countries on a global level.
Is the historical AFM data retained after a reboot of the firewall? No, the current AFM design stores data in RAM which is flushed by rebooting the firewall.
Which monitoring, logging and reporting option(s) offered by SonicOS is recommended? Not one but mostly several of the available methods is recommended depending on the requirements.
Visualization: Real time and application flow monitoring of network usage over relatively short intervals.
External flow reporting collector like SonicWall Scrutinizer: Recommended for highly efficient and scalable near real time monitoring and network usage over an extended period.
SNMP: Recommended method to discover network devices, connectivity and link utilization without identifying traffic types and volume between client and servers.
Packet Monitor: Recommended for troubleshooting and tracking down network errors with the highest packet-level detail.
Local logs and alerts with log automation: The smallest deployments can fully rely on this method, although it is not very administrator friendly and therefore the potential for missed critical events is relatively high. All deployments are recommended to do some level of local logging.
Syslog, ViewPoint/ Analyzer or GMS (UMS/ UMA): Recommended for log collection, reporting and management purposes over an extended period.
Solera Capture Stack: For deployments requiring the highest level of monitoring and analysis Solera Capture Stack allows for capturing, archiving, regenerating network traffic and accessing the captured data in time sequenced playback for analysis on a live network.
Why is the external collector not reporting flows for encrypted traffic? Provided sending dynamic flows for VPN tables on the firewall using IPFIX with extensions was not disabled, the 3rd party flow reporting application might not support SonicWall VPN reports yet or exclude encrypted protocols by default to prevent overstated numbers in the flow reports.
Which 3rd party vendors support flow reporting using IPFIX with SonicWall extensions? SonicWall offers Scrutinizer, for other solutions please check with your reseller and/ or flow reporting software vendor(s) for confirmation on this.
What is the bandwidth impact of flow reporting to an external collector? This depends on the flow volume, reporting type, mode, settings and in case of IPFIX with extensions the static and dynamic tables send. The impact will be limited and hardly noticeable on high speed links, considering the nature of flow reporting traffic.
Why are flows reported for specific App Rules not showing? If flow reporting is set on the App Rule and interface-based flow reporting is NOT enabled, first check the App Rule’s policy statistics for times matched. This value should be incrementing and > 0. If not, an overlapping firewall access rule which takes precedence might be matched instead. Note that when using an external collector IPFIX with extensions is required for App Rule flows.
Is flow reporting supported on TZ100 and TZ200 series? No, visualization and flow reporting is supported on TZ210 and higher Gen5 units running 126.96.36.199 or higher and all Gen6 units/ firmware.
My marketing department needs access to Facebook to build awareness—but I want to block the games... what can I do? Application Intelligence, Control and Visualization enables you to guarantee bandwidth and allow access for some applications while restricting or blocking either specific applications, categories of applications or to block application sub-components such as the Farmville game or file transfer from within Instant Messaging applications. You can even create rules in order to block specific lists from accessing these applications. This is all tracked by logging in to the Application Flow Monitor to see, through real-time visualization, the effectiveness of your new application control policy.
How do I ensure that my sales team has prioritized bandwidth to productive Web-based applications like Salesforce.com® over unproductive Web-based applications such as ESPN® Live View? Application Intelligence, Control and Visualization easily enables you to ensure bandwidth for key business applications while restricting or blocking application categories such as gaming or streaming video. With this solution, you can select critical applications or groups of applications, specify the bandwidth you would like to guarantee to your specified user groups and create an application control policy to prioritize bandwidth for the selected critical applications and users. Finally, you can check the Application Flow Monitor to see in real-time the results of your new application policies.
My sales team users really need to access corporate Web seminars and videoconferences—but I do not want them hogging bandwidth on YouTube®. What can I do? Application Intelligence, Control and Visualization enables you to ensure bandwidth for key Web-based collaboration tools, while restricting or blocking non-productive Web applications such as YouTube. Simply select the critical Web-based applications or urls and choose how much bandwidth you want to guarantee to this group. You can also restrict bandwidth or block non-productive Web-based applications or URLs and select the users or group that will be included in this policy. By logging in to Application Flow Monitor, you can visualize the results of your new policy.
I need to be able to scan all traffic simultaneously to defend against malware, as well as visualize and control any type of application on the network. What can I do? Application Intelligence, Control and Visualization with enhanced visualization and Reassembly-Free Deep Packet Inspection enables you to scan and protect 100% of all application traffic entering your network while simultaneously performing traffic shaping without introducing latency. With the Application Flow Monitor, you get a real-time view of application and user traffic, bandwidth utilization and other user activity. It's easy to export data continuously to any NetFlow/IPFIX analyzer for off-box monitoring, troubleshooting and analysis of historical network activity. Meanwhile, you can continue to adjust your network policies based on critical observations, plus monitor threats that are stopped at the gateway before entering your network.
I need to protect confidential content from being leaked over outbound Web mail transmissions or social media sites. What can I do? Application Intelligence, Control and Visualization with enhanced visualization and Reassembly-Free Deep Packet Inspection enables you to scan 100% of all application traffic leaving your network, including email attachments and FTP uploads. You can simply create a filter for the type of content that identifies confidential data, and then apply the filter to outbound mail, FTP and other traffic, and set the firewall to block the transfer. By monitoring the logs, you can easily detect the transfer of confidential data. You can also observe traffic patterns with the Application Flow Monitor and adjust network policy accordingly.
Why are the RTM, AFM and/ or external flow collector only showing services and no applications? Most likely the firewall’s DPI engine is shut down. In the Firewall Settings > Advanced > Connections section ensure either Recommended or Least connections is set. Maximum connections disables the DPI engine on which the visualization of applications through the RTM, AFM and external collector dependents. Note that re-enabling the DPI engine requires a reboot and afterward the applications are expected to be visualized, provided flows matching application signatures are traversing the firewall.
Why while troubleshooting connectivity issues, the firewall’s capture shows a drop code for IDP Detection of the Network module despite the fact the initiating host is excluded from all security services; Application Intelligence/Control, IPS, Gateway AV, and Anti-Spyware? If flow reporting is enabled, traffic still runs through the DPI engine, in order to identify applications and threats, even if all security services are disabled or hosts are excluded. This can be worked around by changing the flow reporting type from the default ‘All’ to ‘Interface-‘or Firewall/ App Rules-based and ensuring the relevant flows are not reported. More drastically, flow reporting can be disabled on a global level.
Why after backup preferences file import on a factory default booted unit, flow reporting stops working? Flow reporting requires licensing, impacts the connection cache size and is dependent on the unit’s registration and license synchronization. Because of this, it is expected that the 'Enable Flow Reporting and Visualization' and 'Report to EXTERNAL flow collector' flags remain unchecked while the other flow reporting settings are reflected in the configuration. The AFM, status and flow reporting settings will clearly indicate this fact and imply these flags need to be re-enabled also requiring a restart afterward.
What is the reason for the AFM’s status to show yellow/ orange? This is expected when Content Filtering is disabled since a significant part of the AFM functionality relies on it. Note that if the Anti-Spam status is showing disabled since CASS is not licensed/ enabled the green AFM status indicator will be shown, because not much of the visualization depends on this service.
Why does the AFM status keep showing ‘Not Downloaded’? Make sure the firewall’s sanctioned DNS server is able to resolve geoipdata.global.sonicwall.com. If not either set a different sanctioned DNS server which is able to resolve it or change the default internal diag page setting for ‘Location Remote Server Address’ to ‘Always use this IP for geoipdata.global.sonicwall.com’. In cases where previous resolution of the FQDN geoipdata.global.sonicwall.com was successful but for some reason erroneous, hit the ‘Clear Location Map Database’ and ‘Clear Database Tables’ buttons in the Flow Reporting section of the internal diag page settings.
How does the firewall determine the country of a flow? If for a flow no existing IP connection cache entry exists, a new one is added after which a DNS request is sent to SonicWall’s Geo-IP backend server at geoipdata.global.sonicwall.com. The server replies with region and location IDs which is cached by the firewall and used for the AFM and in 188.8.131.52 and above also for Geo-IP filtering. The firewall will retry 3 times if initial lookup fails and if still not successful delete the entry from its Geo cache.
Why are custom AppRules not uniquely identified by the AFM as such? The design and intent of the flow reporting flag on firewall and App rules is for filtering purposes, implied by the pop-up warning message when checking the ‘Enable firewall/app rules based reporting (advanced)’ flag in the Log > Flow Reporting section.
While ping-ing the remote VPN site’s interface IP, it’s replying but AFM’s VPN tab does not show reports for it, why? Only fully traversing flows are reported.
Other Netflow reporting products use active timeout flow reporting values. Where can I find these on SonicOS? SonicOS reports flows based on triggers for scalability purposes and therefore it does not use active timeouts.
Why are changes and additions to service objects not reflected afterward in the flow reports? SonicOS does not support editing of flow tables since it stores the index of services. The changes will only be reflected after a restart of the firewall.
What is the reason for a large number/ majority of AFM initiator and responder sessions grouped on country, showing ‘unknown’? Reference the ‘unknown’ flow table initiator or responder sessions for private and multicast IP addresses, which is the likely cause of most of the unknowns. In case a large number of public addresses fail Geo location, list them and contact support. Note that future firmware will add a ‘private/ multicast IP’ group and that not considering these addresses the Geo location accuracy is expected to be 95% and higher. External flow collectors use their own Geo location databases/ services if supported and therefore issues regarding it should be addressed to the appropriate vendor.