SonicWall CSE: Netagent Binary

Key Capabilities and Features of the Access Tier

The Access Tier is an identity-aware cloud-managed proxy written in Golang, and it runs on Linux servers. Access Tiers are deployed similarly to open-source proxies (such as Nginx) but have the advantage of being built with zero-trust security functionalities.

Access Tier Architecture #

The internal modules that comprise Access Tiers are depicted in the diagram below:

Access Tiers function primarily as a reverse proxy, intercepting traffic on specified ports on a Linux server. By default, Access Tiers intercept traffic on the following ports:

• 443/tcp - for web services, accessed via a browser
• 8443/tcp - for infrastructure services, accessed via the desktop app
• 51820/udp - for Service Tunnels that enable network access

Access Tiers leverage various Linux kernel functionalities to manage and forward traffic, including the following:

• iptables
• conntrack
• WireGuard

Note that Access Tiers are delivered as a Linux package (and NOT a Virtual Appliance) so can be used in a variety of cloud-native deployments.

Supported Operating Systems #

Netagent is the core technical component of an Access Tier. The Netagent binary can be installed on virtual or physical 64-bit Linux servers.

Officially Supported: the following distros are supported, when run with an officially released kernel:

• Amazon-Linux (2011.09-2018.03) and Amazon-Linux-2
• CentOS 7.x and RHEL 7.x
• Ubuntu 20.04+

Support Deprecated: the following Linux distros are no longer supported:

• CentOS 6.x
• Ubuntu 12.04, 14.04

Access Tier features #

Health Check, logs, metrics #

Overview

The Netagent provides a health check endpoint that provides a real time indicator for the status of the Netagent (up or down) and preliminary stats that can be used to evaluate the performance of the Netagent.

The Netagent provides the health check on port 9998 and the data presented from the health endpoint refreshes every 10 seconds.

Determining Netagent Health Status

Determining if the Netagent can service traffic to support the end users requests is critical for daily operations. To assist Loadbalancers in making the decision, the Netagent returns a 200 when it is available on port 9998. When the Netagent determines it is not healthy, the entire service will crash thus not returning anything on port 9998.

Stats provided by the Netagent Health Status

The Netagent has a health endpoint, /health, where it returns information about the services it provides and some critical information about the container or OS that the netagent is running on.

Here is an example output from the /health endpoint:

{
  "netagent": {
    "status": "up",
    "timestamp": "2023-07-18T01:28:45.080950798Z",
    "info": {
      "hostname": "tunnel-579fcb68-3a9d-4d7f-bc12-a6233e108e0b",
      "version": "2.7.1-39_gc907b778458",
      "start_time": "2023-07-18 01:27:40.961620437 +0000 UTC m=+0.041945553",
      "uptime": "1m4.119308781s"
    }
  },
  "shield_connection": {
    "status": "up",
    "timestamp": "2023-07-18T01:28:45.080952615Z",
    "info": {
      "seconds_since_status_change": 57.462623696,
      "seconds_since_rx": 45.666838646,
      "seconds_since_tx": 20.588735957
    }
  },
  "service_tunnels": {
    "status": "up",
    "timestamp": "2023-07-18T01:28:45.081091759Z",
    "info": null
  },
  "hosted_services": {
    "status": "up",
    "timestamp": "2023-07-18T01:28:45.081092718Z",
    "info": null
  },
  "system_resources": {
    "status": "up",
    "timestamp": "2023-07-18T01:28:45.081119661Z",
    "info": {
      "disk_used_percent": 40.484639586189786,
      "disk_inodes_used_percent": 4.942452566964286
    }
  }
}

When configuring a loadbalancer to check for the health of the Netagent we recommend an HTTP connection with return of 200.

Logs and Metrics #

Operations teams can use Netagent’s built-in logs and metrics capabilities to set up high-availability configurations and detailed monitoring.

• Logs - Netagent saves logs to both console and file; you can manage the log level via the Netagent Configuration. To change the location (default: /var/log/banyan/netagent.log), set the environment variable LOGFILENAME. You can also similarly configure syslog daemon to gather Netagent logs.
• Metrics - Netagent uses the statsd format to aggregate and summarize its metrics. Once you set the Netagent Configuration to emit to your statsd daemon, you can start collecting Netagent metrics including: tx/rx bytes, http response code, http response time, unauthorized attempts, error counts, etc. Metrics are emitted per service id.

Production Tuning

Background #

Modern Linux systems can handle thousands (or millions) of connections, but the system default settings are often inadequate for such high throughput.

This guide provides our recommended settings for moderate to heavy usage. The settings can be increased beyond our recommendations for extremely heavy workloads. Please consult our support team for advice.

The file descriptor and conntrack settings are required for production use. The additional TCP stack settings are optional for most systems.

These recommendations apply to all Netagents, whether running in Host Agent mode or Access Tier mode.

Recommended settings #

File descriptors

A busy server uses a high number of file descriptors for incoming and outgoing connections. The default for most user accounts is only 1024 descriptors. You can check the current limit for your account with this command:

$ ulimit -n
8192

To increase the limit beyond the default, create a file called /etc/security/limits.d/banyan.conf and add these lines to it.

* soft nofile 100000
* hard nofile 100000

If your system does not have a limits.d directory, append the above lines to the bottom of /etc/security/limits.conf instead.

Repeat the ulimit command to confirm that the new limits have taken effect.

$ ulimit -n
100000

There is also a system-wide limit that may need to be increased.

$ cat /proc/sys/fs/file-max
94158

If the output is less than 100000, adjust it by creating a file in /etc/sysctl.d called 90-banyan.conf and add the following to it:

fs.file-max = 100000

Reboot for the change to take effect, or change the limit temporarily with this command:

sysctl -w fs.file-max=100000

Confirm the change by repeating the cat command.

$ cat /proc/sys/fs/file-max
100000

If you are using RedHat, CentOS, Fedora, or Scientific Linux, you may need to add session required pam_limits.so to /etc/pam.d/login. Consult your OS documentation.

Kernel conntrack parameters #

Netagent uses a kernel module called nf_conntrack to help track connections to backend services. Connections are tracked in a kernel hash table, and if the hash table fills up, the module will drop incoming packets.

If you see messages in the kernel log like these, you need to increase the hash table size:

[34625.043999] net_ratelimit: 29 callbacks suppressed  
[34625.044003] nf_conntrack: table full, dropping packet

To increase the size of the hash table, create a file in /etc/modprobe.d called banyan.conf and add the line:

options nf_conntrack hashsize=65536

If you do not have a modprobe.d directory on your system, you can add the above line to the end of /etc/modprobe.conf instead.

This will adjust the size of the hash table to 64K entries. It takes effect on the next reboot. If you cannot reboot the system right away, you can also adjust the settings temporarily with these commands:

$ echo 65536 > /proc/sys/net/netfilter/nf_conntrack_buckets
$ echo 262144 > /proc/sys/net/netfilter/nf_conntrack_max

Confirm the settings have taken effect:

$ cat /proc/sys/net/netfilter/nf_conntrack_buckets
65536
$ cat /proc/sys/net/netfilter/nf_conntrack_max
262144

Kernel TCP parameters #

Here are some optional kernel tunables that should suffice for even very busy systems. Add these to the /etc/sysctl.d/90-banyan.conf file you created above:

net.ipv4.tcp_rmem = 4096 25165824 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.somaxconn = 100000
net.ipv4.tcp_max_syn_backlog = 100000
net.core.netdev_max_backlog = 100000
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824