Duplicate initiator IP entries in Web Activity reports

03/26/2020 1046 11805

DESCRIPTION:
Duplicate initiator IP entries in Web Activity reports

RESOLUTION:

Question: Web Activity reports contains duplicate (repeated) initiator ip’s
Web activity reports for a specific source ip (ex : 10.3.29.87) is listed multiple times (some entries with ip , hostname , but no username , some entries with ip , username , but no hostname and one entry with just ip—address and not hostname , username info )

Answer: The data is not in duplicate 
 
Row 1 - The ip + hostname + user together makes up the key
Row 2 - The ip + empty hostname + user name makes up the key
Row 3 - The ip + hostname + empty user name makes up the key
Row 4 - The ip + empty hostname + empty user name makes up the key
 
Note the distinction - "EMPTY"
 
In each of the above instance the data was extracted from the syslog (triggered by user activity) sent by the firewalls as is. For event 1 the firewall sent all three values but for other events the firewall sent the syslog with 1 one or more fields as empty.
 
Why is the syslog from the firewall so flaky, or unreliable, or any other adjective you can use to describe the discrepancy?
It is not any of the above. The firewall’s core job is to inspect the traffic and send it the request out as soon as possible. So, at times the Name Resolution engine that scrubs the request and translates the ip into readable names for host and user fields is skipped if the processing is taking too long. However, on the GMS and Analyzer front we are taking as many steps as possible to scrub the data and present it to the user in nicer format.
 
How can you improve the quality of the syslogs to make sure that the hostname is always included in the syslog?
This question to firmware team has come up several times and here is the response:
 
Why do Analyzer reports show ip instead of name of the web site?
KB 5261 & 5264
Sonicwall 'displaying authenticated username'
 
Background: ViewPoint consumes the contents of dst, dstname, etc tags from the syslog. The syslog format and tags depends on the version of the firmware.
 
Syslog Samples:
 
Firewall World
Without “Name Resolution” Feature in Firmware
<134>id=firewall sn=0006B101200E time="2006-09-23 02:37:47 UTC" fw=75.41.120.48 pri=6 c=1024 m=97 n=221 src=10.50.162.84:1929:LAN dst=209.62.180.191:80:WAN proto=tcp/http op= rcvd=530 result=302 dstname=

With “Name Resolution” Feature in Firmware
<134>id=firewall sn=0006B1020470 time="2005-10-10 22:54:28 UTC" fw=64.220.173.243 pri=6 c=1024 m=537 msg="Connection Closed" n=8848 usr=jhu src=172.18.2.148:3498:X3: dst=10.50.128.204:1054:X0:US0EXB04 proto=tcp/1054 sent=2178 rcvd=664
 
CSM World
With “Name Resolution” integrated with AD Connector:
<134>id=sonicwall sn=0006B127B660 time="2008-02-18 20:24:36 UTC" fw=10.0.0.4 pri=6 c=1024 m=97 n=582 usr="SCANNERPC/Local User" src=10.0.0.38:1321:X0 dst=76.163.52.97:80:X1 proto=tcp/http op=GET sent=392 rcvd=1500 result=404 dstname=www.steelsolutions.com arg=/images/stripe.jpg  code=64 Category="Not Rated"

After the syslog is captured, the summarizer processes the syslog as follows:
 
In Gen 1 Summarizer Mode
If FQDN is not available in the syslog ViewPoint application performs a simple reverse look up using the ip. Once the ip is tagged appropriately all reports going forward will consume the name associated to this ip. One drawback, multiple URL’s or a change in the future is not captured.
 
In Gen 2 Summarizer Modes
ViewPoint consumes the content of the syslog and this also depends on the version of the firmware.
 
Firmware without “Name Resolution” features
Capture the contents of the syslog as is.
 
Firmware with “Name Resolution” feature
The summarizer processes the usr tag first if available else consume the contents of the src tag.
 
How does DNS Reverse Resolution work/not work in Firewalls (for syslogs) 

1. Since a name resolution takes milliseconds and the time from event issuance to syslog can take microseconds, the firewall cannot wait for name resolution to complete before rendering a message. This means that the first time an IP address is encountered, there will be a delay before its name will be available to the rendering mechanism.

1. The name resolution cache is currently set to 256 entries, so once the table fills up, we reuse old entries. This means that an IP address can go in-and-out of name resolution. This area could use more analysis to determine the optimum table size. When I did the implementation, I assumed the peak rate of new IP addresses was 50 per second. I thought that giving them 5 seconds of “life” would be adequate. I assumed the average case had smaller rates of new IP addresses and smaller “working-sets” of IP addresses. Probably a better data point to base this number on would be the average number of hosts/users (i.e., working-set) Viewpoint users expect to observe in reports.

1. The name resolution scheme respects DNS TTL values, so this is another cause of losing an IP address’s name. NetBios doesn’t have the concept of TTL so we default to 1200 seconds (20 minutes) which was chosen to be similar to typical DNS TTLs.