Troubleshooting Replication in Split Configuration Environment

03/26/2020 17 11745

DESCRIPTION:
Troubleshooting Replication in Split Configuration Environment

RESOLUTION:

In situations where there are issues with updated settings being replicated in split mode configuration environments, the user may receive alerts such as usermap is stale from a Remote Analyzer (RA). This document will help with troubleshooting steps to prioritize between basic connection issues and how to reference them through logs as well as commandline instructions. Periodically, a user may receive a usermap is stale alert from the Remote Analyzer or one of the Remote Analyzers may be rejecting or deleting legitimate messages due to DHA.

Troubleshooting Questions:  

1)       Make sure all servers of the split configuration environment are running the same version.

2)       Is the server using IP address to connect or hostname?

3)       Is the IP address being displayed correctly in server.xml and hostlist.xml?

4)       Can the Control Center telnet to the Remote Analyzer on the configured port, for example port 80?

5)       Can the Remote Analyzer telnet to the Control Center on port 2599?

6)       Are the servers on the subnet?

7)       Is there a firewall between Control Center and Remote Analyzer?

8)       Are NIC’s set to auto negotiate? What kind of NIC, switch?

Logs to Look At:

rpl.log                          Logging level has no affect on the contents of this log. This file lists all files that have been sent and received at the Control Center

                                      and the Remote Analyzer.

mlftunnel.log               This log will help to troubleshoot at the data level as it confirms data flow from Control Center To Remote Analyzer. Set to log level 2

                                        in order to capture additional data in this log. This log will help show if the request is being made from CC to RA or and if the request

                                        is being accepted or denied.

mlfreplicator.log         In log level 2, this log helps to troubleshoot at the application level. It will show information on the max gsn.idx as well as which files

                                        are being replicated to which Remote Analyzer.

Quick TIP:

To verify that the copy of any config file on the RA is the same copy of the latest updated config file sitting on the Control Center, you can run the following commandline instructions to verify the indexed number. If replication has taken place properly, you should be able to run these commandline instructions and the numerical output should be the same from both servers.

Windows:

mlfreplicator –d gsn.idx | find “usermap.xml”

Appliance:

./MlfReplicator –d gsn.idx | grep  “usermap.xml”

Look at the rpl.log for the relevant day on the Remote Analyzer to confirm that the RA  had received the updated config file form the Control Center.
 

Other than telnetting from CC to RA over port 80 and telnetting from RA to CC on port  2599 to test connectivity, you can run the following as well to check if mlftunnel is running correctly and able to connect properly.

Mlftunnel –l –p 80 –d remoteanalyzer

Most Common Scenarios:

1)    Something on the network has changed (ie: firewall or different subnet)

2)    Sometimes the gsn.idx on the RA has become corrupt and needed to be rebuilt.

NOTE: It is not recommended to have the gsn.idx on the Control Center be rebuilt. Especially in a large deployment with many users.

mdcxxx.log                 This log helps to identify if the system is able to download the needed data from the datacenter. There have been cases whereby if for example Thumbprint downloads are failing, other processes may be stalled.