Troubleshooting Replication in Split Configuration Environment
08/25/2023 19 People found this article helpful 484,662 Views
Description
Troubleshooting Replication in Split Configuration Environment
Resolution
In situations where there are issues with updated settings being replicated in split mode configuration environments, the user may receive alerts such as usermap is stale from a Remote Analyzer (RA). This document will help with troubleshooting steps to prioritize between basic connection issues and how to reference them through logs as well as commandline instructions. Periodically, a user may receive a usermap is stale alert from the Remote Analyzer or one of the Remote Analyzers may be rejecting or deleting legitimate messages due to DHA.
Troubleshooting Questions:
1) Make sure all servers of the split configuration environment are running the same version.
2) Is the server using IP address to connect or hostname?
3) Is the IP address being displayed correctly in server.xml and hostlist.xml?
4) Can the Control Center telnet to the Remote Analyzer on the configured port, for example port 80, 443, and 51212?
5) Can the Remote Analyzer telnet to the Control Center on port 2599?
6) Are the servers on the same subnet?
7) Is there a firewall between Control Center and Remote Analyzer?
8) Are NIC’s set to auto negotiate? What kind of NIC, switch?
Logs to Look At:
rpl.log Logging level has no affect on the contents of this log. This file lists all files that have been sent and received at the Control Center
and the Remote Analyzer.
mlftunnel.log This log will help to troubleshoot at the data level as it confirms data flow from Control Center To Remote Analyzer. Set to log level 2
in order to capture additional data in this log. This log will help show if the request is being made from CC to RA or and if the request
is being accepted or denied.
mlfreplicatorv2.log In log level 2, this log helps to troubleshoot at the application level. It will show information on the max gsn.idx as well as which files
are being replicated to which Remote Analyzer.
Quick TIP:
To verify that the copy of any config file on the RA is the same copy of the latest updated config file sitting on the Control Center, you can run the following commandline instructions to verify the indexed number. If replication has taken place properly, you should be able to run these commandline instructions and the numerical output should be the same from both servers.
Windows:
mlfreplicatorv2 –d gsn.idx | find “usermap.xml”
Appliance:
./MlfReplicatorv2 –d gsn.idx | grep “usermap.xml”
Look at the rpl.log for the relevant day on the Remote Analyzer to confirm that the RA had received the updated config file form the Control Center.
Other than telnetting from CC to RA over port 80 and 51212 and telnetting from RA to CC on port 2599 and 51212 to test connectivity, you can run the following as well to check if mlftunnel is running correctly and able to connect properly.
Mlftunnel –l –p 80 –d remoteanalyzer
Most Common Scenarios:
1) Something on the network has changed (ie: firewall or different subnet)
2) Sometimes the gsn.idx on the RA has become corrupt and needed to be rebuilt.
NOTE: It is not recommended to have the gsn.idx on the Control Center be rebuilt. Especially in a large deployment with many users.
mdcxxx.log This log helps to identify if the system is able to download the needed data from the datacenter. There have been cases whereby if for example Thumbprint downloads are failing, other processes may be stalled.
Related Articles
Categories