Why am I getting disk space alerts and blank emails sent from GMS/Analyzer?

03/26/2020 84 16015

DESCRIPTION:

Disk Space Utilization alerts or Blank Email alerts received from GMS/Analyzer is a common issue. This article describes how to identify where disk space is being used in order to help identify further possible issues.

RESOLUTION:

• Description of locations storing data in GMS
• Identifying which directories are using up large quantities of disk space
• Identifying usage and/or issues

Locations in GMS/Analyzer that store large amounts of data

• Windows
  
  [installDir]:GMSVPsyslogs
[installDir]:GMSVPsyslogsarchivedSyslogs
[installDir]:GMSVPsyslogsbadSyslogs
[installDir]:GMSVPinfobrightdatasyslogs
[installDir]:GMSVPbackup
[installDir]:GMSVPTemp
[installDir]:GMSVPTempoptimizer
  
  
• VA/UMA
  
  /opt/GMSVP/syslogs
/opt/GMSVP/syslogs/archivedSyslogs
/opt/GMSVP/syslogs/badSyslogs
/opt/GMSVP/backup
/opt/GMSVP/Temp
/opt/GMSVP/Temp/optimizer
/var/lib/infobright/syslogs
  
  
  
• Description
  
  [installDir]:GMSVPsyslogs
/opt/GMSVP/syslogs
  
  
• This is the working directory for incoming syslogs.Syslogs are written to this directory when received and then processed into either the badSyslogs or archivedSyslogs directories.
  
  [installDir]:GMSVPsyslogsarchivedSyslogs
/opt/GMSVP/syslogs/archivedSyslogs
  
  
• Stores the raw syslogs after the information is successfully processed into the Reports Database.
• Processed data is periodically zipped and stored according to the Raw Syslog Data configuration on the Console tab under Reports|Summarizer.
  
  
  
  [installDir]:GMSVPsyslogsbadSyslogs
/opt/GMSVP/syslogs/badSyslogs
  
  
• Syslogs are sent to the badSyslogs directory when errors during summarization are encountered.
• These files are periodically zipped, but are permanently stored unless admin manually deletes these files.
  
  [installDir]:GMSVPinfobrightdatasyslogs
/var/lib/infobright/syslogs
  
  
• All processed syslogs are uploaded the reports database.
• Tables for the reports database are stored in this directory.
• Data in this directory is stored according to the Reporting Data configuration on the Console tab under Reports|Summarizer.
  
  
  
  [installDir]:GMSVPbackup
/opt/GMSVP/backup
  
  
• Backups taken periodically by the GMS server are stored in this directory.
• In Windows the storage location of the backups can be modified.
• Backups and store location is configured in the System Interface (https://IPAddress:port/appliance) under System|BackupRestore.
  
  
  [installDir]:GMSVPTemp
/opt/GMSVP/Temp
  
  
• The Temp directory is used to create temporary files that are needed during processing of syslogs as well as other tasks.
• Common file extensions stored in this directory are
  
  .sqlload
  .sqlload.err
  .reso
  
  
• These files should be purged periodically, however, service failures and other issues may result in the need to manually purge these files.
  
  [installDir]:GMSVPTempoptimizer
/opt/GMSVP/Temp/optimizer
  
  
• The optimizer directory is used by the Reports Summarizer service during the data optimization process.
• The data optimization process is kicked off by default (in GMS/Analyzer 7.2; prior version required manual config), usually between 1 am - 2am every morning.
• Files in this directory should be purged automatically after the optimization process is completed.
• Issues encountered by the Summarizer service may create a situation where files in this directory will need to be manually purged.

Identifying large directories in GMS/Analyzer

• A Tech Support Report (TSR) can be used to identify how much space is used in the directories mentioned in the previous section of this article.
• The following KB article can be used to help obtain a TSR from either GMS or AnalyzerHow to Download Tech Support Report (TSR) from SonicWall Global Management System(GMS).
• Here are some examples of output from the TSR from Windows and Virtual Appliance/UMA as the two platforms list this very differently.

Windows

• If you have direct access to the Windows server, it is easiest to look directly in the file system at the size of the directories mentioned in the article.
• Free software such as TreeSize Free can also be installed to quickly identify large directories in Windows.
• Output from Windows TSR.
  
  Install Dir (G:GMSVP) Available Space: 259.91 GB (of Total 1863.01 GB)
  Syslogs Dir (G:GMSVPsyslogs) Available Space: 259.91 GB (of Total 1863.01 GB)
  
  +++++++++++++++++++++++++++ Start of Disk Usage Diagnostics +++++++++++++++++++++++++++
  +++++++++++++++++++++++++++ MySQL +++++++++++++++++++++++++++
      Size: 1,724.88 MB
  +++++++++++++++++++++++++++ Reporting Database +++++++++++++++++++++++++++
      Size: 537.56 MB
  +++++++++++++++++++++++++++ Syslogs +++++++++++++++++++++++++++
      Size: 666.74 MB
  +++++++++++++++++++++++++++ Archived Syslogs +++++++++++++++++++++++++++
      Size: 621.99 MB
  +++++++++++++++++++++++++++ Bad Syslogs +++++++++++++++++++++++++++
      Size: 44.60 MB
  +++++++++++++++++++++++++++ Temp +++++++++++++++++++++++++++
      Size: 621.81 MB
  +++++++++++++++++++++++++++ Backup +++++++++++++++++++++++++++
      Size: 0.80 MB
  ++++++++++++++++++++++++++++ End of Disk Usage Diagnostics ++++++++++++++++++++++++++++
  
  
  

Virtual Appliance/UMA

• Since no file system access is available for our Virtual Appliance or UMA platform, the TSR will be the primary location to identify the size of each directory.
• Output from VA TSR:
  
  Install Dir (/opt/GMSVP/) Available Space: 0.16 GB (of Total 0.81 GB)
  Syslogs Dir (/opt/GMSVP/syslogs) Available Space: 30.09 GB (of Total 33.75 GB)
  
  +++++++++++++++++++++++++++ Start of Disk Usage Diagnostics +++++++++++++++++++++++++++
  +++++++++++++++++++++++++++ /var/lib/mysql folder usage +++++++++++++++++++++++++++
  1.0M    /var/lib/mysql/mysql
  4.0K    /var/lib/mysql/test
  388.0K    /var/lib/mysql/sgmsdb_archive
  6.5M    /var/lib/mysql/sgmsdb
  1.5G    /var/lib/mysql
  +++++++++++++++++++++++++++ /var/lib/infobright folder usage +++++++++++++++++++++++++++
  964.0K    /var/lib/infobright/mysql
  2.1M    /var/lib/infobright/syslogs/raw_201502.bht
  2.1M    /var/lib/infobright/syslogs
  92.0K    /var/lib/infobright/sys_infobright
  4.0K    /var/lib/infobright/test
  4.0K    /var/lib/infobright/cache
  236.0K    /var/lib/infobright/BH_RSI_Repository
  3.5M    /var/lib/infobright
  +++++++++++++++++++++++++++ /opt/GMSVP/syslogs folder usage +++++++++++++++++++++++++++
  4.0K    /opt/GMSVP/syslogs/badSyslogs
  2.9M    /opt/GMSVP/syslogs/archivedSyslogs
  2.9M    /opt/GMSVP/syslogs
  +++++++++++++++++++++++++++ /opt/GMSVP/Temp folder usage +++++++++++++++++++++++++++
  4.0K    /opt/GMSVP/Temp/Data__offloader__d52161aa9daf422fb1043a26a6dca7e1__20150216192617
  4.0K    /opt/GMSVP/Temp/Data__offloader__f180168a7f64432d95d1e4f3ec85b28a__20150216193634
  4.0K    /opt/GMSVP/Temp/Data__offloader__11fa63ef0db34cc1a860f8ea7d04c322__20150216193643
  4.0K    /opt/GMSVP/Temp/optimizer
  4.0K    /opt/GMSVP/Temp/Data__offloader__a3ad7d5f235f4f069ca6404b5dd142b0__20150216201013
  4.0K    /opt/GMSVP/Temp/Data__offloader__8842ae6105d2444b92604b05178e7f41__20150216192628
  4.0K    /opt/GMSVP/Temp/Data__offloader__412d414f55824df589e8bc88f67021e4__20150216202941
  4.0K    /opt/GMSVP/Temp/Data__offloader__5d7d450c20484ff5bd54ab8425971b41__20150216194032
  4.0K    /opt/GMSVP/Temp/Data__offloader__84689c8f4bb6452caea7f5756a2f83d2__20150216195729
  4.0K    /opt/GMSVP/Temp/Data__offloader__3258430e56a34dd1b5ae2b62c5d99811__20150216192635
  4.0K    /opt/GMSVP/Temp/Data__offloader__3d4c1e4ea8f34f708f30d30855667444__20150216201922
  4.0K    /opt/GMSVP/Temp/Data__offloader__6bd5178a9ef04ebdb8adebdf11a5aeb9__20150216201433
  4.0K    /opt/GMSVP/Temp/Data__offloader__5d06f991f9244689a72435911d0c4228__20150216192443
  4.0K    /opt/GMSVP/Temp/Data__offloader__0eb18e7e5313452d91807621ffc60c05__20150216195427
  80.0K    /opt/GMSVP/Temp
  +++++++++++++++++++++++++++ /opt/GMSVP/backup folder usage +++++++++++++++++++++++++++
  4.0K    /opt/GMSVP/backup/temp
  24.0K    /opt/GMSVP/backup
  ++++++++++++++++++++++++++++ End of Disk Usage Diagnostics ++++++++++++++++++++++++++++
  
   

Troubleshooting (Identifying and troubleshooting issues)

[installDir]:GMSVPsyslogs
/opt/GMSVP/syslogs

• Since this a working directory for syslogs; syslog files should be received, processed and moved to another directory for more permanent storage.
• Under normal conditions there should be very few items in this directory including.

1. One .log file used to write newly received syslogs
2. archivedSyslogs and badSyslogs sub-directories
3. Few possible .src or .unp files; typically < 10 files (file extensions seen during processing)

• There may be times that .src and/or .unp file counts will increase such as busy periods of incoming syslogs.
• As long as files continue to process and the file count drops over time, this does not indicate a problem.
• To better understand the process of Syslog Summarization, click here.
• If issues are seen, use the following steps to try and resolve the issue

• Restart the following services or reboot server.
  

1. Syslog Collector
2. Reports Database
3. Reports Summarizer
   
   

   TIP: Possible related KB's:Syslogs are being placed in BadSyslog folder with .pre extension .
   
   

• Same issue can occur without files being moved to badSyslogs directory; causing files to backup in syslogs directory.
  
  [installDir]:GMSVPsyslogsarchivedSyslogs
/opt/GMSVP/syslogs/archivedSyslogs
  
  
• After syslog files are processed successfully, the final .prd files will be stored in this directory.
• The storage configuration specified in the above description controls how long this information is stored.
• This information is only deleted on a monthly basis according to the Data Deletion Schedule/Data Storage Config located on the Console tab under Reports|Summarizer.
  
  
• Issues can be seen in this directory if data deletion fails at any point in time, resulting in data being stored longer than what is configured.
• If the size of this directory becomes a concern, you will need to verify the contents of this directory for items older the than the Data Storage Config for 'Keep Raw Syslog Data Files for' option.
• If there are no files older than the specified configuration, you will need to either increase the amount of disk space available to the server or alter the Data Storage Config to store for a lesser timeframe.
• If data deletion issues are seen, use the following steps.

1. Upgrade the server to the latest Service Packs and hotfixes.  Check again after the next data deletion schedule to see if data is removed.
   
                                                                                                    or
2. Manually delete the older .zip files containing .prd files older than storage config - (workaround).
   
                                                                                                   or
3. Restart the following services.  Check again after the next data deletion schedule to see if data is removed
   
   Reports Database
   Reports Summarizer
   
   [installDir]:GMSVPsyslogsbadSyslogs
/opt/GMSVP/syslogs/badSyslogs

• Files that fail to upload to the Reports Database are stored in this directory permanently.
• Initially the file will be written with a .pre extension, but files are periodically grouped in a .zip file
• The most common issue causing this problem can be resolved using the following KB:https://www.sonicwall.com/support/knowledge-base/syslogs-are-going-to-wrong-syslogs-folder-on-analyzer/170502778984237/.
• Occasionally, few syslog files processed by the Reports Summarizer may have issues causing the file or syslog message to be corrupt.
• These few files will be stored permanently as not configuration is available is scheduled to automatically remove them.
• The contents of this directory can be deleted manually, however, failure to correct any issues causing all syslog files (including non-corrupted files) to be written to this directory and reprocessing these files will result in loss of reporting information.
  
  [installDir]:GMSVPbackup
/opt/GMSVP/backup
  
  
• GMS/Analyzer can perform automated backups on the server in the event of failure.
• There are three different types of backups that can be performed.
• The Basic and Application backups are typically smaller than 2 GB in size, however, the Complete backups may be quite large (dependent on amount of data stored on the server).
• By default, GMS/Analyzer will only store 1 copy of each type of backup (previous backups will be overwritten).
• If this directory appears to be larger than expected, verifying the directory used to store the backups is needed (specified in backup config - screen shot in above description).

Common issues seen with the backups directory

• Backups from previous GMS version are not deleted during the upgrade

Resolution

• Manually delete these backup files from the directory.
  
  
  
• Failures in the backup process result in multiple backup files in the specified directory.

Possible Resolution/Workaround

• Manually delete these backup files from the directory.
• Upgrade to the latest Service Packs on hotfixes and monitor the next backup process.
  
  [installDir]:GMSVPTemp
/opt/GMSVP/Temp
  
  
• The temp directory may house many different file types, all of which should be purged automatically (except GMSVPTempoptimizer sub-directory).
• File types include
  
  .sqlload
  .sqlload.err
  .reso
  
  
• Service interruption or failure on the server may result in many files being left in this directory.
• In general, all files older than the current date can be deleted, however, may not resolve the issue with the files not being purged.
• Failure to resolve the primary issue may result in files building up again in coming days/weeks.

Typical resolutions include

.sqlload
.sqlload.err

• Issues with these file types are typically caused by the same issues corrected in the following KB article:https://www.sonicwall.com/support/knowledge-base/syslogs-are-going-to-wrong-syslogs-folder-on-analyzer/170502778984237/.
• Once this issue is corrected, these file types can be deleted and monitored to verify issue is resolved.
  
  .reso
  
  
• The .reso files are created when the following options are enabled on the Console tab under Reports|Summarizer.
  
  
  
  
  
• Disabling these options and rebooting the server will resolve the issue, however, .reso files will need to be deleted manually.
  
  [installDir]:GMSVPTempoptimizer
/opt/GMSVP/Temp/optimizer
  
  
• This is a temp working directory used by the Reports Summarizer service.
• Large .txt files may be seen in this directory if a failure occurs during the optimization process.
• These files will need to be deleted manually.
• Additional issues may be seen if there is not enough disk space to perform the data optimization task.
• The working space required for data optimization is estimated by the formula.
  
  Higher or [Number of millions syslogs per day x 2GB] and [20 GB]
  Number of millions syslog per day is shown on Console tab under Diagnostics|Summarizer Status

EXAMPLE:

• [Higher of 30 million syslogs per day x 2 GB] or [20GB].
• Higher of [30x2] or [20].
• 60 GB of space is the minimum needed for a server averaging 30 million syslogs received per day.
  
  [installDir]:GMSVPinfobrightdatasyslogs
/var/lib/infobright/syslogs
  
  
• This directory stores the data processed into the Reports Database during the Summarizer process.
• All reporting data is stored according to the Data Storage Config as described in the above descriptions.
• This data is deleted on a monthly basis similar to data stored in the archivedSyslogs directory.
• Using the Tech Support Report is a quick way to confirm how much data is currently store within the Reports Database.

Excerpt from TSR

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Table sizes++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+----------------+-------------+--------------------+-------------+---------------------------------------------------------+
| Table          | Rows        | Average Row Length | Size        | Compression                                             |
+----------------+-------------+--------------------+-------------+---------------------------------------------------------+
| raw_201411     | 220,230,205 | 34                 | 7,163.99 MB | Overall compression ratio: 276.867, Raw size=1983469 MB |
| raw_201412     | 218,264,219 | 26                 | 5,477.70 MB | Overall compression ratio: 358.872, Raw size=1965794 MB |
| summary_status | 46          | 36                 | 0.00 MB     |                                                         |
+----------------+-------------+--------------------+-------------+---------------------------------------------------------+

Naming Convention for Reports Database tables:

raw_201411 = raw_yyyymm = Reports Table for November, 2014

• The above excerpt represents a default configuration of 1 month of storage.
• In GMS, since data is stored on a monthly basis, 1 month of data is equal to 'Current Month+1 previous month' (2 months = Current Month+2 previous months).
• If the monthly data being stored matches that of the storage configuration, additional disk space will need to be added to the server.
• Alternatively, data storage config can be lowered in order to store less data and free up disk space during next data deletion cycle.
• If data stored is older than the data storage config for Reporting Data, please test the following:

1. Upgrade to the latest service packs and hotfixes.
   
                                                 or
2. Restart the Reports Summarizer service (responsible for data deletion) and wait for next data deletion cycle.