Troubleshooting

Once an XProtect VMS is deployed and running, it may be needed to investigate and troubleshoot the recording server and storage performance if issued are experienced.

According to Milestone Support, performance issues often originate in the storage system not being able to cope with the load the XProtect VMS puts on it.

To investigate if the storage system of a particular system is overloaded, “Windows Perfmon” can be started on the server and configured to capture and save values for below counters for an extended period.

Perfmon counters

Recommendation

Processor Information(_Total)\% Processor Time

Should not exceed 80%

PhysicalDisk(<DISK>)\% Idle Time

Should be above 20%

PhysicalDisk(<DISK>)\Avg. Disk sec/Read

Below 20 ms

PhysicalDisk(<DISK>)\Avg. Disk sec/Transfer

Below 20 ms

PhysicalDisk(<DISK>)\Avg. Disk sec/Write

Below 20 ms

VideoOS Recording Server Database(_total)\Average GOP Write Time (ms)

Below 20 ms

VideoOS Recording Server Pipeline(_total)\Medias in queue

Between 0 and 100

VideoOS Recording Server Pipeline(_total)\Medias lost/sec

Should be 0 all the time

The results, either as a graph in “Perfmon” or as data in a .csv file format, should be examined to check that the values in general stay within the recommended range, and don’t fluctuate to exceed the recommendations.

If the values indicate that the storage system is overloaded, one of the below cases might be the cause for it:

  • Too many devices are recorded to the storage system

  • The devices are recording with a too high bitrate

  • Dual stream recording have been enabled, increasing the number of media databases being recorded to, and increasing the bitrate being recorded

  • If recording based on motion detection, more cameras detect motion at the same time than anticipated

  • Too many cameras are configured to record MJPEG streams

  • Archiving is enabled, causing increased load during periods when recordings are archived

  • Many users play back, or export recorded media simultaneously

  • Third-party integrations read recorded XProtect VMS media excessively

  • Broken or replaced disks in a RAID system cause performance degradation

To address the experienced issues, it is recommended to implement one or more of the below recommendations even though they may require changing the settings from the preferred settings, and might be costly to implement:

  • Add one or more additional recording servers and move devices from an overloaded recording server to a new recording server until the load has normalized

  • If possible, increase the performance of the storage system – for instance by using a faster RAID configuration, adding more disks, or by using short-stroking

  • Add a second storage system to the recording server and distribute the devices across the two storage systems

  • If archiving is not used, add a smaller and faster storage system to record the live media data, and use the existing one for storing archives

  • Reduce the bitrate of cameras using MPEG-4, H.264 or H.265

  • If dual stream recording is enabled, disable it

  • Reduce the framerate of cameras using MJPEG or change them to use MPEG-4, H.264 or H.265