Failover recording server (explained)

Available functionality depends on the system you are using. See the complete feature list, which is available on the product overview page on the Milestone website (https://www.milestonesys.com/products/software/xprotect-comparison/).

A failover recording server is an extra recording server which takes over from the standard recording server if this becomes unavailable. You can configure a failover recording server in two modes, as a cold standby server or as a hot standby server.

You install failover recording servers like standard recording servers (see Install a failover recording server through Download Manager). Once you have installed failover recording servers, they are visible in the Management Client. Milestone recommends that you install all failover recording servers on separate computers. Make sure that you configure failover recording servers with the correct IP address/host name of the management server. The user permissions for the user account under which the Failover Server service runs are provided during the installation process. They are:

  • Start/Stop permissions to start or stop the failover recording server
  • Read and Write access permissions to read or write the RecorderConfig.xml file

If a certificate is selected for encryption, then the administrator must grant read access permission to the failover user on the selected certificate private key.

If the failover recording server takes over from a recording server that uses encryption, Milestone recommends that you also prepare the failover recording server for using encryption. For more information, see Secure communication (explained) and Install a failover recording server through Download Manager.

You can specify what type of failover support you want on device-level. For each device on a recording server, select full, live only or no failover support. This helps you prioritize your failover resources and, for example, only set up failover for video and not for audio, or only have failover on essential cameras, not on less important ones.

While your system is in failover mode, you cannot replace or move hardware, update the recording server, or change device configurations such as storage settings or video stream settings.

Cold standby failover recording servers

In a cold standby failover recording server setup, you group multiple failover recording servers in a failover group. The entire failover group is dedicated to take over from any of several preselected recording servers, if one of these becomes unavailable. You can create as many groups as you want (see Group failover recording servers for cold standby).

Grouping has a clear benefit: when you later specify which failover recording servers should take over from a recording server, you select a group of failover recording servers. If the selected group contains more than one failover recording server, this offers you the security of having more than one failover recording server ready to take over if a recording server becomes unavailable. You can specify a secondary failover server group that takes over from the primary group if all the recording servers in the primary group are busy. A failover recording server can only be a member of one group at a time.

Failover recording servers in a failover group are ordered in a sequence. The sequence determines the order in which the failover recording servers will take over from a recording server. By default, the sequence reflects the order in which you have incorporated the failover recording servers in the failover group: first in is first in the sequence. You can change this if you need to.

Hot standby failover recording servers

In a hot standby failover recording server setup, you dedicate a failover recording server to take over from one recording server only. Because of this, the system can keep this failover recording server in a "standby" mode which means that it is synchronized with the correct/current configuration of the recording server it is dedicated to and can take over much faster than a cold standby failover recording server. As mentioned, you assign hot standby servers to one recording server only and cannot group it. You cannot assign failover servers that are already part of a failover group as hot standby recording servers.

Failover recording server validation

To validate a merge of video data from the failover server to the recording server, you must make the recording server unavailable by either stopping the recording server service or shutting down the recording server computer.

Any manual interruption of the network that you can cause by pulling out the network cable or blocking the network using a test tool is not a valid method.

Failover recording server functionality (explained)

  • A failover recording server checks the state of relevant recording servers every 0.5 seconds. If a recording server does not reply within 2 seconds, the recording server is considered unavailable and the failover recording server takes over
  • A cold standby failover recording server takes over for the recording server that has become unavailable after five seconds plus the time it takes for the failover recording server's Recording Server service to start and the time it takes to connect to the cameras. In contrast, a hot standby failover recording server takes over faster because the Recording Server service is already running with the correct configuration and only has to start its cameras to deliver feeds. During the startup period, you can neither store recordings nor view live video from affected cameras
  • When a recording server becomes available again, it automatically takes over from the failover recording server. Recordings stored by the failover recording server are automatically merged into the standard recording server's databases. The time it takes to merge, depends on the amount of recordings, network capacity and more. During the merging process, you cannot browse recordings from the period during which the failover recording server took over
  • If a failover recording server must take over from another recording server during the merging process in a cold standby failover recording server setup, it postpones the merging process with recording server A, and takes over from recording server B. When recording server B becomes available again, the failover recording server takes up the merging process and allows both recording server A and recording server B to merge back recordings simultaneously.
  • In a hot standby setup, a hot standby server cannot take over for an additional recording server because it can only be hot standby for a single recording server. But if that recording server fails again, the hot standby takes over again and keeps the recordings from the previous period. The recording server keeps recordings until they are merged back to the primary recorder or until the failover recording server runs out of disk space
  • A failover solution does not provide complete redundancy. It can only serve as a reliable way of minimizing the downtime. If a recording server becomes available again, the Failover Server service makes sure that the recording server is ready to store recordings again. Only then is the responsibility for storing recordings handed back to the standard recording server. So, a loss of recordings at this stage of the process is very unlikely
  • Client users hardly notice that a failover recording server is taking over. A short break occurs, usually only for a few seconds, when the failover recording server takes over. During this break, users cannot access video from the affected recording server. Client users can resume viewing live video as soon as the failover recording server has taken over. Because recent recordings are stored on the failover recording server, they can play back recordings from after the failover recording server took over. Clients cannot play back older recordings stored only on the affected recording server until that recording server is functioning again and has taken over from the failover recording server. You cannot access archived recordings. When the recording server is functioning again, a merging process takes place during which failover recordings are merged back into the recording server's database. During this process, you cannot play back recordings from the period during which the failover recording server took over
  • In a cold standby setup, setting up a failover recording server as backup for another failover recording server is not necessary. This is because you allocate failover groups and do not allocate particular failover recording servers to take over from specific recording servers. A failover group must contain at least one failover recording server, but you can add as many failover recording servers as needed. If a failover group contains more than one failover recording server, more than one failover recording server can take over.
  • In a hot standby setup, you cannot set up failover recording servers or hot standby servers as failover for a hot standby server

Failover steps (explained)

A failover recording server takes over from an unavailable recording server in the XProtect VMS.

Description

Involved servers (numbers in blue):

  1. Recording Server
  2. Failover Recording Server
  3. Management Server

Failover steps for Cold standby setups:

  1. To check whether it is running or not, a failover recording server has a non-stop TCP connection to a recording server.
  2. This connection is interrupted.
  3. The failover recording server requests the current configuration of the recording server from the management server. The management server sends the requested configuration, the failover recording server receives the configuration, starts up, and starts recording on behalf of the recording server.
  4. The failover recording server and the relevant camera(s) exchange video data.
  5. The failover recording server continually tries to re-establish connection to the recording server.
  6. When the connection to the recording server is re-established, the failover recording server shuts down and the recording server fetches video data (if any) recorded during its downtime and the video data is merged back in to the recording server database.

Failover steps for Hot standby setups:

  1. To check whether it is running or not, a hot standby server has a non-stop TCP connection to its assigned recording server.
  2. This connection is interrupted.
  3. From the management server, the hot standby server already knows the current configuration of its assigned recording server and starts recording on its behalf.
  4. The hot standby server and the relevant camera(s) exchange video data.
  5. The hot standby server continually tries to re-establish connection to the recording server.
  6. When the connection to the recording server is re-established and the hot standby server goes back to hot standby mode, the recording server fetches video data (if any) recorded during its down-time and the video data is merged back in to the recording server database.