Site Navigation: Servers and hardware: Failover servers

Failover recording servers (explained)

A failover recording server is an extra recording server which takes over from the standard recording server if this becomes unavailable. You can configure a failover recording server in two modes, as a cold standby server or as a hot standby server.

You install failover recording servers like standard recording servers (see Install new XProtect components). Once you have installed failover recording servers, they are visible in the Management Client. Milestone recommends that you install all failover recording servers on separate computers. Make sure that you configure failover recording servers with the correct IP address/host name of the management server. The user rights for the user account under which the Failover Server service runs are provided during the installation process. They are:

• Start/Stop permissions to start or stop the failover recording server
• Read and Write access permissions to read or write the RecorderConfig.xml file

If a certificate is selected for encryption, then the administrator must grant read access permission to the failover user on the selected certificate private key.

If the failover recording server takes over from a recording server that uses encryption, Milestone recommends that you also prepare the failover recording server for using encryption. For more information, see Before you start installation and Install new XProtect components.

You can specify what type of failover support you want on device-level. For each device on a recording server, select full, live only or no failover support. This helps you prioritize your failover resources and, for example, only set up failover for video and not for audio, or only have failover on essential cameras, not on less important ones.

While your system is in failover mode, you cannot replace or move hardware, update the recording server, or change device configurations such as storage settings or video stream settings.

Cold standby failover recording servers

In a cold standby failover recording server setup, you group multiple failover recording servers in a failover group. The entire failover group is dedicated to take over from any of several preselected recording servers, if one of these becomes unavailable. You can create as many groups as you want (see Group failover recording servers for cold standby).

Grouping has a clear benefit: when you later specify which failover recording servers should take over from a recording server, you select a group of failover recording servers. If the selected group contains more than one failover recording server, this offers you the security of having more than one failover recording server ready to take over if a recording server becomes unavailable. You can specify a secondary failover server group that takes over from the primary group if all the recording servers in the primary group are busy. A failover recording server can only be a member of one group at a time.

Failover recording servers in a failover group are ordered in a sequence. The sequence determines the order in which the failover recording servers will take over from a recording server. By default, the sequence reflects the order in which you have incorporated the failover recording servers in the failover group: first in is first in the sequence. You can change this if you need to.

Hot standby failover recording servers

In a hot standby failover recording server setup, you dedicate a failover recording server to take over from one recording server only. Because of this, the system can keep this failover recording server in a "standby" mode which means that it is synchronized with the correct/current configuration of the recording server it is dedicated to and can take over much faster than a cold standby failover recording server. As mentioned, you assign hot standby servers to one recording server only and cannot group it. You cannot assign failover servers that are already part of a failover group as hot standby recording servers.

Failover steps (explained)

Description
Involved servers (numbers in red): Recording Server Failover Recording Server Management Server
Failover steps for Cold standby setups: To check whether it is running or not, a failover recording server has a non-stop TCP connection to a recording server. This connection is interrupted. The failover recording server requests the current configuration of the recording server from the management server. The management server sends the requested configuration, the failover recording server receives the configuration, starts up, and starts recording on behalf of the recording server. The failover recording server and the relevant camera(s) exchange video data. The failover recording server continually tries to re-establish connection to the recording server. When the connection to the recording server is re-established, the failover recording server shuts down and the recording server fetches video data (if any) recorded during its down-time and the video data is merged back in to the recording server database.
Failover steps for Hot standby setups: To check whether it is running or not, a hot standby server has a non-stop TCP connection to its assigned recording server. This connection is interrupted. From the management server, the hot standby server already knows the current configuration of its assigned recording server and starts recording on its behalf. The hot standby server and the relevant camera(s) exchange video data. The hot standby server continually tries to re-establish connection to the recording server. When the connection to the recording server is re-established and the hot standby server goes back to hot standby mode, the recording server fetches video data (if any) recorded during its down-time and the video data is merged back in to the recording server database.

Failover recording server functionality (explained)

• A failover recording server checks the state of relevant recording servers every 0.5 seconds. If a recording server does not reply within 2 seconds, the recording server is considered unavailable and the failover recording server takes over
• A cold standby failover recording server takes over for the recording server that has become unavailable after five seconds plus the time it takes for the failover recording server's Recording Server service to start and the time it takes to connect to the cameras. In contrast, a hot standby failover recording server takes over faster because the Recording Server service is already running with the correct configuration and only has to start its cameras to deliver feeds. During the startup period, you can neither store recordings nor view live video from affected cameras
• When a recording server becomes available again, it automatically takes over from the failover recording server. Recordings stored by the failover recording server are automatically merged into the standard recording server's databases. The time it takes to merge, depends on the amount of recordings, network capacity and more. During the merging process, you cannot browse recordings from the period during which the failover recording server took over
• If a failover recording server must take over from another recording server during the merging process in a cold standby failover recording server setup, it postpones the merging process with recording server A, and takes over from recording server B. When recording server B becomes available again, the failover recording server takes up the merging process with recording server A, after which it begins merging with recording server B.
• In a hot standby setup, a hot standby server cannot take over for an additional recording server because it can only be hot standby for a single recording server. But if that recording server fails again, the hot standby takes over again and keeps the recordings from the previous period. The recording server keeps recordings until they are merged back to the primary recorder or until the failover recording server runs out of disk space
• A failover solution does not provide complete redundancy. It can only serve as a reliable way of minimizing the downtime. If a recording server becomes available again, the Failover Server service makes sure that the recording server is ready to store recordings again. Only then is the responsibility for storing recordings handed back to the standard recording server. So, a loss of recordings at this stage of the process is very unlikely
• Client users hardly notice that a failover recording server is taking over. A short break occurs, usually only for a few seconds, when the failover recording server takes over. During this break, users cannot access video from the affected recording server. Client users can resume viewing live video as soon as the failover recording server has taken over. Because recent recordings are stored on the failover recording server, they can play back recordings from after the failover recording server took over. Clients cannot play back older recordings stored only on the affected recording server until that recording server is functioning again and has taken over from the failover recording server. You cannot access archived recordings. When the recording server is functioning again, a merging process takes place during which failover recordings are merged back into the recording server's database. During this process, you cannot play back recordings from the period during which the failover recording server took over
• In a cold standby setup, setting up a failover recording server as backup for another failover recording server is not necessary. This is because you allocate failover groups and do not allocate particular failover recording servers to take over from specific recording servers. A failover group must contain at least one failover recording server, but you can add as many failover recording servers as needed. If a failover group contains more than one failover recording server, more than one failover recording server can take over.
• In a hot standby setup, you cannot set up failover recording servers or hot standby servers as failover for a hot standby server

Set up and enable failover recording servers

If you have disabled the failover recording server, you must enable it before it can take over from the standard recording servers.

Do the following to enable a failover recording server and edit its basic properties:

1. In the Site Navigation pane, select Servers > Failover Servers. This opens a list of installed failover recording servers and failover groups.
2. In the Overview pane, select the required failover recording server.
3. Right-click and select Enabled. The failover recording server is now enabled.
4. To edit failover recording server properties, go to the Info tab.
5. When done, go to the Network tab. Here you can define the failover recording server's public IP address and more. This is relevant if you use NAT (Network Address Translation) and port forwarding. See the standard recording server's Network tab for more information.
6. In the Site Navigation pane, select Servers > Recording Servers. Select the recording server that you want failover support for and assign failover recording servers (see Failover tab (recording server)).

To see the status of a failover recording server, hold your mouse over the Failover Recording Server Manager tray icon in the notification area. A tooltip appears containing the text entered in the Description field of the failover recording server. This may help you determine which recording server the failover recording server is configured to take over from.

The failover recording server pings the management server on a regular basis to verify that it is online and able to request and receive the configuration of the standard recording servers when needed. If you block the pinging, the failover recording server is not able to take over from the standard recording servers.

Group failover recording servers for cold standby

1. Select Servers > Failover Servers. This opens a list of installed failover recording servers and failover groups.
2. In the Overview pane, right-click the top-node Failover Groups and select Add Group.
3. Specify a name (in this example Failover Group 1) for and a description (optional) of your new group. Click OK.
4. Right-click the group (Failover Group 1) you just created. Select Edit Group Members. This opens the Select Group Members window.
5. Drag and drop or use the buttons to move the selected failover recording server(s) from the left side to the right side. Click OK. The selected failover recording server(s) now belongs to the group (Failover Group 1) you just created.
6. Go to the Sequence tab. Click Up and Down to set the internal sequence of the regular failover recordings servers in the group.

Read failover recording server status icons

The following icons represent the status of failover recording servers (icons are visible in the Overview pane):

Icon	Description
	The failover recording server is either waiting or "watching". When waiting, the failover recording server is not configured to take over from any recording server yet. When "watching", the failover recording server is configured to watch one or more recording servers.
	The failover recording server has taken over from the designated recording server. If you place your cursor over the server icon, you see a tooltip. Use the tooltip to see which recording server the failover recording server has taken over from.
	Connection to the failover recording server is broken.

Multicast tab (failover server)

If you are using failover servers, and you have enabled multicasting of live streaming, you must specify the IP address of the network interface card you are using, on both the recording servers and the failover servers.

For more information about multicasting, see Multicast tab (recording server) or Multicast tab (recording server).

Info tab properties (failover server)

Specify the following failover recording server properties:

Name	Description
Name	The name of the failover recording server as it appears in the Management Client, logs and more.
Description	An optional field that you can use to describe the failover recording server, for example which recording server it takes over from.
Host name	Displays the failover recording server's host name. You cannot change this.
Local web server address	Displays the local address of the failover recording server's web server. You use the local address, for example, for handling PTZ camera control commands, and for handling browsing and live requests from XProtect Smart Client. The address includes the port number that is used for web server communication (typically port 7563). If the failover recording server takes over from a recording server that uses encryption, you also need to prepare the failover recording server to use encryption. If you enable encryption to clients and servers that retrieve data streams from the recording server, a padlock icon appears, and the address includes https instead of http.
Web server address	Displays the public address of the failover recording server's web server on the internet. If your installation uses a firewall or NAT router, enter the address of the firewall or NAT router so that clients that access the surveillance system on the internet can connect to the failover recording server. You specify the public address and port number on the Network tab. If you enable encryption to clients and servers that retrieve data streams from the recording server, a padlock icon appears, and the address includes https instead of http.
UDP port	The port number used for communication between failover recording servers. Default port is 8844.
Database location	Specify the path to the database used by the failover recording server for storing recordings. You cannot change the database path while the failover recording server is taking over from a recording server. The system applies the changes when the failover recording server is no longer taking over from a recording server.
Enable this failover server	Clear to disable the failover recording server (selected by default). You must disable failover recording servers before they can take over from recording servers.

Info tab properties (failover group)

Field	Description
Name	The name of the failover group as it appears in the Management Client, logs and more.
Description	An optional description, for example the server's physical location.

Sequence tab properties (failover group)

Field	Description
Specify the failover sequence	Use Up and Down to set the wanted sequence of regular failover recording servers within the group.

Failover recording server services (explained)

A failover recording server has two services installed:

• A Failover Server service, which handles the processes of taking over from the recording server. This service is always running, and constantly checks the state of relevant recording servers
• A Failover Recording Server service, which enables the failover recording server to act as a recording server.

  In a cold standby setup, this service is only started when required, that is when the cold standby failover recording server takes over from the recording server. Starting this service typically takes a couple of seconds, but may take longer depending on local security settings and more.
  In a hot standby setup, this service is always running, allowing the hot standby server to take over faster than the cold standby failover recording server.

View encryption status on a failover recording server

To verify if your failover recording server uses encryption, do the following:

1. In the Site Navigation pane, select Servers > Failover Servers. This opens a list of failover recording servers.
2. In the Overview pane, select the relevant recording server and go to the Info tab.
   If encryption is enabled to clients and servers that retrieve data streams from the recording server, a padlock icon appears in front of the local web server address and the optional web server address.

View status messages

1. On the failover recording server, right-click the Milestone Failover Recording Server service icon.
2. Select Show Status Messages. The Failover Server Status Messages window appears, listing time-stamped status messages.

View version information

Knowing the exact version of your Failover Recording Server service is an advantage if you need to contact product support.

1. On the failover recording server, right-click the Milestone Failover Recording Server service icon.
2. Select About.
3. A small dialog box opens that shows the exact version of your Failover Recording Server service.