High Availability

Configure High Availability Telephony server

The ScopTEL IP PBX supports different methods for High Availability and replication of MySQL databases; this article will explain the easiest way to configure two (2) servers in a failover (active/passive) scenario.

We will show how to replicate the configuration, voicemail, and prompts from the Master with the IP address 192.168.99.164 to a Slave (192.168.99.165) using a Floating IP Address (192.168.99.100).

 

Requirements

Both servers must use the ScopServ installation disk (ISO).  The image is available to download at http://download.scopserv.com/iso/

The shared partition must exist on both servers as a primary partition and the mount point must be /share

In order to simplify the configuration of the High Availability setup, we automatically created a shared 10GB partition on the disk layout.

We highly recommend not to exceed a shared partition of 10Gb due to the length of time required to sync the shared partition.  The maximum shared partition size is 50Gb

You must absolutely use  Server version 2.6.9 or greater and Telephony version 2.7.16 or greater.

 

Recommendations

We highly recommend that you update all packages to the latest versions. To proceed, go to Server -> Packages Manager and click on Update Now.

We strongly recommend reading the ‘DRBD User’s Guide’ available at http://www.drbd.org/users-guide-8.3/ that covers all aspect of Shared Storage (DRBD), including troubleshooting informations.

Update Now on Packages Manager

Preparing your Network Configuration

It is recommended, though not strictly required, that you run your Shared Storage (DRBD) replication over a dedicated connection. The most reasonable choice for this is a direct, back-to-back, Gigabit Ethernet connection. When the Shared Storage (DRBD) service is run over switches, use of redundant components is recommended.

It is generally not recommended to run Shared Storage (DRBD) replication via routers, for reasons of fairly obvious performance drawbacks (adversely affecting both throughput and latency).

 

– The Floating IP Address that will be used by VoIP devices is set to 192.168.99.100

– The Master Server is configured to use the hostname master.local and the IP Address 192.168.99.164

– The Slave Server will use slave.local as the hostname and the IP Address 192.168.99.165

It is very important that both servers can ping each other using the hostname and IP address. On both servers, go to Network -> Configuration and verify that the specified hostname matches. On Network -> Static Hosts,  you can create a hostname for each server.

Network Static Hosts configuration

MySQL Configuration

On both servers, go on Server -> MySQL Server -> Configuration and check the option Enable High Availability Support and click Save (do not change any of the default settings).

 

High Availability Configuration

First, we have to edit the High Availability configuration on the GUI of BOTH servers. We have to enable Automatic Failover (Heartbeat) and Shared Network Storage (DRBD) on Server -> High Availability.

We have to specify the Floating IP Address (192.168.99.100), the Primary (192.168.99.164), and Secondary (192.168.99.165) Server IP Addresses, and Hostnames.

For Shared Storage, you will have to specify the disk partition used for the shared storage. From SSH, you can use the df  command to retrieve the disk device. In this sample, the shared device is /dev/hda2

[note color=#ddd]# df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 65164472 2303744 59497120 4% /
/dev/hda1 101086 17217 78650 18% /boot
/dev/hda2 10080168 153732 9414384 2% /share[/note]

Make sure the High Availability configuration is correctly set as follows:

High Availability Configuration on Master and Slave servers

 

On the Options tab, you must absolutely enable the Telephony Server (Asterisk) and MySQL Database modules. It is recommended to enable all modules for a complete replication.

High Availability Options (Modules)

High Availability Options (Modules)

Before continuing, verify on both servers that directory /share exists. If it’s not the case, create it (located at root) using the command mkdir share, and affect it the right owner user/group with chown -R scopserv:scopserv /share

Reboot both servers to affect all network changes.

Enabling Services and Initializing Shared Storage

Under Configuration -> Server -> General, make sure that  Heartbeat (Failover) and Shared Storage (DRBD) services are enabled. You must start these services on both servers.

Once the services are running, you must initialize the Shared Storage partition. On the Master server, click on Initialize DRBD and a popup window will appear and ask you to confirm the initialization. You must wait for completion before executing the next step. Please be patient, completion of this step can take approximately 5-10 minutes.

Initialize DRBD on the Master server

 

Now that the initialization is complete on the Master server, you can go to the Slave server, click on Initialize DRBD and the following popup window will appear:

Initialize DRBD on the Master server

 

Services Status and Verification

The High Availability and Shared Storage are configured and the synchronization of the disk must be in progress (see following image).

When the synchronization is complete, the status available on Server -> General must look like the following:

At this point, HA is configured and running. It’s now time to make some modifications in configuration to affect floating IP address.

On both servers:

  • Server -> Configuration -> Provisioning:
    • SIP Server Address
    • TFTP Provisioning
    • HTTP Provisioning

On Master Server only:

  • Telephony -> Configuration -> Provisioning: Default SIP Server
  • Telephony -> Configuration -> Channels:
    • SIP Channel: Binding Address
    • IAX Channel: Address
Read More

Enable Automatic Failover (Heartbeat) with ScopTEL IP PBX

How to Enable Automatic Failover ?

This article illustrates how to implementing VERY simple active/passive IP failover using ScopTEL IP PBX.

The Automatic Failover (heartbeat) module allows two (2) servers to share an high availability IP address.

Each server has its own normal IP address used to administer the server. There is then a 3nd  Floating IP address that SIP/IAX2 clients and SIP Gateways/Trunks connect to. This normally runs on the primary server as an IP alias for the WAN (eth0:0) Interface.

The backup (slave) server then monitors the health of the primary server, and if it crashes the backup (slave) takes over the service IP address. The backup monitors the primary through the LAN network and optionally through a null modem cable connected between the serial ports on each server.

 

For the purposes of this article we will have two (2) servers, and three (3) IP addresses, for this setup we have two (2) internal/LAN IP’s and one (1) Public/WAN IP address.

The IP/Hostnames/DNS we will be using are as follows:

  • pbx.dr.scopserv.com: 205.237.45.129 (WAN) (Floating IP, not a physical server/node)
  • pbx01.dr.scopserv.com 192.168.120.101 (LAN) (Server/Node 1)
  • pbx02.dr.scopserv.com 192.168.120.102 (LAN) (Server/Node 2)

 

To configure Heartbeat on ScopTEL PBX is pretty easy. Do the steps below on both servers :

  • Log into ScopTEL PBX GUI as administrator
  • Go on Configuration -> Server -> High Availability module.
  • Click on Edit and check the Enable Automatic Failover (Heartbeat) option.
  • Set Floating IP Address used by SIP/IAX2 clients and/or Gateways/Trunks.
  • Define the IP Address and Hostname for both servers.

You also have others options to manage the Failover (Heartbeat) service. By sample, if you want to use a null model cable instead of Network  broadcast, you can change the Interface Mode to Serial Port.

 

Server -> High Availability -> Heartbeat

Server -> High Availability -> Heartbeat

 

That’s it!!! Fire up both servers/nodes, pull the plug on your primary node (server 1), and check backup server (server 2) to see that it has taken over your High Availability services and Floating IP Address.

 

Read More

How to recover Shared Storage (DRBD) from split brain

After split brain has been detected, one node will always have the resource in a StandAlone connection state. The other might either also be in the StandAlone state (if both nodes detected the split brain simultaneously), or in WFConnection (if the peer tore down the connection before the other node had a chance to detect split brain).

 

Shared Storage (DRBD) Status

Shared Storage (DRBD) Status

 

If you get degraded Shared Storage (DRBD) you have to manually resolve split brain situation. You will need to log into Master/Slave server using SSH and connect as root.

 

On Master (primary) server :

[note color=#ddd]drbdadm connect all [/note]

 

On Backup (secondary) server :

[note color=#ddd]drbdadm invalidate all
drbdadm connect all
[/note]

 

Upon connection, your split brain victim immediately changes its connection state to SyncTarget, and has its modifications overwritten by the remaining primary node.

Read More