Telium Support Group

Depending on the combination of Linux, Qt library version, and local timezone (> UTC) this problem might appear. We have modified HAAst to detect the combination of libraries+Linux issue to avoid the problem. Please upgrade to HAAst version 2.3.1.16 or later and the problem should be resolved.

Dual active contention means that both peers in the cluster are active, and one or both peers discover that the other is active as well (they are contending). Upon discovering this situation the peers automatically negotiate which should remain active, and which should demote itself.

At a high level the cause of this problem is that the (previously) standby peer thought that the other peer was dead/unresponsive and needed to take over, so it promoted itself to active. However, the real question that needs answering is why did that peer think the other was dead/unresponsive?

The most common causes are:

1. HAAst Misconfiguration: If one of the peers continually cycles back and forth between active and standby then the most likely cause is peerlink misconfiguration in haast.conf (peer A can talk to peer B, but peer B can’t talk to peer A). To resolve this carefully check the settings in the peerlink stanza of haast.conf. If they look correct perform telnet connectivity tests to the HAAst port from each peer to the other, and continual ping tests (using only management IP’s) between peers and watch what happens before/after a contention.
2. Network Misconfiguration: If one of the peers occasionally cycles back and forth between active and standby then the most likely cause is network misconfiguration. Again, peer A can talk to peer B, but peer B can’t talk to peer A. To resolve this ensure the network settings at the OS level are correct, and the network settings are correct in HAAst.conf (voipnic stanza). This includes checking default routes (which may change if using a shared IP), accidentally reusing an IP address already in use, etc. If they look correct perform continual ping tests (using only management IP’s) between peers and watch what happens before/after a contention.
3. Peer Load/Responsiveness: If one of the peers suffers from periodic extreme load then HAAst will correctly assess its health as failing and allow the other peer to take over. To resolve this problem examine both hosts for CPU load, runaway process, high IO processes, etc. For example, the backup script included in FreePBX is poorly written and will cause very high CPU and/or IO load when it runs (causing the PBX to become unresponsive briefly). To resolve this problem identify the process(es) or device(s) causing the high load and correct their behavior (e.g.: switch to a real backup program).
4. LAN/WAN Latency: In cases where peers are separated by large geographic distances the maximum latency setting in haast.conf may be set too low. On rare occasions, an overloaded or problematic LAN can cause the same problem. Although the root cause of the problem can be accommodated by increasing HAAst’s maximum latency setting, this is not always desirable. Be sure to understand the implications on detection and fail-over time (for legitimate peer failure situations). As well, if you are running the Commercial Unlimited edition of HAAst then latency is already being compensated for dynamically – so the maximum latency setting will reflect how severe the problem really is, and may warrant a general network diagnostic.
5. Network Interruption: This is actually not a problem. It means there was a network outage (one node could not reach the other), and the standby node correctly promoted itself. Once network connectivity was restored, it demoted itself. If this problem occurs rarely then there is nothing you need to do – HAast is doing it’s job! If this problem occurs frequently, then you should investigate a network outage/intermittency.

This type of problem can be one of the most challenging to resolve. You will need to enable full debugging in the HAAst logs, as well as system logs, to capture the data needed to diagnose. You may need to involve your network admin, and possibly your WAN carrier. Telium will often work with clients through SSH to help identify the root cause, and suggest a resolution.

The Free Edition of HAAst will synchronize Asterisk configuration files, but not information held in SQL databases. So your Asterisk configuration files (in /etc/asterisk) will synchronize and the peers will work properly, but if you edit the configuration through FreePBX on one peer you will not see these changes appear in the GUI of the other peer.

To synchronize the SQL database you will need a commercial edition of HAAst.

Assuming this is not a switch issue (which you confirmed by the arp packet missing in the packet capture), we can diagnose the problem as follows:

1. In one shell set a simple packet capture as follows:

   tcpdump -i ethX -vvv -x arp

2. ethX is the ‘physicaldevice’ setting in haast.conf which should match the ethernet adapter you are using for VoIP traffic.

[*]In another shell issue an arping from the command line (exactly as follows):

arping -U -I ethX -c 5 sharedIP
[*]ethX is the ‘physicaldevice’ key set in haast.conf which should match the ethernet adapter you are using for VoIP traffic.
[*]If ‘vlanid’ is set in haast.conf the “.haast” is appended to the ethX adapter name
[*]sharedIP is the IP address moving between peers, set in the ‘address’ key of the ‘voipnic’ stanza of haast.conf
[*]Stop the tcpdump and show the interface details using ifconfig

ifconfig
[*]Post the full output of command and response from all 3 steps above. If any of the information above is security sensitive (eg: a public IP is in the output) email the above to support@autocommander.aws2.ocg.ca , do not obfuscate the data.

If traffic did not start to flow with the above command (and assuming there were no errors reported above), can you try a variation of the arping syntax that consistently works for you? Please post what worked for you.

If the above arping command generated any errors then we can offer a workaround. Although rare, we have seen the above command syntax fail on a Linux distro that customized its arping command. What distro and arping (from what package) are you using? And what exact syntax would allow the arping command to be successful? If we see enough need for that particular arping syntax we’ll add support directly to HAAst. As well, we can offer a workaround if that’s the only arping package available for your distro.

First of all, you should be aware that Fail2Ban is not a security system – it depends completely on Asterisk to say that a user attempted to register/dial without a valid account. Fail2ban has no intrusion detection, no hacking detection, no geofencing, no fraud pattern detection, etc. It is simple a tool that reads log files to determine if an IP should be banned. Digium warns users not to use Fail2Ban as a security measure; see http://forums.asterisk.org/viewtopic.php?p=159984 To underscore Digium’s point, most SIP attacks don’t even show up in the Asterisk log files, so these attackers are not stopped by fail2ban.

Fail2ban is certainly better than nothing – so if you don’t want to use SecAst (even the Free Edition of SecAst), then install fail2ban. If all you want is Asterisk log trolling then SecAst can respond to these same messages from Asterisk if you choose, just like Fail2Ban, but that is among the least significant features of SecAst. SecAst uses event information from the Asterisk AMI, data from the network interface card, SIP data (including dialing digits, rate of dialing, etc), and more to create a profile of each user/device and identify potential hacking and fraud. SecAst also uses proprietary databases of phone numbers used in fraud, known source IP addresses of telecom hackers or intrusion attempts, and all IP addresses mapped to cities/regions/countries/continents worldwide to dramatically reduce the risk of fraud or intrusion. SecAst even uses heuristic detection (like Antivirus software) to identify behavioral patterns indicative of hacking attempts, or indicative of calls being made using stolen credentials. And finally, SecAst continually monitors endpoint activities (even after registration) to protect the PBX and stop fraud.

So comparing Fail2Ban to SecAst is like comparing a screw driver to a toolbox full of tools. Many of our customers have come to SecAst from Fail2Ban after their first $100,000 bill from their ITSP. Products like FreePBX tend to give users a false sense of security by calling Fail2Ban their “security system” – because it’s not. Digium makes it quite clear that if you think Fail2Ban is a security system then you risk being hacked / defrauded.

The problem you are experiencing is most likely due to a PHP caching/optimization program installed in your server. For example, the APC (alternative PHP caching module) has some bugs that will cache an included file and they try to include it again (resulting in a PHP redefinition error, require_once error, etc). For details of the APC bug and possible solutions check out this link: https://pantheon.io/docs/alternative-php-cache/

There are caching modules from other vendors (eg: Zend) with some similar issues. So you may also wish to disable caching of SecAst files since they don’t create much load on a server (relatively static, low volume). This is not a SecAst bug, but future versions of SecAst will try to detect the caching software and work around the issue.

Most Linux distributions are starting to include Qt version 5.7 (as of December 2016) so check available repos (including testing repos) first to ensure it’s not there. If you can find Qt version 5.5 or later as a package it’s best to install from your package manager. Otherwise continue on below.

If you are running Ubuntu you can check this link for a step-by-step guide to installing Qt 5.x: http://sourcedigit.com/19858-how-to-install-qt-5-6-1-on-ubuntu-16-04/

If your system runs headless (i.e. no graphical shell) then you can also modify the Qt installer to run without its GUI as described here:
http://stackoverflow.com/questions/25105269/silent-install-qt-run-installer-on-ubuntu-server

If the above suggestions don’t work, then we recommend you download ‘Qt Creator’ directly from http://www.qt.io This package is overkill, but it does an excellent job of installing everything you need (and more) relating to Qt.

After that you should have Qt 5.7 or later installed, including other Qt dependencies listed in the installation guide.

You should use the Asterisk pre-stop event handler to run qloaderd (which uploads queue_log data into a MySQL database for further analysis).

Create a bash script called asterisk.stop.pre and place it in the HAAst event’s directory. In that script call qloaderd with any parameters you need.