New Year’s Network Issue

We had an interesting support issue within our own network on New Year’s Eve. The developer in charge tells his story below:

Right after 1:00am CET, I started to receive notifications that the Resource Limit matched the network for an average load >10. First I thought it was a scheduled task, but quickly realized that nothing that “heavy” would be scheduled at midnight (GMT). So I decided to stop watching my boring movie and look into what was going on.

I found a lot of failed calls with the same reason – that CAC was exceeded. I also realized that there were a lot of calls, even though it was New Year’s Eve and most of our customers businesses were closed for the holidays.

The top command showed all FreeSWITCHES processes and mysqld had >100% CPU usage. It was a mystery because there were no hard SQL queries and even FreeSWITCH had zero calls and the CPU was >100%.

First, I restarted the FreeSWITCHES one-by-one and tried different switches for the RT clock and clock sync. Nothing helped. FreeSWITCH compiled 2 years ago and nothing changed. How had it passed last New Year’s Eve?

1

The reason that FreeSWITCH was rejecting calls was a constraint that the minimum idle CPU must be 5% to ensure transcoding quality. I decided to disable this and left 3 FreeSWITCHES working to keep the load less and allow calls to pass. FreeSWITCH would occupy 3 cores and have room for calls.  I tried several calls and had no problem with quality. It seemed that would be enough while I tried to find the problem and solve it.

This is how it looked in the morning when I finally had to get sleep, unable to solve the problem.

2

Later that day I decided to try a new FreeSWITCH 1.6 and 1.8. I compiled, prepared, and tried, but it did not help. The CPU usage was >100% again. Then I compiled a debug version and added it as the fourth FreeSWITCH in production to get the core dump and figure out why the idle FreeSWITCH was eating the CPU. I realized that FreeSWITCH had a problem with timers. I tried a different one, but it did not work. There were problems with the nanosleep function in FreeSWITCH, but it was fixed already. Somehow FreeSWITCH squeezed time and jumped into a loop.

Then I checked the time on the host, it was correct. Even /proc/interrupts looked correct.

My last option was to try adjusting the time on the host and sync with NTP. Restarting the ntp-client failed. Fortunately it was a problem with the DNS. Actually, the host was unable to access our local DNS resolver. I changed it temporarily to an external nameserver to resolve the NTP server and correct the time.

At the same moment when the NTP finished syncing, all processes went back to <5% CPU usage.

I realized that it was a problem with the leap second bug on an old kernel and it affected FreeSWITCH and MySQL, but it was corrected with NTP when NTP synced the time.

What is the Difference Between a Session Border Controller and a SIP Proxy?

session border controllerWe often find that our customers have a misconception of Session Border Controllers (SBC). Customers that request a SBC often times need a SIP Proxy rather than a true Session Border Controller.

In general, people think of SBCs as a proxy that can be used for routing, security, and
other types of network administration. Customers seeking some sort of routing
administration – Least Cost Routing, for example – will request an SBC, thinking that is
the solution to their need.

However, an SBC is actually a back-to-back user agent SIP application with a wide variety of other uses such as policy-based access control, transcoding, topology concealment, call accounting, QoS, and call quality statistics.

On the other hand, a SIP Proxy is an IP PBX component used for call functions such as routing. A SIP proxy will send SIP requests to the appropriate destination and return a response. SIP Proxies may also be used for registration, authorization, security, network control, and other call functions.

Download our SIP Proxy PDF to learn more. Or For a more detailed and technical explanation, visit Likewise.am.

This post was featured here.

More posts like this one:

Bicom Support

Bicom Systems support techs are always prepared to respond to errors as quickly and effectively as possible. Recently we found reports that the Outcall2 Plugin had an issue with Outlook 32-bit on Windows 10 x64.

One of our support techs was able to quickly rebuild the plugin to support the 32-bit version and update the release. He responded to error reports with the update and instructions to reinstall the new version.

One user responded with the following:

After reinstalling, the Outlook add-in does work perfectly on my Windows 10 x64 with Outlook 2016! I’d like to know, are you with the OutCall Project or with Bicom? If Bicom’s support is this responsive and effective, I can only imagine how awesome their support is on products they actually sell!”

If you encounter any issues or bugs like the above, please contact us for a solution!

More posts like this one:

5 Considerations to Moving Your Business VoIP System

voip systemMoving a business is often a lengthy and complicated process. While VoIP technology certainly means less hardware and cables to move, it still involves some planning and forethought. So the big question is : to take the VoIP System along or start over at the new address?

Take It or Leave It : Moving Your Business VoIP System from Ziff Davis addresses that question and offers five considerations to take into account when making the decision.

1. Circumstances Around Your Move

Take the VoIP System if : The move is well organized or planned in advance. The business is being merged or relocated to somewhere without a VoIP System.

Leave the VoIP System if : The move is spur-of-the-moment or disorganized. The business is being merged somewhere with a better system.

2. Type of Environment

Take the VoIP System if : The new environment is similar or smaller than the current.

Leave the VoIP System if : The new environment is a shared space or an expanded space.

3. Your Phone System

Take the VoIP System if : You are still satisfied with your system and it is still a newer system. The system includes better applications than the competition.

Leave the VoIP System if : You are no longer satisfied with the system or have been wanting to change or expand. The system does not include the newest applications and capabilities.

4. Changing Communications Needs

Take the VoIP System if : The current system will continue to serve you for the next several years; if it is flexible and expandable.

Leave the VoIP System if : The move is about making change or growing and expanding.

5. What VoIP Vendors Can Offer

Take the VoIP System if : Your VoIP Provider offers options and deals to migrate the system to your new location.

Leave the VoIP System if : Other VoIP Providers are offering something more lucrative.

To read more, see Take It or Leave It : Moving Your Business VoIP System from Ziff Davis.