Tag Archives: rtp

Best Practices for Troubleshooting Voice Quality Issues

Last week, we focused on whether or not your network could handle more real-time data, like video and voice over IP. The blog outlined how video and voice over IP affect the network, whether or not your network can potentially handle more of this real-time protocol (RTP) traffic, and how to best prepare for this confluence of highly sensitive data.

Now that you have determined that your network can handle more VoIP, it’s time to take it a step further: What is the quality of your calls, and how do you troubleshoot voice quality issues?

When troubleshooting real-time data like VoIP, it is important to have the right analysis tools in place that can baseline your network and drill down into the most granular details of packet traffic on your network, but more importantly it’s critical to have tools that let you focus quickly on telephony calls and quality, which is where we will begin.

MOS and R-Factor – automated measurement of call quality
Mean Opinion Score (MOS) is a rating system that helps you assess the relative quality of a call. It has been used for years in telecom landlines, where real people were asked to listen to sample telephone calls, rating the call quality from 1 (very poor quality) to 5 (perfect). That data was used to create automated measurements which give you a reliable estimate of how the VoIP users in your network perceive call quality. Anything above 4 is considered to be a good quality call, with 3 being “slightly annoying,” 2 “annoying,” and 1 “very annoying.” In practice, there will be few complaints if your VoIP calls are above 3.5.

While MOS is a score historically based on human perception, modern implementations use a mathematical analysis. Besides MOS, another key mathematical factor in use today is R-Factor, which is designed to scale from 1 to 100, and encompasses both narrowband and wideband codecs (a weakness in MOS). Reasonable R-Factor scores for a call are about 75 and above. Interestingly, as VoIP codecs have gotten better, it has become possible for calls to have an R-Factor greater than 100. Modern codecs on a low-latency, low-jitter connection can carry a call with a score of up to 120.

Another factor that affects call quality is jitter. Jitter is the measurement of the variability of the time delay between each RTP packet. Jitter is important as a factor in call quality because, if uncorrected, it can result in gaps and lost data (missing audio) in the call. While most equipment has a jitter buffer to smooth out the gaps, the correction introduces latency. Too much latency will be noticeable by the callers as a time lag when talking, which itself reduces perceived call quality.

Dissecting RTP and RTCP to discover where the issues lie
When it comes to troubleshooting a network, it really comes down to understanding and being able to visualize the most granular details. In the case of VoIP, the most granular details for solving problems are held in the Real-Time Transport Protocol (RTP) and Real-Time Transport Control Protocol Packets (RTCP).

RTP provides more than just the packaging and delivery of voice data, it also contains details that help you troubleshoot issues, including:

  • A sequence number that is handy for detecting lost or out of sequence packets;
  • The payload type, which can be a well-known type or dynamically assigned by the signaling protocol;
  • A sync source identifier to uniquely identify an RTP stream; and
  • A timestamp set, used to determine the expected packet arrival rate to the listener and for calculating jitter.

The RTCP packets are useful because they report on various conditions during a call in progress, including the percentage of packets lost since the last sender report, cumulative packet loss and jitter. RTCP packets can be captured anywhere in the path between two VoIP users, and analyzed to see how many packets are being dropped according to the listener.

What should I have in my VoIP troubleshooting tool belt?
When you are planning to troubleshoot poor VoIP call quality, you need to have a tool in place that measures overall metrics like MOS and R-Factor, and analyzes network conditions like latency, jitter, and packet loss, based on an independent analysis of RTP streams. This way, if a device does not support RTCP, you can capture RTP streams on the same segment as the listener to get call quality information close to that user.

A robust VoIP analysis tool must go beyond simply reporting the jitter included in the RTCP packets. It must also be capable of examining independent sources of jitter information by calculating jitter directly from the RTP packet stream itself. It is also essential to have notifications not only when reported by an end-device via RTCP, but also in real-time via RTP stream analysis as packets are captured. Having it both ways allows the analyst to better pinpoint where in the network jitter starts to become a problem.

Analyzing VoIP is a two-way street
Don’t forget to analyze at the end-points. To reiterate, the VoIP packets captured at the core is not where the call terminates. Capturing at multiple points allows you to see exactly where the quality of the call drops.

Remember that it’s very important to analyze each side of the call independently when studying existing VoIP calls. For instance, during a call there will be an RTP stream from device A to device B and a second, completely independent RTP stream, from device B to device A. Why? You need to know in which direction the call has degraded to solve the problem.

When looking at factors for VoIP over wireless (VoFi), there’s an entirely new set of considerations, which we will dive into in more detail in a few weeks. Stay tuned!

Can Your Network Handle More Real-time Data?

Real-time data, especially video, is the wave of the future, and as this wave begins to break there are others already crashing onshore. Will you be inundated, or do you have the appropriate network infrastructure in place to stem the tide?

Global Internet video traffic surpassed global peer-to-peer (P2P) traffic in 2010, and the percentage of all forms of video is predicted to be approximately 90% of global consumer Internet traffic by 2015 (full details in this Cisco report). Coincident with this acceleration in video traffic is a general enterprise transition towards a shared ecosystem using SaaS and cloud computing. Though at first glance these trends may seem mostly unrelated, adding significant volumes of video traffic, or any real-time protocol (RTP) traffic for that matter, can have a significant, and typically adverse, effect on your application performance. Whether business critical or not, when there’s video available on the Internet it will find its way onto your corporate network so it’s important to have a full understanding of the benefits as well as the consequences of adding additional RTP (voice or video over IP) traffic to your network.

RTP traffic is highly sensitive to network latency, and the trend towards SaaS and Cloud computing typically increases overall network latency. Hence, as both these waves come ashore, it’s imperative that you have the infrastructure in place to monitor both application and RTP traffic, simultaneously, so you can keep from inundating your resources.

RTP has unique, and typically more demanding, requirements for network performance than “traditional” network data. However, you have only one network to carry all your traffic, both data and RTP, so you need to assess the effects that RTP will have on your application performance, and vice versa. Networks are often configured to give routing priority to RTP traffic, making application performance vulnerable when RTP traffic loads are high.

So, first and foremost, you need to have an accurate measurement of how much of the traffic on your network is RTP, and how much of that is mission critical.
At this point, it’s beneficial to make a distinction between VoIP and video. VoIP traffic is probably all enterprise driven, and is therefore mission critical. I don’t know of many rogue VoIP users on corporate networks.

Video is a different story. Though growing, the number of mission critical video-based applications is still very low, with web-based conferencing being by far the most prevalent. Most web-based conferencing still takes the form of relatively low bandwidth connections with an audio stream, and a real-time video stream thrown in from time to time.

And then there’s real-time video streaming. Maybe it’s YouTube, or Hulu, or something else, but in most cases it’s probably not mission critical. But it is likely to be far more bandwidth-intensive than anything else on your network, and if it’s being routed using QoS (Quality of Service) then it’s impacting your ability to deliver mission critical application data in the most efficient manner.

So how do you know where your vulnerabilities lie? Using packet-based network analysis and following these steps will help you characterize what’s going, and whether or not RTP traffic is threatening your network.

  1. Determine the mix of RTP and TCP traffic on your network.
    Any solution that can determine overall network protocols can do this, but some make it easier than others. It helps to be able to graph both data types in relation to the other, and over time, as the relationship is likely to be very dynamic. You should look for average and peak utilizations of each data type as well as times of day where the mix seems to be different. This should be done day after day for several weeks to get a solid baseline.
  2. Dissect your RTP traffic.
    Once you know your overall RTP traffic behavior, drill in and break that traffic down between VoIP and video. As we discussed, VoIP traffic is likely all mission-critical, while video traffic requires additional analysis.
  3. Dissect your video traffic.
    Be suspicious of all video traffic, especially high-bandwidth real-time video streams. Since most video traffic is web-based, one of the best ways to study video traffic is to look at the servers generating the traffic. All mission critical video activity, like web conferences, will involve well known servers from brand name service providers, like WebEx or GoToMeeting. Using filtering, eliminate data from these sources, making the remaining search easier. Then filter on other well known servers that are not likely to be mission critical, like YouTube, and you will quickly get a “picture” of unauthorized, or at least unnecessary, video traffic on your network. If the volumes are low, maybe you just let it ride, but if the volumes are high, or if application performance is degrading while these real-time video sessions are active, action is required.

Network utilization is very much like the ocean. Utilization ebbs and flows over longer cycles, like the tides, with occasional spikes in traffic similar to waves. And then there’s the occasional rogue wave that takes everyone by surprise. But if you’ve accurately assessed your overall usage as outlined above even the rogue wave will be manageable. Then it’s just the tsunami that you need to worry about…

3 Tips to Ensure Video Quality on Your Network

As we discussed in last week’s blog, video is slowly encroaching upon both home and enterprise networks. In a recent Cisco report, all forms of video will be approximately 90% of the global consumer Internet traffic in 2015. We predict that more than 50% of enterprise network traffic will be video by 2015.

Video data types are unpredictable; require a lot of bandwidth; are sensitive to latency, jitter, and packet loss; and demand the highest QoS delivery. As video becomes more pervasive on your enterprise network you’ll need the right tools and approach to manage this demanding data type. Here are our top tips for preparing for video and monitoring video usage and quality.

Determine Overall Video Usage
One of the first things you need to account for is how much video is already being used on your network. The best way to determine this is by using packet-based network analysis systems capable of analyzing networks for all types of traffic simultaneously. With such a system, you will easily be able to see the throughput associated with data versus that associated with video (and audio for that matter) and to determine if the ratio is what you were expecting.

Armed with that data, you want to go one level deeper and review the packet loss, media quality, and number of video sessions/VoIP calls to determine 1) if video may be underperforming on your network or 2) how much video is affecting the performance of other mission critical applications.

Identify Unauthorized Video Traffic
Whether someone outside your company is pilfering your Wi-Fi to access YouTube or someone inside your company is spending too much time watching the World Cup, these three approaches can help you determine who is inappropriately using video and bogging down your network.

First approach: Look at your top nodes and protocols and see what they’re doing. If you have nodes that are exceeding your typical baselines check these first by simply expanding the node to see which protocols are in use. (We’ve covered baselining before! For a refresher, check out Tim McCreery’s Getting Network Baseline Right article (PDF) or Jim Thor’s Baseline Product Tips and Tricks.) RTP? You have a possible culprit. HTTP? Don’t stop there. You have all the packets so dig in bit deeper to see where the user is going. YouTube? It’s probably not work related.

Second approach: Check your overall network utilization and zoom in on spikes in traffic, which are often indications of video downloads. Zooming in on a spike will identify not only the user, but also from the protocols in use and the servers they are communicating with. You might find that someone is simply using the telepresence program you’ve installed.

Third Approach: Create filters and alarms. If you build custom filters for RTP (Real Time Protocol) and Dynamic RTP you can easily see the activity happening on your network that relates strictly to video and voice. You can also create address filters, like for YouTube, to determine if users are abusing certain sites and if this having a negative effect on your network.

Monitor High-Level Video Delivery
It may be that it’s quality, and not abuse, that’s of importance to you, especially when telepresence is being used. The best way to analyze for success is to look into each individual media stream, breaking it down into its primary audio and video components, and glance at your metrics. This approach allows you to determine the quality of service on each segment of video from picture to sound quality. You can continue to dive deeper into the packet by packet IP conversation, identifying exactly where problems such as quality of service is not being applied to certain packets or jitter exceeds typical guidelines for video. With this information, you have everything you need to find and fix any problems.

Designing your network to meet the influx of video on your network, as well as instilling a proper monitoring system for this sensitive data, will ensure that your network continues to stay stable and colleagues continue to stay happy while watching their favorite YouTube video or using Skype for conference calls.