Recently in Network Troubleshooting Category

With SaaS (software as a service) adoption continuing to increase, some have postulated that appliance-based network analysis and troubleshooting is in jeopardy. In a SaaS model, an application is hosted by a provider and then offered as a service to enterprises across the Internet. In other words, no on-premise hardware is required. This type of software delivery is attractive because it removes the requirements to download, install, and maintain software on-premise. Not having to buy the IT infrastructure is also on obvious financial benefit.

SaaS is taking off. In fact, Gartner recently forecasted that SaaS' worldwide revenue would surpass $8.5 billion in 2010. That doesn't mean it's the right choice for everything, namely network analysis and troubleshooting. We believe network analysis is difficult and complex and the process cannot be effectively transitioned to a service via the cloud. We believe that network analysis and troubleshooting is still best handled with on-premise software and hardware appliances. Here's why:

1.  You still own the network

Just because you've outsourced the application, doesn't mean you've outsourced the network. And odds are your internal network has grown to be quite complicated, and your users are relying on that network for access to your SaaS provider. Ongoing monitoring, with a capability to drill down into suspicious issues and quickly achieve root-cause analysis, remains as important as ever. Perhaps even more so, if the issue is with the SaaS provider and they do not own up to it.

2. Independent (and in-house) verification is key

You have the vested interest in the performance of your overall network design, whether that design is totally within your control or involves third parties like SaaS providers. Relying on a third party to monitor your network is like ignoring your vehicle's check engine light. You know the light is on; you have no idea what's wrong and no ability to dig in and figure out the problem.

3. Multiple SaaS vendors, then what?

Once you start down the SaaS track, you'll likely work with multiple application providers for best-of-breed applications. It's imperative that network monitoring, analysis, and troubleshooting remain within your control. What if applications interact? Who will notice, or care, besides you? And what about network latency, between you and your SaaS providers or perhaps between the SaaS providers themselves? The best location to measure latency is within your own network, right where your users are. This will give the most accurate latency measurements and constant monitoring will ensure acceptable end-user performance.

Network speeds are constantly evolving and the arrival of 10G brings a whole new set of challenges for network managers. Proper network analysis and troubleshooting is key in this high-speed world as our networks become even more complex. While SaaS may be convenient and inexpensive, appliance-based network analysis and troubleshooting is the best way to ensure organizations continue to meet critical operational practice needs. We do believe, however, that the future of enterprise computing is not going to just be on-premise or SaaS. Instead, they will likely exist in symbiotic harmony.

Network troubleshooting using deep packet inspection is a tried and true approach for network engineers - no one would ever doubt this approach when a difficult problem is plaguing the network. But suggest the use of deep packet inspection for troubleshooting slow applications and you'll likely get some blank stares. Deep packet inspection is the domain of network engineers, not application engineers, right?

Not necessarily. Analyzing decoded packet headers is definitely more suited to a network engineer, but what about the payload data from all the packets? Most network engineers find little value in the payloads, often times they filter them out to conserve analysis resources. But payloads can be of high interest to application engineers investigating application problems, including slow response time.

Consider the example of a help desk application used by a large online retailer. Support professionals, who access the help desk application for each customer call they receive, begin to experience slow response times from the application and of course the initial report is that the network is the problem. A network engineer begins his investigation and after a short time, calls in the application designer stating the problem is the application, not the network. "How do you know that," asks the application engineer? "Simple", says the network engineer. "Your application is generating the following messages. Your server command was deadlocked with another process and has been chosen as a deadlock victim. Re-run your command. And The rollback transaction request has no corresponding BEGIN TRANSACTION". Flabbergasted, the application engineer exclaims, "Where did you get that information?" "From the packet payloads involved in the slow response time transactions on the network, using deep packet inspection network analysis troubleshooting. You should try it sometime."

Deep packet inspection can provide greater value than simple network troubleshooting. Application engineers can certainly benefit, both in troubleshooting application problems and analyzing the overall behavior of an application before trouble is reported. The following four steps should be done to quickly isolate and analyze specific application data:

1.  Capture data at the appropriate location
For application analysis and tuning, it's best to locate a distributed network analysis software probe or appliance in the data center in line with the application and database servers. This will capture the data for all application users, whether local or remote.

2. Save packet data to disk
By saving packets to disk and employing post-capture analysis, you can take your time in doing your analysis, without worrying about missing key data.

3. Filter stored packet data for the application of interest
Some solutions make this easier than others, but this step is crucial in creating a data set that is manageable and applicable to the application you wish to analyze. This can often be done by filtering out all traffic except traffic that has the source or destination IP of your application server or servers. If it's a troubleshooting exercise and the problem is isolated to a particular client, you can even further refine the filter to just the IP to IP conversation between the client and the server.

4. Use built-in analysis features to look for common faults

Before diving in and looking for complex, one-off faults, use the built-in fault detection and analysis capabilities of your analysis software (again, these features can vary wildly between competing solutions) to look for common issues, like one-way traffic, database client errors, slow server response time, failed login, resource errors, etc.

Overall, even though you may get some awkward stares for suggesting deep packet inspection  to troubleshoot slow applications, in the end it's worth it because, you'll not only keep the network healthy but employees happy and productive.

It's called World Cup fever for a reason. With millions of people watching and reading related news online, it undoubtedly increased Internet traffic. In fact, people apparently care more about soccer than they do the presidential election. The very first day of the tournament, traffic exceeded the previous record of 8.5 million views per minute (vpm), which occurred when Barack Obama was elected. According to measurements by Akamai, traffic for news sites on June 11 started to climb steadily at 6 am ET and peaked six hours later, reaching nearly 12.1 vpm.  And the fact that June 11 was a workday didn't stop people from watching. People turned to their office computers to follow the action. Being able to identify high talkers and effectively manage traffic is a must for organizations that want to stay productive and successful.

Below are some considerations for enterprises in regards to their networks during times of high traffic: 

Understand how your network performs normally

The only way a company can improve network performance is to know where they stand in terms of network demands. Then they can measure success against those baselines. By looking at Internet connections, WAN links, WLAN environments and the data center, enterprises can get a sense of how their network normally acts. A network analyzer can also help organize this information into a report that can be used to not only solve issues that currently exist, but also to allow organizations to go back in time to validate performance and bandwidth utilization as necessary.

Prune and clean WLAN traffic

Remove unnecessary traffic. Devices like printers, support stacks and protocols that aren't in use in the environment can be eliminated. Sometimes, protocols that help manage the network, like routing protocols and SNMP can be found, hogging valuable bandwidth without any purpose.

Know your options

Consider a Web surfing policy. The 2010 FIFA World Cup lasts from June 11 through July 11, 2010 and will likely suck up a lot of bandwidth. Having a set policy in place will help keep traffic down and is an option to be explored. However, getting employees to abide by regulations it's often more of challenge than it's worth.

The fact is, it is important to see new trends approaching and make changes to the network to account for an enterprise's behavior. Popular events can erode profitability and corporate security, as employees not only watch but also participate in social media discussions and gamble online. Review network analysis measures and policies now, so when 2014 rolls around, networks stay healthy and humming.

Roaming occurs when a handset moves out of the range of one access point into the range of another. It gives users the mobility to move around within a local coverage area and still be connected to the network. However, roaming is one of the primary reasons why users experience problems on wireless networks. Excessive roaming times lead to poor quality for voice and video over wireless and can lead to dropped calls or data connections.

Roaming usually involves a channel change, but that depends on the type of technology deployed. If it's a multi-channel architecture, which is most likely the case, a channel change is required. When roaming occurs, the client needs to be re-authenticated and re-associated with the new access point, which takes longer than 150 milliseconds in most instances, especially when advanced features like WPA2 and WMM are in use. Most organization's wireless networks are outfitted with multiple access points (APs) and users can experience poor signal strength and performance despite proper coverage in the area if the client is connected to the "wrong" AP. Even in the most modern, centrally managed systems, the wireless client is the one who decides when to switch from one AP to another. This decision is typically determined based on the current signal strength and is executed by the underlying software controlling the wireless client radio (the "supplicant"). This software is different from manufacturer to manufacturer and from device to device, so the way the decision to roam is made varies widely. In most cases, the wireless client will wait too long and as a result the available signal strength lowers, before the client switches to an AP with greater signal strength.

New and improved standards are available that specify the conditions for "fast roaming," enabling transitions that take as little as 5 - 10 milliseconds. These specifications include:

• 802.11i - with opportunistic key caching so there is no re-authentication step
• 802.11r - fast BSS transition, which optimizes the hand-off as clients move from one access point to another
• 802.11k - radio resource management of WLANs allows re- authentication to be maintained between multiple APs and has predictive capabilities

These new standards (802.11i isn't new, but it's still part of an improving situation for roaming) allow APs greater control in determining when roaming should occur and the APs are more in tune with the current performance of, and demands on, the wireless network. However, this situation is even better when the overall wireless network is under the control of a centralized manager. The issue is that adoption of 11k and 11r has been very slow, especially in wireless clients, and until adoption increases significantly users will continue to suffer slow AP transitions when roaming, leading to poor voice and video over IP performance.

In the meantime, the best approach is to carefully monitor and analyze the roaming activity on your network. Obtaining a complete and accurate view requires real-time aggregation of data from multiple channels and APs, with integrated analysis that leads to detailed reporting - who is roaming, how long each event is taking and what does the average look like for each AP. The end result is simple, yet the process is complex, demonstrating why proper network analysis tools are key to staying productive. 

Online mobile VoIP (or VoFi) is coming. In-Stat anticipates 171.3 million users by 2013, with annual revenues projected at $10.8 billion ("Mobile VoIP - Transforming the Future of Wireless Voice; In-Stat In-Depth Analysis," Frank Dickson, Sept. 2009). Previously on our blog we've talked about why VoFi and why now, specifically the benefits of VoFi. Now we'll focus on VoFi monitoring, analysis, and troubleshooting.

Before you panic, take a deep breath. Analyzing VoFi traffic is basically the same as analyzing VoIP traffic. Remember though that wireless exacerbates factors such as jitter, latency, and packet loss that affect VoIP. Watch Using VoIP Metrics to Identify Network Problems for the specifics.

Begin at the Beginning: Your End User's Call

When problems arise with VoIP or VoFi applications, you start in the same place. Your first step - before you begin to worry about statistics or packets - is to take the time to listen to representative calls. You want to hear what your end users are experiencing. Your ear will reveal telltale signs of latency, jitter, and packet loss. Be sure your VoIP analysis application supports playback of call audio, specifically the playback of individual RTP streams as well as the playback of the complete call. Without the audio, you can spend hours tracking down problems that aren't due to either the application or the network - for example, clicking due to a damaged handset.

Take Your Network's Pulse

Once you have listened to the call, you'll want to take a look at what's going on in your network.

Figure 1: Overview of Network Health

Immediately you see what you heard - the call quality was poor. The Mean Opinion Score graph gives an average over all calls occurring on your network. In this example there's just one call, so you see the average for the duration of that call.

Dig Deeper

With Expert Events you're able to verify what your ear told you.

Figure 2: Event Summary

With this call, you can see that there are a lot of physical errors: late packet arrival, retries, out of sequence packets, packet loss, excessive jitter, and more. With the cause identified, you can quickly begin to fix the problem. Looking at the call in its entirety, you'll notice the call is closed, it had a successful ending - meaning the call wasn't truncated - what CODEC was used, how long it was, and what the Mean Opinion Score was.

Figure 3: Call Statistics

In this example, the mean opinion score of 2.5 lets you know that the quality of the call was pretty poor. In the media view, you can drill down into each segment leg to determine why the quality was poor.

Figure 4: Call Details - R Factor, Mean Opinion Score, Packet Loss Percentage, One Way Delay, Etc.

Understand the Differences between Wired VoIP and VoFi Calls

The next two figures show both a Wired VoIP call and a VoFi call packet-by-packet. (For an in-depth discussion of these calls, watch Anatomy of a VoFi Call: Packet-by-Packet.) You'll notice that they're pretty similar. The protocols used are different and with VoFi there's the additional step of authentication.

Figure 5: The Anatomy of a Wired VoIP Call

The differences involve: wireless segments instead of wired segments; signal interference; and wireless roaming.

Figure 6: The Anatomy of a Mobile VoIP (VoFi) Call

Learn More

Last week in Toronto, Joe Habib, Director of Global Services, presented "QoS of IP Telephony: Slaying the Three-Headed Beast of Jitter, Latency, and Packet Loss" at IT360. His presentation (PDF) is now available online. If you're interested in ensuring QoS for your current (or future) VoFi deployment, you should definitely check it out.

In the presentation, you will learn:

• What six factors contribute to poor voice quality
•  How to establish metrics for evaluating VoIP call quality
• How to balance high-speed, bursty data requirements with requirements of high quality voice calls
• How to capture data for VoFi Analysis and use VoIP metrics to identify developing problems
• How to analyze a VoFi call packet-by-packet and verify voice quality with call playback

While the cost for 10GbE is coming down and adoption is rapidly rising, there remain challenges in analyzing 10GbE traffic, most notably because the industry has yet to achieve real-time analysis at 10GbE line rates. However, 10GbE analysis is available and does not have to be limiting in terms of results. Below are six questions that will help determine if your organization is fully optimized for analyzing 10GbE traffic.

1. Are you being specific enough?

It's important to know exactly what you want to capture and what information is going to be most beneficial for your analysis. Your requirements will likely vary between each network segment and odds are you are going to have to capture data at several locations. An excellent way to analyze 10GbE traffic, especially when utilization is high, is to use post-capture analysis and only save the data to a disk in real-time. Trying to capture and analyze simultaneously, in real-time, on highly utilized network segments puts much more strain on the system than if you just save data to a disk for post-capture analysis.

2. Do you REALLY know your network?

Knowing how you expect the network to be performing is all the more critical when trying to analyze highly utilized 10GbE segments. If you're already embroiled in a complex network analysis firefight it's too late to realize that your ability to assess "normal" conditions on the network may be lacking. To get a sense of "normal" conditions before trouble arises, you should perform and archive baseline measurements across specific network traffic like HTTP and key business applications over typical cycles - like an hour, a day, and a week, for the network as a whole. Other metrics to consider include understanding packets size distribution as well as protocol and node usage over time, uncovering cycles in these metrics, which provide a "fingerprint" of your utilization. That way you will always have a clear view of the network for comparison when trouble arises. Only after convincing yourself that the basic data is in place and being collected and analyzed should you embark on detailed analysis and drill-down of packet-level data.

3.  Are you sticking to the essentials?

The temptation is to try to capture and analyze everything, especially when the source of the problem is not immediately known. But quite often certain conditions can be immediately ruled out, and using these clues to limit the collection and analysis to only what is necessary dramatically improves network analysis performance. You always have the option to customize analysis by turning off modules that are not important to the current exercise. Modules such as wireless network performance can be turned off, especially in 10GbE analysis, because odds are they are not relevant to the problem being investigated. The key is to customize your usage and take advantage of it.

4. Do you know your limits?

Even after analysis has been streamlined to only essential areas of the network, data capture for network analysis on 10GbE networks generates a great deal of data quickly, and managing the data becomes a significant challenge. Regardless of the system used, the data is typically stored for subsequent retrieval and post-capture analysis. The two most common formats are standard packet files and databases. In either case, two metrics to manage closely are file size and frequency of disk writes. Though intuition may lead you to think that the larger the file size the better, this is often not the case as very large files require very large memory footprints to open. If the files are too large they will be unworkable on the computer being used for analysis. Smaller files, however, typically lead to more frequent disk writes, and this can rob the system of precious resources for performing the actual packet capture. Optimum performance is achieved with a balance of these two demands, and this is different depending on the hardware resources available. One rule of thumb to keep in mind is that if files are being created every 30 seconds or less, it's going to increase strain on achieving the maximum packet capture rate significantly. Starting with reasonable sized buffers and files makes all the difference. We recommend that you start with 256MB buffer for packet capture and 128MB for files to be created. After a few captures you'll quickly determine if either of these parameters can be better optimized for your system. Also, try to use the lowest number of simultaneous captures as possible. In several systems, you're allowed to create as many captures as you want, but you need to remember that for each capture you open more memory is reserved for buffering and less is available for data processing.

5. Are you filtering and slicing?

Filtering is a way of limiting the overall number of packets captured and stored based upon user-specified criteria. Slicing captures and stores all of the packets, but it truncates the packets after a certain length, typically allowing you to store the header information but slice off the payloads. In both cases the same result is achieved, the overall amount of data to store is significantly reduced, freeing up more processing power for capture and analysis and more disk space for storing the data that's truly important to the current analysis task.

6. Are you being reasonable?

Most network analysis systems allow multiple users to connect to the hardware performing critical network data capture and analysis tasks. Put a limit on users. Nominate an owner for each system that will monitor filters and captures. Make sure it's understood who has the authority to go kill a capture. Too many users with too many options is a recipe for disaster. You can always scale with additional systems if needed.

Recently, we posted "Three tips for determining whether latency is caused by the network or application," which investigated the potential causes of latency along with suggestions for effective latency monitoring. It turns out that a lot of people were interested in the topic, as evidenced by the volumes of detailed comments generated on LinkedIn Group forums. So we've decided to offer additional insight on latency issues and explain how Latency Monitor, a free add-on application for our OmniPeek protocol analyzer, can help organizations fight latency issues and ultimately, save money.

If latency issues cause system failures, it can cost a lot of money to fix. It needs to be monitored 24/7 in order to successfully run business critical applications. However, not all latency issues are created equal and monitoring applications need to be intelligent and flexible to accommodate specific situations. 

Latency Monitor provides an intelligent and flexible way to determine and react to latency. It is designed to produce meaningful notifications across groups of servers as well as specific applications. This type of high value notification system can be used to build procedures, both manual and automated as well as react and fix problems in advance. By fixing problems before they occur, companies save money.

By monitoring the level of latency, Latency Monitor makes sure it does not exceed an acceptable threshold. If levels become unmanageable, a notification is sent. Appropriate actions can be taken either automatically or manually to lower the latency. By lowering the latency before end users are affected, they can continue working, the IT department remains undisturbed and the company ultimately saves money.

There is no doubt that latency can be expensive and a serious threat to productivity if not monitored closely. By providing intelligent notifications, the Latency Monitor for OmniPeek helps organizations save money by identifying and solving latency problems quickly.

Unless you've been on another planet, you've most likely heard of cloud computing - a computing approach in which the delivery, operation and management of applications, typically those used by corporations, is outsourced to a third party capable of highly scalable operations. In fact, not only has everyone heard of cloud computing, they're adopting it. Pew Research Center reports that 69 percent of U.S. Internet users are now utilizing some form of Internet-based computing.

This sudden surge to the cloud is leading to the perception that monitoring the network within enterprises is less pertinent in maintaining a functional system since applications and problems are being outsourced. Will cloud computing lead to the demise of network analysis? 

We think not. In fact, we believe that network analysis is very important, if not more critical, in cloud computing. Moving to the cloud will surely alter current network analysis processes and priorities, but a solid network is still needed to handle communication between users and applications. The only difference is that the focus of network management shifts from managing infrastructure to managing service availability and performance.

Cloud computing can be thought of as merely a shift of your application servers from your facility to a different location. Common issues within the network remain. Potential bottlenecks, bandwidth hogs and unauthorized protocol usage still adversely affect application traffic. In order to combat these issues, more diligence must be applied in monitoring application performance. It is essential to establish clear, long-term baselines before transitioning the application to the cloud. Additionally, verifying the performance of transactions that cross multiple applications is key in a new cloud environment. 

As with everything related to the network, security is always an area of concern and transitioning to the cloud is no exception. Concerns with security start the instant the cloud vendor is selected. The relationship with a third party to store and process highly sensitive data and applications needs to be closely monitored. The cloud vendor's security policies and procedures should be understood and the appropriate test processes should be implemented within your own organization to ensure these policies are not violated.

By utilizing cloud computing, a company is simply moving the location of applications and shouldn't interpret it as an excuse to forget about the network. The most important matter for companies considering the cloud is to be certain they have their network analysis processes firmly in place before the transition. This will ensure that they are not relying solely on the cloud vendor for advice. Transitioning to the cloud for any business is evolutionary and will be slow and methodical, but sufficient network analysis will further support the company's success in implementing this new technology.  

For a more in-depth discussion of how cloud computing impacts network analysis watch our free 1 hour OnDemand Webcast.

The OmniPeek Plug-ins are flying off the shelves.  And they should, since they are free to customers with maintenance.   There are over 40+ plug-ins available on MyPeek.   Some are more popular than others.   To help you decide which ones to try first, we have taken a look at the stats, and have come up with the top 10 most downloaded plug-ins.

Compass Forensics Dashboard with enhanced Milliseconds view