WildPackets Network Analysis and Monitoring Blog

Whenever a new application is rolled out on the network, IT must be able to predict with confidence how it will perform for the business. Moreover, other mission-critical applications must not be negatively impacted. Solutions that provide network performance analysis capabilities should be able to provide immediate feedback on how new applications behave on the network and can predict the impact to existing, business-critical applications.

Here are three warning signs that a new application on the network is not performing well on the network and / or is negatively affecting other business-critical applications.

1) Performance in production doesn't match pre-deployment testing/analysis
Pre-deployment testing can take many forms, from a relatively simple set of tests followed by predictions of the network impact, to complete test networks that simulate the entire usage pattern of the application. While the latter approach can be complicated and time consuming, the former technique is simple to perform with a few basic tools. By installing the application and capturing some representation transactions with packet-based network analysis software, one can perform "what if" analysis that allows one to "scale" these representative transactions to the expected number of users of the application once in production. The resulting analysis provides a clear assessment as to the readiness of the network for the new application. Additionally, it will give the data needed to compare application performance once the application is in production. Any discrepancies are a red flag and should be addressed.

2) Network baselines change dramatically after deployment
Without network baseline data organizations are operating blind, whether it pertains to the specific issue of deploying a new application, or just general day-to-day network performance management. Baseline data should include long-term (weekly cycles work well) sets of network utilization data from key network interfaces; node, protocol and application utilization over time; and an assessment of the typical "noise floor" of the network - typical packet loss numbers, number of retransmissions, latency measurements for key applications, etc. After deploying a new application (and as part of the pre-deployment testing) a new set of baseline data should be generated and compared against data from before the rollout. Unexpected differences should be addressed.

3) A change in network "health"
Network health ties into the aforementioned "network noise floor", but changes in network health can usually be seen much more quickly than comparing long-term baseline data. Enterprise-quality network analysis software will constantly monitor the vital signs of the network by performing background expert analysis, and it will send alerts when vital signs begin to drift. For example, if an organization deploys a new application and its network analysis software begins sending alerts that there are too many TCP retransmissions, this is a strong indication that this new application has tipped a balance. (even without even knowing the network's current threshold for TCP retransmissions).
 

No matter what solution an organization selects for analyzing how a new application behaves on a network, it must meet the following criteria ...

1)       Provides detailed information through all seven layers of the OSI model

2)       Enables collection of key network data over meaningful time cycles (like a week) to develop high quality network baseline data

3)       Quickly identifies if the network is being stressed and where

4)       Offers "What If" capabilities for analyzing the impact of new developments and implementations

The holy grail of effective network troubleshooting is the ability to pinpoint issues quickly so that they can be fixed. Any approaches to better optimize this particular network analytics process mean more uptime and healthy networks over the long run.

Here's a suggestion - instead of loading all packets, shave off time by using utilization statistics about network traffic to provide clues that answer questions like "What happened?" "When?" "Who did it?" Only then determine what slice of time you want to perform deeper network analysis on.   

To this end, WildPackets is releasing Compass, a freely available interactive forensics dashboard for the OmniPeek Network Analyzer. Compass' dashboard graph (see screenshot) allows users to select specific time periods for analysis, add and remove nodes and protocols to the same graph, and compare and correlate these for different periods of time, over long periods of time.  

In some cases, seeing the utilization in the Compass graph for the nodes and/or protocols in question may solve the problem. Otherwise, once a slice of time has been selected, the packets for just that slice of time can be loaded into OmniPeek by hitting the "Load Packets" button.  If that slice wasn't the problem, just go back to the graph, slide the time window, and load a different slice of packets.

Networks typically run various applications, from single-tier, locally-hosted applications like e-mail, to multi-tier web-based applications, or even time-sensitive, multi-hop applications like VoIP. While application traffic typically resides within the data center, SaaS and cloud computing are rapidly growing trends that are driving application traffic outside of the traditional enterprise network, making network latency even more of an issue. Pinpointing and correcting slowdowns is therefore a necessity, and can be a real challenge. So what is responsible for the latency creating poor application performance- the network or the application itself?

First, we must distinguish between the two basic types of latency - network and application latency. Network latency is the amount of time it takes for an application to make a request and the server to respond with an acknowledgement (a packet message used in the transmission control protocol to acknowledge receipt of a packet). Application latency is the amount of time needed for the application to process the request and send a response containing real data.

Most network-monitoring products provide some sort of latency-monitoring features, typically either one or the other, not both. Here are three tips to determine whether latency is be caused by the network or the applications (or both) in your environment.

1)       Clearly determine network versus application latency

Every application issue is blamed on the network until proven otherwise - "guilty until proven innocent." Clearly measuring network latency vs. application latency is the proof the network engineer needs for acquittal. Packet-level monitoring is ideal for accumulating evidence. By visually inspecting a packet-level conversation between a client and a poor performing application, one can see whether the network (or a network device) is the source of the delay, or if the application is the bottleneck. This is done by comparing the responsiveness of the TCP ACK to a client request versus the application response, which includes actionable payload data. Quite often the network acknowledges the client request quickly (within milliseconds) while the application may take tenths of seconds or even seconds to respond with payload data. When you see this, you know it's the application causing the problem.

2)       Periodically test and monitor key applications and network connections

Periodic, active monitoring can also provide insight into network performance on key interfaces, and can alert you when conditions begin to degrade. While this technique only addresses network latency (not application latency), it can still provide important data when determining whether the issue is network or application related. For example, let's say the organization's CRM application is via a web-hosted service. Running periodic traffic in the background (even just pings) to the CRM application host can provide an ongoing baseline of the performance of the network between users and the host. If the baseline increases, alarms are used to notify that the network latency is becoming an issue. User complaints about CRM performance without a marked change in the network latency baseline almost always indicate the application host or the application itself is at fault.

3) Graphing latency over time

Graphing latency over time helps to identify patterns and anomalies that deserve closer attention. Latency monitoring can help correlate areas of latency with other relevant statistics, as well as the actual network traffic occurring at that time. This type of high-resolution forensic analysis can help to detect latency problems at the highest level and drill down quickly for closer inspection.

Ideally, network latency and application latency measurements can be graphed together over time, making clear whether the problem lies with the network or the application. Comparing the measurements of the two types of latency over time and seeing the differences can provide information that might have otherwise been overlooked.

Latency monitors can include a feature that sets thresholds on latency, so alarms will go off when normal conditions are exceeded. The administrator can be made aware of excessive latency before application performance becomes a widespread issue, allowing him or her to make necessary adjustments to the network proactively. This type of proactive latency monitoring allows the administrator to detect and correct problems in the network and applications before users even notice a slowdown.

Consideration #1: Can you record your network traffic and search though the data at the time the issue occurs?

This is also known as network forensics. Network forensics refers to the capture, storage and analysis of digital evidence that flows through your enterprise network. The most complete solutions record every single packet that is transmitted over your corporate networks. So, any emails, instant messages, FTP traffic or any other form of communication that takes place on the network can be reconstructed from the original transmissions. Forensics essentially allows you to reconstruct the history of your entire network.

With more businesses relying on the cloud for their IT infrastructure or to deliver their service/products to customers, it's crucial to be monitoring both operations and the infrastructure. While the network has become more reliable, reliance on web-based and cloud-served applications or storage has lead to more frequent outages of that infrastructure. By collecting digital evidence via a network recorder, the once laborious, time-consuming searches (including top talkers, most delays, application type, etc.) involving multiple tools and large transfers of data can be reduced to a quick, convenient search.

Consideration #2: Is the problem stemming from one user or many users on the same switch or segment?

Determining the scope the problem can point the administrator in the right direction of where to start the network analysis providing and what information is most useful determining and correcting the issue. There are several ways of determining whether or not a problem is stemming from one user or many users on the same switch. One of the more common, but least desirable ways is by monitoring the number of trouble tickets. Calls spike - most users are on the same subnet - this is a telltale sign of a possible hardware problem. A far more proactive approach is to use background analysis and monitor for conditions like non-responsive client or server, or low client-server or server to client throughput. You will quickly see if these issues are being reported for a single client, or across many clients. If for a single client, isolate this client for analysis. Determine what other network activities this client is engaged in, and examine these network flows. This will quickly shed light on the issue. If the problem is stemming from many users, is the problem isolated to a single application, or is the issue broadly affecting overall connectivity? If confined to a single application, that's the place to dig. If the issue is overall connectivity for many users, find the connectivity point common to these users and see check for hardware issues.

Consideration #3: Is the problem connectivity or utilization related?

Is the network traffic getting to the specified destination? Is a specific machine over-consuming its allocation of bandwidth and crippling other users connectivity while doing some action? On the utilization front, non-work related, "bandwidth sucking" download activities (music, videos, games, etc) are a common culprit. Utilization-related issues are typically intermittent in nature. One, or perhaps several, clients are over-utilizing a network segment, but that comes and goes. Even if the oversubscribing event is long in nature (like streaming video) the remaining utilization still goes up and down with normal network usage, creating intermittent periods of over-utilization. This can easily be monitored by graphing the network utilization in real-time. Connectivity-related issues are typically more binary - users either can or cannot connect to a particular network segment or a particular application. If the issue is utilization related, the next step is to determine if it is client or application driven. This is fairly easy to determine by looking at the top talkers on the network. If the top talkers are clients, drill down and see what protocols the client is using. This typically reveals the source of the problem quite readily. If the issue is connectivity related, the next step is to determine if connectivity is being affected by network congestion, or hardware problems. Network congestion is again easily seen by monitoring network utilization is real time. If not congestion, then the issue is likely to be with hardware within the user(s) connectivity path.

While 2009 ended with cyber security dominating headlines with the Wall Street Journal reporting hackers had stolen tens of millions from Citigroup, TechCruch reporting about Twitter getting hacked, and the New York Times reporting President Obama naming Howard A. Schmidt as the U.S.'s Chief of Cybersecurity, 2010 picked up right where 2009 left off. Google has been hit, likely via an inside job at their office in China, by a cyber-attack on its network that resulted in theft of its intellectual property.

There's a lot more malware-related issues brewing under the surface, as Nemertes senior VP and Network World columnist Andreas M. Antonopoulos points out, "While no new major malware outbreaks made huge headlines, the silent spread of stealthy keyloggers, trojans and botnets continued. As predicted, more computers fell prey to these silent threats while the lack of headlines is broadly and incorrectly seen as 'success' against malware."

It's not enough to know you were the victim of a cyber-attack. With today's network forensic technologies, organizations should be able to answer the following questions:

1. Who was the intruder?

2. How did the intruder penetrate security?

3. What damage has been done?

4. Did the intruder leave anything behind?

5. Did the organization capture sufficient information to effectively analyze and reproduce the attack?

In the past, classic forensic technologies typically provided an incomplete diagnosis because of incomplete reconstruction. In other words, when an attack bypassed a firewall, only partial attack data was processed using the IDS / IPS system, yielding incomplete data and leaving many of the key questions unanswered. Methods are changing. Today, when an attack bypasses the firewall, a network recorder records and aggregates data throughout the attack, supplementing the partial attack data processing of the IDS. With this approach, post-event analysis reveals answers to the aforementioned questions and exposes attacker, method, and damage; with the entire attack recorded the fingerprint is captured and it never needs to happen again.

While data recorders will not prevent a zero-day cyber-attack, the information they provide can lead to an informed and efficient security posture within the organization, allowing accurate attack fingerprinting and rapid retooling of security technology and processes to deter similar attacks.

It is also important to create a capture buffer large enough to hold the generated packets.   Typically, there will be many more packets generated by the NetFlow Analyzer adapter than there are in the NetFlow trace file.  The alternative is to enable capture to disk, so that the generated packets are saved to a file.