WildPackets Network Analysis and Monitoring Blog

 

Obviously, there are a number of considerations and best practices for troubleshooting a flakey network connection. That being said, here are three considerations that, in most cases, will expedite the process of identifying and pinpointing the problem and shorten the time to getting the network humming once again. 

Consideration #1: Can you record your network traffic and search though the data at the time the issue occurs?

 

This is also known as network forensics. Network forensics refers to the capture, storage and analysis of digital evidence that flows through your enterprise network. The most complete solutions record every single packet that is transmitted over your corporate networks. So, any emails, instant messages, FTP traffic or any other form of communication that takes place on the network can be reconstructed from the original transmissions. Forensics essentially allows you to reconstruct the history of your entire network.

 

With more businesses relying on the cloud for their IT infrastructure or to deliver their service/products to customers, it's crucial to be monitoring both operations and the infrastructure. While the network has become more reliable, reliance on web-based and cloud-served applications or storage has lead to more frequent outages of that infrastructure. By collecting digital evidence via a network recorder, the once laborious, time-consuming searches (including top talkers, most delays, application type, etc.) involving multiple tools and large transfers of data can be reduced to a quick, convenient search.

 

Consideration #2: Is the problem stemming from one user or many users on the same switch or segment?

 

Determining the scope the problem can point the administrator in the right direction of where to start the network analysis providing and what information is most useful determining and correcting the issue. There are several ways of determining whether or not a problem is stemming from one user or many users on the same switch. One of the more common, but least desirable ways is by monitoring the number of trouble tickets. Calls spike - most users are on the same subnet - this is a telltale sign of a possible hardware problem. A far more proactive approach is to use background analysis and monitor for conditions like non-responsive client or server, or low client-server or server to client throughput. You will quickly see if these issues are being reported for a single client, or across many clients. If for a single client, isolate this client for analysis. Determine what other network activities this client is engaged in, and examine these network flows. This will quickly shed light on the issue. If the problem is stemming from many users, is the problem isolated to a single application, or is the issue broadly affecting overall connectivity? If confined to a single application, that's the place to dig. If the issue is overall connectivity for many users, find the connectivity point common to these users and see check for hardware issues.

 

Consideration #3: Is the problem connectivity or utilization related?

Is the network traffic getting to the specified destination? Is a specific machine over-consuming its allocation of bandwidth and crippling other users connectivity while doing some action? On the utilization front, non-work related, "bandwidth sucking" download activities (music, videos, games, etc) are a common culprit. Utilization-related issues are typically intermittent in nature. One, or perhaps several, clients are over-utilizing a network segment, but that comes and goes. Even if the oversubscribing event is long in nature (like streaming video) the remaining utilization still goes up and down with normal network usage, creating intermittent periods of over-utilization. This can easily be monitored by graphing the network utilization in real-time. Connectivity-related issues are typically more binary - users either can or cannot connect to a particular network segment or a particular application. If the issue is utilization related, the next step is to determine if it is client or application driven. This is fairly easy to determine by looking at the top talkers on the network. If the top talkers are clients, drill down and see what protocols the client is using. This typically reveals the source of the problem quite readily. If the issue is connectivity related, the next step is to determine if connectivity is being affected by network congestion, or hardware problems. Network congestion is again easily seen by monitoring network utilization is real time. If not congestion, then the issue is likely to be with hardware within the user(s) connectivity path.

While 2009 ended with cyber security dominating headlines with the Wall Street Journal reporting hackers had stolen tens of millions from Citigroup, TechCruch reporting about Twitter getting hacked, and the New York Times reporting President Obama naming Howard A. Schmidt as the U.S.'s Chief of Cybersecurity, 2010 picked up right where 2009 left off. Google has been hit, likely via an inside job at their office in China, by a cyber-attack on its network that resulted in theft of its intellectual property.

 

There's a lot more malware-related issues brewing under the surface, as Nemertes senior VP and Network World columnist Andreas M. Antonopoulos points out, "While no new major malware outbreaks made huge headlines, the silent spread of stealthy keyloggers, trojans and botnets continued. As predicted, more computers fell prey to these silent threats while the lack of headlines is broadly and incorrectly seen as 'success' against malware."

 

It's not enough to know you were the victim of a cyber-attack. With today's network forensic technologies, organizations should be able to answer the following questions:

 

1. Who was the intruder?

2. How did the intruder penetrate security?

3. What damage has been done?

4. Did the intruder leave anything behind?

5. Did the organization capture sufficient information to effectively analyze and reproduce the attack?

 

In the past, classic forensic technologies typically provided an incomplete diagnosis because of incomplete reconstruction. In other words, when an attack bypassed a firewall, only partial attack data was processed using the IDS / IPS system, yielding incomplete data and leaving many of the key questions unanswered. Methods are changing. Today, when an attack bypasses the firewall, a network recorder records and aggregates data throughout the attack, supplementing the partial attack data processing of the IDS. With this approach, post-event analysis reveals answers to the aforementioned questions and exposes attacker, method, and damage; with the entire attack recorded the fingerprint is captured and it never needs to happen again.

 

While data recorders will not prevent a zero-day cyber-attack, the information they provide can lead to an informed and efficient security posture within the organization, allowing accurate attack fingerprinting and rapid retooling of security technology and processes to deter similar attacks.

I have discovered a new way to load trace files so that as they are loaded they are filtered.  This is really useful when the filter is an advanced plug-in filter -especially, if the plug-in filters out packets from the file and generates its own packets that are inserted instead.   In my case, I was looking for a way to load a NetFlow file, and have the NetFlow plug-in generate fake packets, instead of loading the NetFlow packets. However, this technique will work for any filter or filter plug-in.  

 

The key to this technique is the SQLFilter plug-in, which can be downloaded from the MyPeek Community Portal (current maintenance required).   It loads packets from a file into a real-time capture window through the SQLFilter Plug-in, instead of directly into a file window.   By using a real-time capture window, there is a user defined ring buffer, instead of a static file buffer.    This makes it possible for the NetFlow Analyzer adapter (current maintenance required) to process and filter out NetFlow packets, and create and insert fake packets into the capture instead.

 

Here are the steps necessary to use this technique:

 

First, create a real-time capture using an adapter that won't actually provide packets when the capture is started. The Microsoft Loopback Adapter is a good example, but there are others.     This is important because certain features, like the PeerMap, won't do anything unless the capture has been started.  Choosing a "NULL" type adapter allows the capture to be started, but not actually capture any packets.

 

It is also important to create a capture buffer large enough to hold the generated packets.   Typically, there will be many more packets generated by the NetFlow Analyzer adapter than there are in the NetFlow trace file.  The alternative is to enable capture to disk, so that the generated packets are saved to a file.

Once the capture window opens, create a new advanced filter on the filter or filter plug-in of your choice.   In the example below, the NetFlow Analyzer is selected.

Next, create a new SQLFilter database and add one or more files to it.   This will add the files to the database. The database file is a single sqlite database that will now contain all the packets for all the files you load into it. This has many advantages, but in this case, we are only doing it in order to load the trace file into the real-time capture window.  In the example below, a NetFlow file is loaded into the database.

 

Now start the capture.   

VRoom!  Do you hear the roar of the engine?  If so, you may have a problem.  You should probably clean out the fan on your computer, or call support. ;-)

Finally, double click on the packet file.  As the packets are being loaded from the database, a progress dialog will appear. This dialog has a Cancel button, so you can cancel the load at any time. On large files, this is a very nice feature.

 

And that's it. I found this to be very useful, and thought I would share.   

Many network monitoring tools are available. All will give you the health of the network, and most will alert you when a problem occurs. However, not all network monitor products provide enough actionable information to really drill down to the root cause of network bottlenecks - that is, network monitoring and root-cause analysis in the same product.

The ability to quickly pinpoint the origin of a problem, thus reducing mean-time-to-recovery (MTTR), is an important business benefit for an increasingly mobile workforce. Additionally, by speeding up the identification of rogue wireless devices - whether a rogue access point or rogue peer (end user computer that has bridging and wireless enabled) - root-cause analysis serves to protect confidential data and critical assets.  Overall, it simply improves the user experience.

To do root-cause analysis, you first need to choose a network monitoring approach that collects the appropriate data. As WildPackets' Jay Botelho wrote in October, there are three primary data sources for network monitoring solutions: Simple Network Management Protocol (SNMP), Flow Records, and the Packet themselves. Regardless of which approach you chose, you're going to have to make compromises. Two metrics that are useful when making those compromises are data granularity and data accuracy.  The compromises you make here determine whether or not you're able to do root-cause analysis.

For example, a help desk may receive a call where a particular user is having a problem with a particular application. This might go unnoticed with a flow record-based approach, as the high-level alerts that have been configured may not flag this for attention. In this circumstance, having the packets and the payload can be important. As all packets are captured (unlike flow records, which rely on statistical sampling), packet-based network monitoring provides information that is 100 percent accurate for each flow. As Botelho notes in his article,

"Generally speaking, you want to use the appropriate monitoring technology for the appropriate need. If you just need to check the status of a device, then SNMP may be all you need...If you are interested in sampled high-level information about who is talking to whom, approximately how much traffic they are generating or receiving, then flow-based analysis may be fine...Lastly, if you need all the detail about what is happening on the network, as well as possibly being able to go back in time to prove what happened on the network, then a packet-based solution would be best."

By planning appropriately and considering issues like data granularity and data accuracy, organizations can move beyond network monitoring and set themselves up for root cause analysis with an approach and solution that best fits their needs.

The use of VoFi, or Voice over Wireless, has been rather limited. But now, with the newly ratified 802.11n standard, we're expecting to see a surge of interest in this technology since 802.11n and its increased throughput and range is what makes VoFi feasible. 

Three benefits of VoFi are:

• Reliable coverage
• Moving billable, cellular minutes to Wi-Fi
• Increased mobility
  

We all continually suffer through the issue of poor cellular coverage indoors, whether at home or in the office. VoFi and VoFi enabled phones provide the capability to transition calls and data activity from cellular to Wi-Fi when in range of an 802.11 network. Since 802.11 is typically deployed to cover indoor spaces, like your home and office, call and data quality will be dramatically improved indoors with VoFi enabled technology.

An added benefit of transitioning a call to your 802.11 network is that it reduces cellular usage, saving minutes on pay-per-minute plans. Granted, this hand-off is still being worked out between carriers and equipment manufacturers, and may not result in a complete minute-for-minute reduction in usage, but more than likely some level of savings will be realized, allowing you to much more quickly capitalize the expense of an 11n upgrade by eliminating some of your billable cellular traffic and carrying it on your 802.11 network.

802.11 has always been about mobility, but up until now it's been manifested more in being able to move from your office to the conference room with your laptop and maintain connectivity. VoFi significantly extends mobility by including voice communications as well. You no longer need to be tethered to a desk phone, or limited by the base-station range of a cordless handset. Wherever there's 802.11 coverage there's voice coverage. This technology was already in use by some industries, large retailers for example, allowing customer service reps to wander the store while helping customers. But 802.11n and VoFi will take this to the mainstream, both in the office and at home.

A key element of VoFi, of course, is the voice component. It's very similar to VoIP in that it's susceptible to jitter and latency, and thus dropped calls, interruptions, and other issues. As a typical wireless network has more latency and interference than a wired network the susceptibilities are that much worse. So with this new technology comes new problems. Are you prepared to manage your new VoFi environment?

On November 18, we're hosting a webinar to explain how best to manage your VoFi environment.

Network administrators often come to us with problems regarding the speed of their network - it's simply too slow. This is a major issue for many companies, as businesses depend on reliable, fast networks 24/7. A slow network can be frustrating and even costly. We've all experienced slow networks and know it's not fun.

In order to optimize network performance, it's critical that the network administrator has complete visibility of their network traffic. A number of factors can contribute to the slow networks problem, but by quickly analyzing faults from multiple network segments and drilling down through multiple layers of analysis, IT managers can pinpoint the exact issues that need correction.

In this whiteboard video, we identify four possible causes of a slow network that you consider while investigating your own organization's network bottlenecks.

Yikes... This week Sega exposed some of Sony's highly sensitive future plans. Information regarding Sony Playstation 3 and motion controllers discussed in a meeting with Sega were leaked in a document that made its way onto Sega's press site.

 

So, who is responsible? How did this happen? If this happened in your company how can you find out? Enter network forensics.

 

Network forensics refers to the capture, storage and analysis of digital evidence that flows through your enterprise network. The most complete solutions record every single packet that is transmitted over your corporate networks. So, any emails, instant messages, FTP traffic or any other form of communication that takes place on the network can be reconstructed from the original transmissions. It doesn't get any more accurate than that. Network Forensics essentially allows you to reconstruct the history of your entire network.

 

IT personnel utilize network forensics to analyze historical network traffic to conduct or assist in many types of investigations. A few common applications for Network Forensics include HR compliance, intermittent issues, security cyber attacks and transaction analysis. This often starts with terabytes upon terabytes of data. Some tools, like OmniPeek, allow you to analyze data at the point of capture, thus eliminating the need for large data transfers (which are typically done) that consume time and bandwidth. OmniPeek also provides simple and intuitive means to drill down into the relevant data, making easy work out of finding the needle in the multi-terabyte haystack.

 

Using network forensics, you can track down the culprit. Of course, network forensics has many uses other than hunting down perpetrators, but it can be helpful in uncovering sensitive leaks. If they're not already, Sega should be using network forensics to get to the bottom of this snafu.

 

If anyone had doubts about the strength of the network monitoring/reporting/troubleshooting market, this week's news should put those doubts to rest. Of course we're referring to the announced purchase of NetQoS by CA for $200M in cash. This is a multiplier of approximately 4 over the reported 2008 annual revenue of $56M from NetQoS. Not a bad deal indeed for the current economic climate.
 
Though not sexy, and often considered commoditized, the network monitoring/reporting/troubleshooting market may be just the opposite. At the core of this market is packets. Yes, I said packets. Geeky, techie, call it what you want but we're back to basics on this one. Accumulating statistics on the basis of packets, whether through direct, deep packet inspection or through the use of flow-based technology available in most contemporary routers and switches, is proving to be a highly valuable capability, especially as networks grow and IT staffs shrink.
 
For years it seems the industry tried to steer clear of packets, looking to higher-level solutions like SNMP, but in the end it's becoming clear that packets never lie, and to obtain reliable network data one must start with network packets.
 
Now, I think we're going to come full circle on this one. Even the solution from NetQoS - which relies on flow-based information from routers and switches - can't provide the level of detail compared to complete packet inspection.  And this technique provides no capability of unequivocally determining a root-cause for reported problems. It merely shows network trends and alerts when thresholds are exceeded - it does not allow for detailed analysis as the offending data has already passed through the network. That's why NetQoS partnered with Network Instruments and OEM'd their Gigastor solution. With that partnership, NetQoS could provide the full monty - network reporting, monitoring, and troubleshooting. Or, as I see it, coming full circle back to packet capture and deep packet inspection -- the only way to truly identify network issues.
 
The purchase of NetQoS by CA provides validation that the network monitoring/reporting/troubleshooting market is not only alive and well, it is thriving. The traditional, "full-featured" network management solutions (like CA) are realizing that quality network monitoring, reporting, and troubleshooting are key features, and that they need capabilities like flow-based monitoring and deep packet inspection to continue to claim that their solutions are complete. The value of these types of solutions continues to rise, and the "big boys" have only two options, start developing fast or acquire. And with the head start many of us have with our network monitoring/reporting/troubleshooting solutions acquisition is by far the better alternative.

As Joanie Wexler points out in her recent Network World article "Prepping for (finally!) a standard 11n world," the imminent ratification of the 802.11n standard will push enterprises to be more serious about investing in 802.11n. Though some early-adopters have already jumped in, either just to test the waters or because their wireless application plans demanded increased performance, most enterprises have been holding off for the final ratification. For those enterprises entering the 11n water for the first time, the Network World article offers some good preparation tips, whether your entry is from the 3m board or a slow stroll in from the shore.

 

In addition to the tips already offered, several other important points come to mind as you prepare your entry. And you guessed it, our tips center around network management.

 

First, the benefits you'll realize as you move towards 11n will likely have you rethinking the way you use wireless, so what better time to also rethink how you  manage wireless. It goes without saying that your wireless management infrastructure will need to be upgraded to include 11n. Some management applications are just getting there, while others, like OmniPeek, have been there for many years already with a substantial amount of real-world testing, not to mention the use of OmniPeek as part of the Wi-Fi Alliance 802.11n interoperability testing. A move to 11n will most certainly include a move to WPA2 for security, if you haven't already made that move, increasing the need for a network management solution that handles both wireless and wired traffic simultaneously so you can monitor your 802.1x authentication all the way back to the wired sources. And with the increased bandwidth of 802.11n, you'll likely be considering applications like voice-over-wireless, which will require additional measurement techniques like wireless roaming to ensure proper operation of your network and ensure wireless call quality. Basically the message is this: plan for wireless management up front as you make the transition to 11n and make sure your wireless management solutions meet the demands of the new applications you intend to deploy.

 

Second, this is an excellent time to consider HOW you plan to monitor the wireless network, either for troubleshooting or 24x7 observation. Wireless networks are becoming much, much larger, and the days of walking around with a laptop running wireless analysis software to do troubleshooting are drawing to a close. However, wireless networks still require a "point of presence" to do adequate monitoring and certainly any troubleshooting, meaning data must be collected near the source of the reported problem. "Overlay" networks have been the standard solution for the past several years, but this is expensive solution requiring duplicative hardware and network resources (network drops, router ports, etc.). This can be mitigated during your 11n planning by designing in just a bit more density in your AP deployment and then relying on wireless management solutions that can leverage deployed APs and turn them sensors when monitoring or troubleshooting is required. This solution is highly cost-effective since the additional density typically only results in about a 10% increase in the number of APs, much less than the number of dedicated sensors you would need to deploy, and every AP can be put to use in the network resulting in even better network performance when not in use as sensors. This is an extremely important consideration as you roll out a new 802.11n deployment and the cost savings over a traditional "overlay" solution can be substantial.

 

So, whether you're diving in head first or just putting in your little toe, this is the time to reconsider not just network upgrades and the new applications you wish to introduce, but the new management challenges for the network as well. The increased throughput, increased mobility and increasing integration between your wireless and wired network put new demands on your wireless network management solutions. Make sure your solution has already proven that it can meet these demands. OmniPeek has been doing this for years.

Conficker is a computer worm targeting the Windows operating system that was first detected in November of last year. Wikipedia explains that Conficker "uses flaws in Windows software to co-opt machines and link them into a virtual computer that can be commanded remotely by its authors." A recent New York Times article reports that Conficker has more five million computers under its control, with estimates from other sources being much higher.

 

So, what is Conficker used for? Generating vast amounts of spam? Stealing information like passwords and logins by capturing keystrokes on infected computers? Delivering fake antivirus warnings to trick naive users to pay by credit card to have the infection removed? Researchers speculate that all of the above may be true. They do agree that the cluster of Conficker-infected computers yields massive, though mostly untapped to date, computer power.

 

There are two primary steps a network administrator needs to take to detect Conficker: apply filters and isolate data by connection.

 

Apply Filters: Filters are a powerful tool in any network management/analysis solution. Filters serve a dual purpose by limiting the number of packets targeted for analysis while isolating packets with key characteristics - in this case those characteristics unique to the Conficker worm. Conficker is known to take many forms, with new variants showing up routinely, making detection that much more difficult. Again, filtering is the perfect technique for keeping up with Conficker.  Filters are typically easy to construct and even easier to modify, so by investing a few minutes whenever a new strain is identified you can keep your network safe.

 

Isolate Data by Connection:        Conficker is known to use HTTP pulls and P2P push/pull to find peers and move around payloads. By focusing on these specific protocols and relying on characteristics of Conficker, like the use of pseudorandom domains of specific lengths in the case of HTTP and custom UDP protocols in the case of P2P, you can isolate suspect client machines. Filters are again a valuable technique, but since these pseudorandom domains are not known in advance, "negative filtering" is more appropriate. In negative filtering, filters are constructed to filter out all your typical, well-known traffic, leaving only packets of a suspicious nature. With this technique you hope to never see a packet - that means there's nothing suspicious on your network.

 

To be successful in detecting Conficker efficiently, a network admin should be able to automatically extract or fetch network data using one or multiple parameters, such as source/destination IP address, source/destination port, time, date, protocol, string, and more. Each kind of expression (IP, MAC, Protocol, Port, Pattern, Value and Length) should be able to be searched individually or in combination with operators (and, or, not, Group) to extract the required data from gigabytes or even terabytes of captured traffic. Solutions like WildPackets' OmniPeek Distributed Analysis Suite do just that.

 

Conficker is out there. Make sure it's not on your network.