WildPackets Network Analysis and Monitoring Blog

The latest flack around Google's joint announcement with Verizon on preserving the open Internet has had its fair share of media attention. Here is the source document that created the latest fracas: Verizon-Google Legislative Framework Proposal.

The joint Verizon-Google proposal addresses nine specific elements focused on preserving the open Internet (net neutrality.) These points include: consumer protections, non-discrimination requirement, transparency, network management, additional online services, wireless broadband, case-by-case enforcement, regulatory authority, and broadband access for Americans.

It's not our intent to address each point individually. Most are self-explanatory. What we want to do is highlight the largest issue with this attempt to make the Internet more accessible. That being said, Verizon and Google are essentially claiming wireless is different, and that data on wireless networks need not be as "neutral" as data on wired networks. According to the Wireless Broadband element, "Because of the unique technical and operational characteristics of wireless networks, and the competitive and still-developing nature of wireless broadband services, only the transparency principle would apply to wireless broadband at this time."

This is a pretty far-reaching statement, as it proposes to exempt wireless broadband networks from the consumer protections and non-discrimination requirement outlined in the proposal - the basic elements of net neutrality. The consumer protections element says service providers cannot prevent sending and receiving lawful content, running lawful applications, and connecting any legal devices to the network. The non-discrimination requirement prohibits service providers "from engaging in undue discrimination against any lawful Internet content, application or service in a manner that causes meaningful harm to competition or to users." It goes on to be even more explicit: "Prioritization of Internet traffic would be presumed inconsistent with the non-discrimination standard ...".

If these proposals are accepted, providers of wireless services CAN block lawful traffic, applications and devices; and they can prioritize Internet traffic according to their own desires, presumably giving advantages to data or media from certain sources. But how did we get to this point? Aren't wired and wireless networks essentially the same, other than the fact that wired networks got a head start? There are obviously some technological differences, but the underlying usage of the networks is the same, and that's what defines a network - its usage. The same rules should apply to both wired and wireless networks, including blocking and prioritizing traffic differently depending on the source of the data. To make the claim that they should be treated differently because broadband wireless is "competitive and still-developing" seems quite self-serving. Couldn't the same claim have been made for broadband-wired services 10 years ago?

We are very interested to see if the proposals get accepted and how this debacle will ultimately turn out. There is definitely something to watch here, especially if you share our view that wired and wireless networks are created equal...

There's a chance you could have a spy. Watching your every move, just waiting for the perfect time to attack and hijack your precious information.

A recent InfoWorld blog serves as a wake up call to those companies who have not taken the increasing threat of electronic espionage and network security seriously. According to the blog, a growing number of companies are being spied on electronically by sources in other countries. This isn't the first we've heard about this though, back in January, hackers from China had broken into several companies' computer networks including Google to steal information about Chinese dissidents as part of "Operation Aurora," which was one of the largest cyber-attacks ever.

These incidents keep occurring because companies believe that their current security software is good enough or they've just simply ignored the issue. The truth is that in order to have protection from these types of stealth spies you have to collect packet history within the network. And the only way to receive this information is by performing forensic analysis.

Network forensics is the capture, recording, and analysis of network events. All pertinent network traffic is collected in a single location, rather than scattered across the network. Data is captured in a common data format and does not need to be transferred or translated in any way for analysis. Using network forensics data mining tools, security teams can reconstruct the sequence of events that occurred at the time of a network breach or cyber attack and get the complete picture. Forensic analysis exposes attackers, methods, and damages. Lucky for us, new and more powerful network forensic products are out there to help defend against electronic spying threats. Even though there is a vast array of network forensic technologies to choose from, organizations should know that there are really only three basic elements to any general-purpose network forensic solution:

1. Data capture and record - This is the ability to capture and store multiple gigabytes of data at high network throughput (for example, 10 Gigabit) without dropping or missing any packets. Every network forensic solution has its limitations, including sustainable throughput, packets per second, data management, search functions, etc. These limitations can and should be determined through practical lab tests, and the results should be repeatable and documented. This includes both wired and wireless networks.

2. Data discovery - Once data are recorded on the storage media, the solution should provide a mechanism to filter particular items of interest, for example, by IP address, application, context, etc.

3. Data analysis - Finally, you want some built-in assistance for examining the patterns and anomalies found during the discovery process to help you determine what actions were recorded in the captured packets.

The information forensic analysis provides can lead to an informed and efficient security posture within an organization to deter similar attacks in the future. As criminals get smarter and savvier, being able to detect and characterize attacks is crucial. Information leakage not only results in monetary losses but also can be a serious threat to national security. Having the right network forensic solution in place can help to discover and eliminate possible threats in your network and to provide lawful interception capabilities when needed.

With SaaS (software as a service) adoption continuing to increase, some have postulated that appliance-based network analysis and troubleshooting is in jeopardy. In a SaaS model, an application is hosted by a provider and then offered as a service to enterprises across the Internet. In other words, no on-premise hardware is required. This type of software delivery is attractive because it removes the requirements to download, install, and maintain software on-premise. Not having to buy the IT infrastructure is also on obvious financial benefit.

SaaS is taking off. In fact, Gartner recently forecasted that SaaS' worldwide revenue would surpass $8.5 billion in 2010. That doesn't mean it's the right choice for everything, namely network analysis and troubleshooting. We believe network analysis is difficult and complex and the process cannot be effectively transitioned to a service via the cloud. We believe that network analysis and troubleshooting is still best handled with on-premise software and hardware appliances. Here's why:

1.  You still own the network

Just because you've outsourced the application, doesn't mean you've outsourced the network. And odds are your internal network has grown to be quite complicated, and your users are relying on that network for access to your SaaS provider. Ongoing monitoring, with a capability to drill down into suspicious issues and quickly achieve root-cause analysis, remains as important as ever. Perhaps even more so, if the issue is with the SaaS provider and they do not own up to it.

2. Independent (and in-house) verification is key

You have the vested interest in the performance of your overall network design, whether that design is totally within your control or involves third parties like SaaS providers. Relying on a third party to monitor your network is like ignoring your vehicle's check engine light. You know the light is on; you have no idea what's wrong and no ability to dig in and figure out the problem.

3. Multiple SaaS vendors, then what?

Once you start down the SaaS track, you'll likely work with multiple application providers for best-of-breed applications. It's imperative that network monitoring, analysis, and troubleshooting remain within your control. What if applications interact? Who will notice, or care, besides you? And what about network latency, between you and your SaaS providers or perhaps between the SaaS providers themselves? The best location to measure latency is within your own network, right where your users are. This will give the most accurate latency measurements and constant monitoring will ensure acceptable end-user performance.

Network speeds are constantly evolving and the arrival of 10G brings a whole new set of challenges for network managers. Proper network analysis and troubleshooting is key in this high-speed world as our networks become even more complex. While SaaS may be convenient and inexpensive, appliance-based network analysis and troubleshooting is the best way to ensure organizations continue to meet critical operational practice needs. We do believe, however, that the future of enterprise computing is not going to just be on-premise or SaaS. Instead, they will likely exist in symbiotic harmony.

Keylogging tracks the keys struck on a keyboard in a discreet manner so that the person using the keyboard is unaware that their actions are being monitored. There are several keylogging methods, ranging from hardware and software-based approaches to electromagnetic and acoustic analysis.

A while back, I found evidence of a keylogger on a local government computer... hunting time.

Using Wildpackets Etherpeek (now included in Omnipeek) as my weapon of choice a live analysis was performed. It turned out that a "smoking gun" email had started it all by explaining how to install the keylogging software. This is obviously a concern, especially in a government setting where information is at high risk of being compromised.

So, what had been stolen? If no keystrokes were captured there wasn't much to worry about. I went to work.

A clone of the computer's hard drive was created to connect the machine to the Internet. Before plugging in the Ethernet cable, I made sure to limit the export of data.

The restored computer was allowed to connect to the Internet through a firewall, which only allowed it to get DNS information. By hacking the Windows hosts file connections were directed to a test machine that was set up with a fake SMTP server. The computer was turned on and text was typed into Notepad. 

If there was an active keylogger, this text would have likely been picked up and emailed off with previously recorded activity. Not long after plugging the Ethernet cable in, I saw activity. The test machine was monitored with a Peek-equipped PC. All the traffic between the restored computer and the test machine was recorded, thanks to Wildpackets.

The first intercepted traffic included the material below:

This information came from the keylogger report that was being sent to an offshore email account. The good thing was that classified reference related to staff categories was not secret government information.

Next came the text that I had typed on the restored machine keypad. This was confirmation that the keylogger was stealing information.  A bit more analysis defined the scope of the loss, allowing repair of the damage.... and identification of the person who installed the keylogger. RIP, keylogger.

Network troubleshooting using deep packet inspection is a tried and true approach for network engineers - no one would ever doubt this approach when a difficult problem is plaguing the network. But suggest the use of deep packet inspection for troubleshooting slow applications and you'll likely get some blank stares. Deep packet inspection is the domain of network engineers, not application engineers, right?

Not necessarily. Analyzing decoded packet headers is definitely more suited to a network engineer, but what about the payload data from all the packets? Most network engineers find little value in the payloads, often times they filter them out to conserve analysis resources. But payloads can be of high interest to application engineers investigating application problems, including slow response time.

Consider the example of a help desk application used by a large online retailer. Support professionals, who access the help desk application for each customer call they receive, begin to experience slow response times from the application and of course the initial report is that the network is the problem. A network engineer begins his investigation and after a short time, calls in the application designer stating the problem is the application, not the network. "How do you know that," asks the application engineer? "Simple", says the network engineer. "Your application is generating the following messages. Your server command was deadlocked with another process and has been chosen as a deadlock victim. Re-run your command. And The rollback transaction request has no corresponding BEGIN TRANSACTION". Flabbergasted, the application engineer exclaims, "Where did you get that information?" "From the packet payloads involved in the slow response time transactions on the network, using deep packet inspection network analysis troubleshooting. You should try it sometime."

Deep packet inspection can provide greater value than simple network troubleshooting. Application engineers can certainly benefit, both in troubleshooting application problems and analyzing the overall behavior of an application before trouble is reported. The following four steps should be done to quickly isolate and analyze specific application data:

1.  Capture data at the appropriate location
For application analysis and tuning, it's best to locate a distributed network analysis software probe or appliance in the data center in line with the application and database servers. This will capture the data for all application users, whether local or remote.

2. Save packet data to disk
By saving packets to disk and employing post-capture analysis, you can take your time in doing your analysis, without worrying about missing key data.

3. Filter stored packet data for the application of interest
Some solutions make this easier than others, but this step is crucial in creating a data set that is manageable and applicable to the application you wish to analyze. This can often be done by filtering out all traffic except traffic that has the source or destination IP of your application server or servers. If it's a troubleshooting exercise and the problem is isolated to a particular client, you can even further refine the filter to just the IP to IP conversation between the client and the server.

4. Use built-in analysis features to look for common faults

Before diving in and looking for complex, one-off faults, use the built-in fault detection and analysis capabilities of your analysis software (again, these features can vary wildly between competing solutions) to look for common issues, like one-way traffic, database client errors, slow server response time, failed login, resource errors, etc.

Overall, even though you may get some awkward stares for suggesting deep packet inspection  to troubleshoot slow applications, in the end it's worth it because, you'll not only keep the network healthy but employees happy and productive.

Our blog last week explored a common misconception that even though several protocol analysis and troubleshooting solutions claim "line-rate" analysis, the actual rate at which packets are written to disk without data loss varies greatly -- the majority of solutions have a capture rate of somewhere in between 4-6 Gbps. We are excited to say that's no longer the case with our newest product, TimeLine Network Recorder!

They say the truth shall set you free...

The fact is that today's networks are actually faster than the available network analysis and troubleshooting solutions, resulting in greatly diminished network visibility. This is significant to network administrators because the 10G technology enterprises are actively deploying can be difficult to troubleshoot, and until now, no vendor has been able to capture at the half-duplex line-rate without packet loss. However, achieving both real-time visibility and historical network traffic storage for post-incident analysis is possible in 10G environments - you just have to add clarity to your analysis by being specific and a bit more selective.

Three tips for being more selective when analyzing 10G Ethernet traffic:

1. Understand the network and the data you need to collect

Do not try to blindly move forward and perform analysis without knowing what data matters most to your organization. It's important to know exactly what you want to capture and what information is going to be beneficial for your analysis. Your requirements will likely vary between each network segment and you are likely going to have to capture data at several locations. The key is to use post-capture analysis and only save the data to a disk in real-time. Trying to capture and analyze simultaneously, in real-time, on highly utilized network segments puts too much strain on the system.

2.  Capture only what you need

There is a great temptation to try to capture and analyze everything because enterprises fear that the source of the problem is not immediately known. When it comes to 10G Ethernet traffic, analyzing every bit of data is nearly impossible  due to the volume of data. However, if you know your network well enough, certain conditions can be immediately ruled out. By using these clues to limit the collection and analysis to only what is necessary, you can dramatically improve network analysis performance.

3. Limits are everything

Even after analysis has been streamlined to only essential areas of the network, data capture for network analysis on 10G networks generates a great deal of data quickly, and managing the data becomes a significant challenge. The data is typically stored for subsequent retrieval and post-capture analysis. The two most common formats are standard packet files and databases. In either case, two metrics to manage closely are file size and frequency of disk writes. If the files are too large they will be unworkable on the computer being used for analysis. Smaller files lead to more frequent disk writes, and this can rob the system of resources for performing the actual packet capture. Optimum performance is achieved with a balance of these two demands, and this is different depending on the hardware resources available. After a few captures, you can determine if either of these parameters can be better optimized for your system.

Until a solution is developed that doesn't just claim "line-rate" analysis but actually allows packets to be written to disk without data loss, these tips are helpful for keeping an organization up to speed with 10G traffic. In the end, paying careful attention to detail when configuring network management systems will reward you with the analysis and troubleshooting results you desire.

It's called World Cup fever for a reason. With millions of people watching and reading related news online, it undoubtedly increased Internet traffic. In fact, people apparently care more about soccer than they do the presidential election. The very first day of the tournament, traffic exceeded the previous record of 8.5 million views per minute (vpm), which occurred when Barack Obama was elected. According to measurements by Akamai, traffic for news sites on June 11 started to climb steadily at 6 am ET and peaked six hours later, reaching nearly 12.1 vpm.  And the fact that June 11 was a workday didn't stop people from watching. People turned to their office computers to follow the action. Being able to identify high talkers and effectively manage traffic is a must for organizations that want to stay productive and successful.

Below are some considerations for enterprises in regards to their networks during times of high traffic: 

Understand how your network performs normally

The only way a company can improve network performance is to know where they stand in terms of network demands. Then they can measure success against those baselines. By looking at Internet connections, WAN links, WLAN environments and the data center, enterprises can get a sense of how their network normally acts. A network analyzer can also help organize this information into a report that can be used to not only solve issues that currently exist, but also to allow organizations to go back in time to validate performance and bandwidth utilization as necessary.

Prune and clean WLAN traffic

Remove unnecessary traffic. Devices like printers, support stacks and protocols that aren't in use in the environment can be eliminated. Sometimes, protocols that help manage the network, like routing protocols and SNMP can be found, hogging valuable bandwidth without any purpose.

Know your options

Consider a Web surfing policy. The 2010 FIFA World Cup lasts from June 11 through July 11, 2010 and will likely suck up a lot of bandwidth. Having a set policy in place will help keep traffic down and is an option to be explored. However, getting employees to abide by regulations it's often more of challenge than it's worth.

The fact is, it is important to see new trends approaching and make changes to the network to account for an enterprise's behavior. Popular events can erode profitability and corporate security, as employees not only watch but also participate in social media discussions and gamble online. Review network analysis measures and policies now, so when 2014 rolls around, networks stay healthy and humming.