WildPackets Network Analysis and Monitoring Blog

802.11n is a newer standard of WiFi LAN, or wireless local area network technology. For context, preceding standards have equally "exciting" names including 802.11a, 802.11b and 802.11g. Joking aside, in what seems to be consistent in the world of technology, the hype around 80211.n in the trades and blogosphere outpace current realities and deployments.

 

In fact, while not yet mainstream - to date it has not been ratified as a standard by the Wi-Fi Alliance - 802.11n has been mentioned in articles and blogs more than 1,000 times in the past six months according to ITDatabase.

 

Beyond the general appeal of 802.11n being the latest and the fastest WiFi LAN, there are certainly benefits with 802.11n - three worth mentioning in particular.

  

1) Better range

 

While users can expect a variance of capabilities in client devices using 802.11n depending on a host of variables, 802.11n products for consumers and small businesses deliver at least twice the range of 802.11g products. If you add in enterprise Access Points (AP), the range can grow well beyond that.

 

2) Runs on 5-gigahertz spectrum

 

802.11n runs on the 5-gigahertz spectrum, which is beneficial because there are fewer devices in that spectrum. By comparison, 802.11b and g run at 2.4- gigahertz. Many have had to move from b and g just to avoid interference with microwave ovens, cordless phones, Bluetooth, etc. Their environments were too noisy yielding too much interference. They literally had to change technology - to 802.11a, which also runs on 5-gigahertz spectrum - to fix their issues.

 

3) Backwards compatibility

 

As mentioned above, 802.11a runs on the 5-gigahertz spectrum while b and g runs at 2.4- gigahertz. Here's the cool part - n also runs on 2.4 and 5. What's the significance? Backward compatibility. Prior, 802.11a was not backwards compatible with b or g. It was a cost benefit issue on changing technologies. Now, those who have invested in either spectrum have an option to start upgrading to n. They simply take parts of their environment as necessary to the 5-gigahertz spectrum, which removes a lot of their interference while at the same time having complete backwards compatibility with all their b and g technology.

OmniPeek is a tool, like a hammer, but not just any hammer.   It is like Thor's hammer, and when you wield the OmniPeek hammer, you are like Thor, the God of Thunder.    You save mortal men every day.  You monitor and troubleshoot their networks, you solve their problems.   You make life better for everyone.  Yes, with OmniPeek, you are a superhero.

And if you are like me, you probably use OmniPeek in some form or another every day for network monitoring, VoIP analysis, network performance analysis, network security, or any of the other capabilities of this analyzer.   And before that, you used EtherPeek, and probably AiroPeek as well.   These products have always been easy to use, provided you with powerful analysis capabilities, and of course the UI looks great.    But as a superhero, you have a lot to do these days.   You either have to many products to test, or more than one network to monitor at the same time.

As a result, you may find yourself performing the same functions over and over again.   And you may ask yourself, how do I work this?   And you may ask yourself, where is that large automobile?   ;-)  Ok, maybe not, but although you may have the test or monitoring process down to an exact science, and a precise number of clicks, you are still not making the best use of your time.   

Sometimes it can be difficult to recognize this, or admit that it is true.  But that is ok, and that is why we are here.   We want to help you help yourself, by introducing you to the amazing world of automation.   Through automation, you can save lots of time and money.   'Nuff said, it is worth it.    To take advantage of automation, you do not have to be a programmer either.   It can definitely help, but we have developed some command line utilities that do the scripting for you. 

But, there is a catch.   In order to get the most out of automation, you first need to upgrade your OmniPeek Console to an OmniEngine, if you have not already.    That is the ante, and you will be pleasantly surprised by how little it costs to upgrade.    Yes, you can automate the OmniPeek Console, and if that is your only option, I would highly recommend it.   There are lots of benefits.

However, if you can upgrade to an OmniEngine, then do it.   Do it for all of the right reasons.   Do it for your girlfriend, or wife and children, who would like to see more of you.   Do it for your boss, who would like to save money.   But mostly, do it for yourself because you are not just a button pusher.   You are smart enough to know that if you put in a little time and effort up front, you can have a better life because of it.   And finally, do it because you can.   The fact is that the OmniEngine is the best platform for automating network monitoring and analysis processes.   

The OmniEngine is extensible in a number of ways, and that is why we call it a platform.  But today, we are focusing on automation.    At a high level, automating an OmniEngine consists of creating a capture, and analyzing the results.   But wow, talk about making a long story short.  Captures have lots of settings, and filters, and adapters, and statistics, and on and on.   And then there are the results; fast vs. slow, up vs. down, pass vs. fail, and all of the analysis required to come to that conclusion.    But for for the processes you perform over and over again, you probably know the exact set of steps required to create the capture, configure the capture, how long to run the capture, and what analysis is required to determine the result.    With OmniScript and the OmniEngine command line utilities, you can automate most, if not at all, of these steps.

So what is OmniScript?   OmniScript is the remote API used by the OmniEngine Command Line Utilities.   With OmniScript you can write scripts that control the OmniEngine, and  can virtually anything that the OmniPeek Console can do.    But with OmniScript, you can write these scripts in such a way that they can run by themselves from start to finish.    What we do here at WildPackets, is write command line utilities that can automate the OmniEngine in different ways, so you don't have to.   You just configure the utilities with all the right parameters, and integrate them right into your work flow.  

The most important thing to understand about automation, is that it is not all or nothing .   Automation is something that you can add a little bit at a time, and every time you add some, your life gets better.   So go ahead, take the next step.   It's easy, just read more about automation, in our Automation Primer on MyPeek.   And then, when you are ready to take the red pill, download OmniScript and the OmniEngine Command Line.

---

My name is Chris Bloom (aka spacepacket), and I do wild and crazy things with OmniPeek.   If you have any questions about automation, or any other questions about the extensibility of WildPackets products in general, please email me at bloom@wildpackets.com

 

 

At Interop 09 this past week in Las Vegas, WildPackets asked attendees of our mini-courses three questions:

1. What types of networks do you manage?
   
2. What is your average network throughput?
3. What are your primary usages for network traffic analysis software?

Results
NOTE: With the exception of question 2, respondents could select more than one answer. For question 1 and 3, percentages will not add up to 100%.

• Types of Networks Managed by Respondents:
  View image
• Average Network Throughput Reported by Respondents:
  View image
• Respondents' Primary Usages of Network Analysis Software:
  View image

Findings
Some of the findings mirrored trends that journalists and analysts have reported on. For example,

• Over 59% of the respondents managed a Gigabit Ethernet network. 35% of the respondents who reported managing a Gigabit Ethernet network managed a 10 Gigabit Ethernet network as well.
• Over 61% of the respondents who managed VoIP/Video in their network, managed either a Gigabit Ethernet network or both a Gigabit and 10 Gigabit Ethernet network.
  
• Over 34% of the respondents do not manage 10/100 Ethernet networks.
• 7% of the respondents manage only a Wireless network. Likewise 7% of the respondents managed only a 10/100 Ethernet network.

One trend we were surprised to find was that very few respondents (7%) use network traffic analysis software for post-capture (forensic) analysis. The majority of respondents continue to use the software for either local (55%) or distributed (41%) real-time troubleshooting and analysis.

Participate
The mini-course session attendees are not a broad sample of North American network professionals. Many of the respondents were from California, Arizona, or Nevada. To get a better representation of the networks managed today by network professionals, we're opening up the survey online from Thursday May 28 through Friday June 19th.

As a thank you for participating in our survey, we'll enter your name into a drawing for the latest iPod Touch. Your personal information will not be associated with your answers. If you prefer, you can also participate in the survey anonymously. We'll publish the results in the July Peeks newsletter, as well as announce who won the drawing.

By Jim Thor - WildPackets Professional Services

Introduction
Times have changed all around us, and they keep on changing. One thing that hasn't really changed is the view many people have about Protocol Analyzers and the function they play in today's networks. Historically, Protocol Analyzers were used in cases where nothing else worked. They were a break/fix tool, meaning that they were generally only used to find and fix 'broken' network or system issues. And usually, possibly many hours after an issue arose, is when the protocol analyzer would be taken out of a cabinet and put to use (more as a last resort then an active participant in the troubleshooting process). Not to say that the break/fix use of an Analyzer is a bad thing (it is often the only way to see the truth), as that is still one of the core competencies of a protocol analyzer. Another issue encountered by only using a Protocol Analyzer once in a while to fix an issue or problem is that it substantially lowers the proficiency scale for the use of the tool. If you only use it once or twice a month or even less, it is very hard to become proficient with it and its features, and therefore, the longer it will take to find and solve your issues. Not to mention that once you are more familiar with the analyzer, you will be able to take advantage of some of the more automated features like triggering, alarming, and reporting.

Background
So let's briefly compare what we are going to be talking about, Protocol Analyzers, with some of the other technologies that are used today to help manage the network. The two most common technologies in this arena are probably Netflow and SNMP. As these are both helpful and very useful in the right scenarios, they both are limited and have significant deficiencies when compared to a Protocol Analyzer. Netflow's major deficiency is that it is a sampling technique. As this may be okay for a high level view, it does not give you a completely accurate representation of the network, nor does it give you any detail if you wish to dig deeper. SNMPs major deficiency is that it is either a polling technology or an alert technology (using traps). Once again, this isn't detail oriented, and you can only get information during a polling period or when a trap is generated. And, SNMP has a certain amount of items in the tree that it watches and reports on, not everything and anything that is happening on the network. This is where Protocol Analyzers shine. They show you everything that is on the network, unless you choose otherwise. They will report or notify you on anything you ask them to, often down to the bit. Not sampled, not on a time schedule, and not only when bad events happen.

The Need
So, as I started off saying, times have changed. In the current world of Protocol Analysis, the tools have become much more than just a way to see the packets and the details of what is in those packets. Often I speak to people that tell me that if no one is complaining, then the network is good. Let's face it, if a tree falls in the woods and you don't hear it, does it mean nothing was damaged? Today, a protocol analyzer will help tell you about the past, as well as the potential for future issues or trends. The other feature of a good analyzer is the visualizations that will help to provide proof to others as to where the problem is or isn't. This is a huge factor in today's network environments, and the fact is, that networks consistently get blamed for issues even when the issue is not related to the network. Being able to quickly and effectively show visualizations of the network and its performance will make you look like a ROCKSTAR. And this is where we are going to focus the rest of this paper, using Protocol Analyzers in the year 2009 and beyond.

So, let's start off with a very simple scenario, and grow from there. I am guessing we all have at least one computer at home, maybe more. And in many cases that system may stay on 24x7, just in case you need to jump on the web for some reason in the middle of the night. That system is surely connected to the Internet, and all the good and bad that is lurking, all the time. Do you know what that system is doing after you go to sleep? Do you know if your system is one of the bots (robots) on a BotNet (group of robots) that is used to send ME spam at all hours of the night? Or, do you know if your system has a piece of malware that is capturing your keystrokes and sending them to a bad person somewhere in the world? Maybe it doesn't, but can you be sure? Yes you can, but only with a Protocol Analyzer. This is seeing what your 'expected behavior' is and knowing what is normal and what is not.

The term 'expected behavior' is a term that's meaning is slightly different depending on who you ask. People often think of expected behavior as always being good traffic, meaning that expected behavior of a network would be when everything is running perfectly. Not so. Expected behavior is just what is currently happening on the network, regardless of whether it is good or bad. Think of it as known vs. unknown traffic. In this case of protocol analysis, we need to have baselines (which is our expected behavior). If we are going to use the protocol analyzer as a tool to build our baselines, we have just overcome the first hurdle to good proactive network management. And, if we know our expected behavior, it is much easier to see that something has changed, and that change is usually what you are interested in when finding issues or oddities on your network, at home or at work.

So, let's get back to our scenario with your computer at your home. Do you have a baseline? Do you know your expected behavior? Unfortunately, probably not. But that is your home network, and may not have the importance to you of an enterprise network. That may be true, but always remember that most networks are connected to the Internet, and therefore, connected to one another. Any issue you have at home could certainly be causing problems for your enterprise network, especially if you VPN into the office from your home network.

Now, I do not want to add to the scare tactics about how dangerous the world is from a network perspective. We have many devices and technologies (Anti-virus, Firewalls, IDS/IPS, etc.) that are there to protect you, and generally they do. But, can you prove it? Yes, you can, once again with a protocol analyzer. When a bad person writes a Virus/Trojan/Worm, one of their top priorities is to try to make it so your security devices do not see or detect it. But, one thing they can't ever do is hide the packets. If they are going to do anything on the network, they will need to send those packets. And, with a protocol analyzer, you WILL see those packets; they cannot hide or be hidden. You may not know what exactly is inside those packets all the time (as it may be encrypted), but just the fact that the packets are there is the clue that you need.

Solution
So, how do we accomplish this in 2009? The vision here is simple, and it is called Infrastructure Analysis. Basically, you know your Infrastructure, and what it should be doing. So the idea here is to ignore everything you know, and focus on what you don't know. As an example, you know that mail traffic goes to your mail servers; DHCP Offers come only from your DHCP servers; DNS traffic to your DNS servers; exactly what nodes or networks connect to your financial systems; that your printers should generally only act as a server and never as a client; and on and on. So you take a few moments and write a filter that captures all that traffic. But wait, I said that we didn't want to see this known traffic. We don't, so all we need to do is 'negate' that filter telling the system to show me only what I don't know. What you end up with is a capture that shouldn't capture any packets (if you know your Infrastructure well!). Any time you see a packet, you know you have work to do, to either track down something or change your filter to reflect your new knowledge.
 
But now we need to automate this a little and with notifications, you can get notified any time something happens on the network that is outside the norm, therefore making it so you do not have to 'watch' the analyzer all day and night. And it should go without saying at this point, but these captures run 24x7x365, always giving you the vision of the unknown. The analyzer is no longer a tool, but much more of an application, running 24x7, analyzing your network, all its good and bad, and giving you reports so that you not only know what is happening today, but also have historical vision to what happened yesterday, last week, or last year.

Now I understand that knowing what is happening on your home network is a lot easier then knowing what is happening on an Enterprise network. But that doesn't preclude you from needing to do this. What it means is that you will need to have a better plan and move forward a little more logically. Rather than focusing on everything you don't know, filter out the unknown traffic and focus on the unknown about the known, if that makes sense. To clarify this a little, if the only thing you are sure of is the Mail traffic, start there. Filter on all mail traffic and ignore any traffic headed for your mail servers. What you end up with is any mail headed directly outbound, which could be at minimum a security risk or a great clue that someone may have a Trojan or Virus on their system. And then add one protocol or application at a time until such a time that you never capture another packet. That day will be the day you KNOW your network.

Enterprise Networks can add much complexity to this task, but once again, it is still a necessary to do it. As we discussed above, it may be a little slower, but with just a little time, it is easily achievable. And keep in mind, it doesn't ever have to be complete, just better than it was yesterday, last week or last month. Every personality and/or aspect of your network that you know and understand is another step in the right direction and substantially increases your chances to fix your next problem or issue in seconds, rather than hours. And in these tough times, quicker resolution equates to less downtime, which equates to less dollars lost. That is where true ROI can be seen.

Another hurdle to the Enterprise class network is possibly due to the scope and breadth of said network. Your environment may have a national or even global presence. And I am sure many of you are thinking to yourself, "No way. My network is way too big and too complex for anything like this to help me." And I am here to tell you; not true. With distributed analysis solutions, you can do Infrastructure Analysis anywhere in your organization. It doesn't matter if the traffic is in the room next door to you, or across the world. The methodologies are the same; the only change is your location of vision on the network. If you have vision (analyzers) in the right places, you will see not only what you know, but also learn what you don't know. And trust me, watching what you don't know is often much more important.

Benefits and Summary
With an OmniPeek network analyzer and OmniEngine software probes you can watch both local and distant traffic, know expected behavior of your Wired or Wireless LANs, use Alarms and Notifications to alert you to something out of the ordinary, use Triggers to automate the starting and stopping of captures, and build reports in a variety of formats to refer to historically. And if the historical reporting isn't enough detail, using OmniEngines will give you the capability to go back in time historically, do a forensic search and get just the packets that you are interested in, even if that interest revolves around just one bit in the packet. Now that is real power. And all of it can be done in seconds or minutes, not hours or days!

To summarize, let's remember a few of the key points that we discussed here.

1. Protocol Analyzers have be used 24x7 to know and understand your network.
2. With a Infrastructure Analysis capture running, you will immediately know when something happens on your network that is outside your understanding of what should be happening, giving you the proactive upper hand to resolve the issue or problem almost immediately.
3. With alarms and notifications you can be alerted to issues via a variety of ways, including Email, SNMP, syslog, etc. Again, this is now a proactive solution.
4. You can do this regardless of what size your network is, whether it is wired or wireless, how many nodes or applications run on it, and where those networks are located.
5. You can save thousands, tens of thousands or even more by applying these methodologies and techniques in your environment. In these tough times this is what is most important to us all.

Protospecs are written in XML and executed for each packet when the packet is either captured or loaded from a file. All of the Protospecs are in the pspecs.xml file. This file is shipped with the product and can be extended with new protocols by the user as well as other third parties.This a story about the marriage of Protospecs and Decoders. It is a love story that begins as a tragedy and ends in harmonial bliss.  The tragedy is that Protospecs and Decoders are separate subsystems that should be able to leverage each other, but until recently could not.  The harmonial bliss is a way to write Decoders as a C Plug-in that can use Protospecs.

The result of the protospecs is the name of the highest known protocol layer and a unique ID that represents it. Protospecs are very fast. Internally, protospecs are represented as a highly optimized binary tree. From the user point of view, the Protospec is the protocol name displayed in the Protocol Column of the Packet List as well as the names of the protocols in the Protocols Tab. The protospec id for a packet is also used by virtually every other feature of the program.   This is important, because it means that modifications and extensions made to protospecs are highly leveraged.

Decoders are written in a specialized language that is appropriately called The Decoder Language. It is a hybrid language, a melting pot if you will, consisting of legacy syntax resembling 68000 assembler as well as newer syntax that looks a lot like C++. Decoders are found in the decodes folder and are separated into files called .dcd files. When OmniPeek is run, all of the dcd files in the decode directory are loaded and compiled into binary form. Decoders are fast, but they are still interpreted and thus are slower than machine code.

From the user point of view, the decoder is the detailed break down of a packet in the Decode View of the Packets Tab as well as the Decode Column of the Packet List.  The two technologies, Protospecs and Decoders, provide similar but different functions and thus many moons ago were implemented as completely different libraries without any connection or integration between the two. And that, my friends, is where our story of a tale of two islands begins.

For millennia, in fly years anyway, man has searched for magic to use the great knowledge of the protospecs from the decoders.  It is believed by many wise men that if one knows the protospec of a packet while in a decoder, less work has to be exerted by the decoder programmer to write the decode. And of course doing less work is the holy grail of the programmer, making this an honorable and worthwhile journey. However, this quest has been made many times and has always ended in frustration, futility, and ultimately failure. Years have passed since the journey has been made.

But today we enter a new era, a Decoder Renaissance. Today, thanks to a premium developer support customer, a phone call for developer support was made and in the process a realization formed that there is a place in the universe where decoders and protospecs can meet. This place is the C Decoder Plugin where the decoder programmer has access to both the decoder functions and the packet, which includes the almighty powerful protospec id. It is in this special place, the "Protocoder", that the protospec id can be used to make important decisions about the decode without having to rewrite logic that the protospec library has already executed.

As the story goes, the customer had extended the pspecs.xml file to include a number of new application layer protocols. Their protocol types are figured out by looking at the IP address, not the protocol type. This is odd, but true, and they have their reasons. So now that they had worked so hard on the protospecs, it was time to write the decoder.

They were tired, for they had labored long and hard, and when they begun writing the decoder they did not want to work so hard again. They also, for other reasons, had decided to write their decoder as a C Decoder Plugin. And although it was not evident early on what the advantages would be, it was realized in a moment of great insight that by writing the decoder in C, the protospec along with other valuable information about the packet is passed into the C decoder function.

So believe it or not, that's how the story goes. And from that moment on it was realized that this special place in a plug-in where Protospecs and Decoders meet, could be used to enhance decoders in many other ways. Who knows, maybe those stories will be passed down through the ages as well.

The End.

*The feature described in this story is available in OmniPeek 4.0.2 and above.

For more information about Protospecs and Decoders, please visit the following URL:

https://mypeek.wildpackets.com/documentation/index.php

Chris Bloom

  Customer Solutions Expert

  WildPackets, Inc. 

  925-274-5443

Traditional telephone services have typically gained a reputation of providing excellent voice quality and superior reliability. Consequently, users take for granted that their phone systems will provide high quality with virtually no downtime. Yet many VoIP installations fail to met these expectations primarily because organizations have not adequately evaluated their network infrastructure to determine whether it can adequately support applications that are very sensitive to latency, packet loss, and jitter. These three "monsters" that may remain unnoticed in a data network will become very obvious and ugly in VoIP systems.

If you're interested in improving your current (or future) IP telephony experience, check out Jay Botelho's (Director of Product Management at Wildpackets) slide presentation that focuses on best practices to troubleshooting converged networks at: http://www.slideshare.net/wildpackets/improving-the-ip-telephony-experience.

In the presentation, viewers will learn about how to test their network(s) before installing a VoIP system to identify and correct performance troubles as well as proactive troubleshooting techniques and using qualitative and quantitative analysis. Learn about ...

• The importance of Quality of Service (QoS) parameters to enable network devices to prioritize and give preference to packet streams that are sensitive to delay, packet loss, jitter, and other performance inhibitors.

• Network traffic pattern changes to look for that adversely affect VoIP

• The importance of providing alerts when VoIP performance declines

• Proactive troubleshooting tricks and tips that lead to a reliable VoIP implementation with first-rate voice quality.

If you are responsible for determining the health of your network before deploying VoIP, need to rid the network of VoIP-killing troubles before installation, or want to be equipped to assess the network when VoIP-killing troubles arise, you'll want to digest this slide presentation.

We are excited that Network World's recent review of WLAN tools awarded WildPackets'
OmniPeek a perfect score of 5.0!

In total, six products were reviewed in a WLAN "bake-off" of sorts. OmniPeek received the highest scores possible in each category - features, ease of use, documentation, and installation.

In the review, Craig Mathias, principal at Farpoint Group, notes -- "OmniPeek's flexibility is first-rate... The product also enables a high degree of customization, including the ability to extend analysis with custom code (for specialized protocols). Complete filtering is also provided, allowing a user to focus only on particular packets or protocols ... Overall, this product was by far the easiest to use. We had to turn to the manual only to rate the documentation's quality, which was also excellent."

We couldn't agree more!

To view the entire review, check out http://www.networkworld.com/reviews/2009/011909/2009/011909-wlan-test-wildpackets.html

Synergy: The working together of two or more things to produce an effect greater than the sum of their individual effects. 

So says the dictionary, but more to the point, synergy is what happens when you use multiple OmniPeek plug-ins together.   The effect is of course not limited to plug-ins, synergy is also found in the use of the many tightly integrated features like summary stats and graphs.   But plug-ins greatly increase the effect of built-in features as well increase the effect of each other.

For example, let's say that you want to capture the same packet from multiple locations on different segments of the network and compare certain things about them like the delta times, window sizes, and hop counts.   By using the Remote TCPDump Adapter Plug-in and the PeekPlayer Plug-in you can capture from multiple remote sources and aggregate all of them into a single capture window, in real-time.   Along the way, you may have used a C Decoder Plug-in, the WebStats Plug-in, or any number of other plug-ins that customize the workflow to the specific needs of your business.

Another example is using multiple RFGrabbers, where each one is capturing packets from a different channel, and using the PeekPlayer Plug-in to redirect all the packets into a single capture window.   In this way you can see the Signal and Channel graphs for all of the channels being monitored in a single capture window.   If you have not done this yet, give it a try.   The PeekPlayer makes it possible to redirect packets from one capture window to another.

Finally, take the first example and add the SQLFilter to the aggregating capture window so that as the packets are aggregated they are indexed into a single database for post capture data mining and forensics.  In this case, we are using OmniPeek, the Remote TCPDump Adapter, the PeekPlayer, and the SQLFilter.    You might also be using an expert logging plug-in, or any number of other plug-ins for filtering and processing of the packets in the final capture window.

One of the advantages in these scenarios is the aggregation of multiple streams into a single capture window in real-time.   This makes it easier to analyze the data and saves the user the time it would have taken to manually aggregate the packets through PeekCat.   But the other advantage is being able to do most of the configuration on the aggregated capture window instead of to all of them.  Again, this improves the workflow and as a result save the user time and money.

This is really all about improving workflow.   And since workflows are different from business to business and network to network, the ability to customize OmniPeek sets it apart from other products that are not quite so configurable.

 

Introduction Just as "Internet Search" was revolutionized by Google, so will "Packet Search" be revolutionized by the WildPackets SQLFilter Plugin.  While Google indexes content from websites, the SQLFilter indexes packet traffic to and from those websites. As more companies save large quantities of network traffic to disk, tools like the SQLFilter make it possible to search through packet data more efficiently. For network engineers, this significantly decreases the amount of time it takes to find the packets in question. Not only does the SQLFilter allow users to search for packets across 1000's of trace files using a simple UI, as well as arbitrarily complex SQL, it also loads the resulting packets directly into OmniPeek. This cuts out many of the steps usually involved in this process and dramatically shortens the period of time it takes to find packets. The free version of the SQLFilter can be downloaded from the MyPeek Community Portal. Although it is easy to use (says me, the guy who developed it ;-)), you should download and read the documentation. It is short, but that way you will be able to make the most of the functionality offered by the plug-in.     For more advanced and scalable packet data mining and network forensics, WildPackets offers a support package through its' Professional Services Team.   This article discusses the features and components included in this support package as well as the different configurations made possible through the support of a client/server database system.

At the core of this solution are WildPackets award winning products, the OmniPeek Console and the Omni Engine.  Built as an application on top of this rich, extensible platform are a variety of database tools.  One of these is the SQLFilter, a plugin  for searching through large numbers of trace files using arbitrarily complex SQL queries. The free version of the SQLFilter, which uses sqlite as the database back-end is extremely easy to install and use.   The UI is intuitive and the use of sqlite as the database allows for powerful and fast queries in a package that requires no database setup or maintenance.

The SQLFilter has also been enhanced to support the MySQL database.   This version is known as the "MySQLFilter".     This high-end versions of the SQLFilter has significant advantages over the sqlite version.  The MySQL version of the SQLFilter is also available for free to maintenance users.   SQLFilter All versions of the SQLFilter bring simple and extensible data mining to WildPackets network analyzers. The plugin lets you perform Structured Query Language (SQL) searches across very large, user-defined sets of packet files from within OmniPeek and EtherPeek. It also allows you to perform SQL, regular expression, or text string searches of the Packets view. Results of a query are packets displayed directly in the Packets view of the local Capture window.  These packets are processed by all of the available facilities like Node Stats, Protocol Stats, the Expert, and the Peermap.  The packets are also processed by plugins, including ones that you can write yourself.   The SQLFilter adds a new Tab to a Capture Window which is used to manage databases that the SQLFilter is aware of.

With the SQLFilter, SQL Queries are made through the use of a query line in the Capture Window where arbitrarily complex SQL can be entered.   The query line also knows about numerous fields like IP Address, Ethernet Address, Ports, and Dates.   These types can be entered into the query line and the SQLFilter Plugin will generate the correct SQL.   A Search Dialog is also provided which can be used to enter query parameters. For specific instructions on how to use the WildPackets SQLFilter Plugin, please refer to the  SQLFilter PDF Manual at the bottom of the SQLFilter Page

MySQL Of course, the manual does not reflect the newly added support being developed for MySQL.   With the MySQL support in the SQLFilter Plugin, and the ability to store the packets themselves in the MySQL database, numerous new features, more configurations, greater scalability, and better manageability become possible.    With MySQL, a single database can contain as much as 64 terabytes of data.   More specifications about MySQL can be found at http://www.mysql.com.

Because of the client/server nature of both MySQL and Omni, different configurations of the system are possible, and different choices can be made about where the data is and how much data there is in each database.   For example, in the simplest configuration, the Database and the OmniPeek Console can reside on the same machine.   This solution can be achieved with both the sqlite version of the SQLFilter, as well as the MySQL version.    The sqlite version is not client/server and each database is a single file.   In the MySQL version, the database is accessed through a MySQL service which can be local or remote on Windows or any other platform (eg Linux) that supports MySQL.

Remote Access With the MySQL version of the SQLFilter, whole packets can be added to a database that can be queried by other users.   This is because once the packets are in the database, they can be retrieved through an SQL query.  This also makes it possible to perform string searches on the packet contents, not just the headers.  Once the packets are in the database, multiple users can make queries against the same database simultaneously by using the SQLFilter Plugin.  Web Browser Access Also available from MyPeek is software that makes it possible to easily query packet databases through a web browser.   Shown below are screenshots of the SQLFilter Web Front-end.   After supplying a query, the web front-end can display a number of different views including Summary, Packets, Nodes, Pairs, Ports, and Files.   In Nodes, Pairs, and Ports views a row can be selected to display the packets that represent that item.  Finally, any packet can be selected, resulting in the display of the decode and hex for that packet.   The SQLFilter web decode uses the same decoder library and decoders used by Omni.  Like in Omni, this means you can write your own decodes.  The added benefit here is that others can see your decode through the web without having to download any dcd files. Architecture The SQLFilter Web Front-end software is made of standard components like a web server (eg Apache),  html, and PHP.   The result is a very flexible and extensible multi-tier system.   The diagram below shows some of the components that make up the architecture.