Network Forensics

Introduction
Just as "Internet Search" was revolutionized by Google, so will "Packet Search" be revolutionized by the WildPackets SQLFilter Plugin.  While Google indexes content from websites, the SQLFilter indexes packet traffic to and from those websites. As more companies save large quantities of network traffic to disk, tools like the SQLFilter make it possible to search through packet data more efficiently. For network engineers, this significantly decreases the amount of time it takes to find the packets in question. Not only does the SQLFilter allow users to search for packets across 1000's of trace files using a simple UI, as well as arbitrarily complex SQL, it also loads the resulting packets directly into OmniPeek. This cuts out many of the steps usually involved in this process and dramatically shortens the period of time it takes to find packets.

The free version of the SQLFilter can be downloaded from the MyPeek Community Portal. Although it is easy to use (says me, the guy who developed it ;-)), you should download and read the documentation. It is short, but that way you will be able to make the most of the functionality offered by the plug-in.     For more advanced and scalable packet data mining and network forensics, WildPackets offers a support package through its' Professional Services Team.   This article discusses the features and components included in this support package as well as the different configurations made possible through the support of a client/server database system. 

At the core of this solution are WildPackets award winning products, the OmniPeek Console and the Omni Engine.  Built as an application on top of this rich, extensible platform are a variety of database tools.  One of these is the SQLFilter, a plugin  for searching through large numbers of trace files using arbitrarily complex SQL queries. The free version of the SQLFilter, which uses sqlite as the database back-end is extremely easy to install and use.   The UI is intuitive and the use of sqlite as the database allows for powerful and fast queries in a package that requires no database setup or maintenance.

The SQLFilter has also been enhanced to support the MySQL database.   This version is known as the "MySQLFilter".     This high-end versions of the SQLFilter has significant advantages over the sqlite version.  The MySQL version of the SQLFilter is also available for free to maintenance users.  

SQLFilter
All versions of the SQLFilter bring simple and extensible data mining to WildPackets network analyzers. The plugin lets you perform Structured Query Language (SQL) searches across very large, user-defined sets of packet files from within OmniPeek and EtherPeek. It also allows you to perform SQL, regular expression, or text string searches of the Packets view. Results of a query are packets displayed directly in the Packets view of the local Capture window.  These packets are processed by all of the available facilities like Node Stats, Protocol Stats, the Expert, and the Peermap.  The packets are also processed by plugins, including ones that you can write yourself.   The SQLFilter adds a new Tab to a Capture Window which is used to manage databases that the SQLFilter is aware of.

With the SQLFilter, SQL Queries are made through the use of a query line in the Capture Window where arbitrarily complex SQL can be entered.   The query line also knows about numerous fields like IP Address, Ethernet Address, Ports, and Dates.   These types can be entered into the query line and the SQLFilter Plugin will generate the correct SQL.   A Search Dialog is also provided which can be used to enter query parameters.

For specific instructions on how to use the WildPackets SQLFilter Plugin, please refer to the  SQLFilter PDF Manual at the bottom of the SQLFilter Page

MySQL
Of course, the manual does not reflect the newly added support being developed for MySQL.   With the MySQL support in the SQLFilter Plugin, and the ability to store the packets themselves in the MySQL database, numerous new features, more configurations, greater scalability, and better manageability become possible.    With MySQL, a single database can contain as much as 64 terabytes of data.   More specifications about MySQL can be found at http://www.mysql.com.

Because of the client/server nature of both MySQL and Omni, different configurations of the system are possible, and different choices can be made about where the data is and how much data there is in each database.   For example, in the simplest configuration, the Database and the OmniPeek Console can reside on the same machine.   This solution can be achieved with both the sqlite version of the SQLFilter, as well as the MySQL version.    The sqlite version is not client/server and each database is a single file.   In the MySQL version, the database is accessed through a MySQL service which can be local or remote on Windows or any other platform (eg Linux) that supports MySQL.

Remote Access
With the MySQL version of the SQLFilter, whole packets can be added to a database that can be queried by other users.   This is because once the packets are in the database, they can be retrieved through an SQL query.  This also makes it possible to perform string searches on the packet contents, not just the headers. 

Once the packets are in the database, multiple users can make queries against the same database simultaneously by using the SQLFilter Plugin. 

Web Browser Access
Also available from MyPeek is software that makes it possible to easily query packet databases through a web browser.   Shown below are screenshots of the SQLFilter Web Front-end.   After supplying a query, the web front-end can display a number of different views including Summary, Packets, Nodes, Pairs, Ports, and Files.   In Nodes, Pairs, and Ports views a row can be selected to display the packets that represent that item.  Finally, any packet can be selected, resulting in the display of the decode and hex for that packet.   The SQLFilter web decode uses the same decoder library and decoders used by Omni.  Like in Omni, this means you can write your own decodes.  The added benefit here is that others can see your decode through the web without having to download any dcd files.

Architecture
The SQLFilter Web Front-end software is made of standard components like a web server (eg Apache),  html, and PHP.   The result is a very flexible and extensible multi-tier system.   The diagram below shows some of the components that make up the architecture.