pointer

Monthly Archives: February 2010

Introducing Compass: Using Utilization Statistics for Network Forensics

Posted on by .

    The holy grail of effective network troubleshooting is the ability to pinpoint issues quickly so that they can be fixed. Any approaches to better optimize this particular network analytics process mean more uptime and healthy networks over the long run.

    Here’s a suggestion – instead of loading all packets, shave off time by using utilization statistics about network traffic to provide clues that answer questions like “What happened?” “When?” “Who did it?” Only then determine what slice of time you want to perform deeper network analysis on.

    To this end, WildPackets is releasing Compass, a freely available interactive forensics dashboard for the OmniPeek Network Analyzer. Compass’ dashboard graph (see screenshot) allows users to select specific time periods for analysis, add and remove nodes and protocols to the same graph, and compare and correlate these for different periods of time, over long periods of time.

    In some cases, seeing the utilization in the Compass graph for the nodes and/or protocols in question may solve the problem. Otherwise, once a slice of time has been selected, the packets for just that slice of time can be loaded into OmniPeek by hitting the “Load Packets” button.  If that slice wasn’t the problem, just go back to the graph, slide the time window, and load a different slice of packets.


    compass_full3.png

      Three tips for determining whether latency is caused by the network or application

      Posted on by .

        Networks typically run various applications, from single-tier, locally-hosted applications like e-mail, to multi-tier web-based applications, or even time-sensitive, multi-hop applications like VoIP. While application traffic typically resides within the data center, SaaS and cloud computing are rapidly growing trends that are driving application traffic outside of the traditional enterprise network, making network latency even more of an issue. Pinpointing and correcting slowdowns is therefore a necessity, and can be a real challenge. So what is responsible for the latency creating poor application performance- the network or the application itself?

        First, we must distinguish between the two basic types of latency – network and application latency. Network latency is the amount of time it takes for an application to make a request and the server to respond with an acknowledgement (a packet message used in the transmission control protocol to acknowledge receipt of a packet). Application latency is the amount of time needed for the application to process the request and send a response containing real data.

        Most network-monitoring products provide some sort of latency-monitoring features, typically either one or the other, not both. Here are three tips to determine whether latency is be caused by the network or the applications (or both) in your environment.

        1. Clearly determine network versus application latency
        Every application issue is blamed on the network until proven otherwise – “guilty until proven innocent.” Clearly measuring network latency vs. application latency is the proof the network engineer needs for acquittal. Packet-level monitoring is ideal for accumulating evidence. By visually inspecting a packet-level conversation between a client and a poor performing application, one can see whether the network (or a network device) is the source of the delay, or if the application is the bottleneck. This is done by comparing the responsiveness of the TCP ACK to a client request versus the application response, which includes actionable payload data. Quite often the network acknowledges the client request quickly (within milliseconds) while the application may take tenths of seconds or even seconds to respond with payload data. When you see this, you know it’s the application causing the problem.

        2. Periodically test and monitor key applications and network connections
        Periodic, active monitoring can also provide insight into network performance on key interfaces, and can alert you when conditions begin to degrade. While this technique only addresses network latency (not application latency), it can still provide important data when determining whether the issue is network or application related. For example, let’s say the organization’s CRM application is via a web-hosted service. Running periodic traffic in the background (even just pings) to the CRM application host can provide an ongoing baseline of the performance of the network between users and the host. If the baseline increases, alarms are used to notify that the network latency is becoming an issue. User complaints about CRM performance without a marked change in the network latency baseline almost always indicate the application host or the application itself is at fault.

        3. Graphing latency over time
        Graphing latency over time helps to identify patterns and anomalies that deserve closer attention. Latency monitoring can help correlate areas of latency with other relevant statistics, as well as the actual network traffic occurring at that time. This type of high-resolution forensic analysis can help to detect latency problems at the highest level and drill down quickly for closer inspection.

        Ideally, network latency and application latency measurements can be graphed together over time, making clear whether the problem lies with the network or the application. Comparing the measurements of the two types of latency over time and seeing the differences can provide information that might have otherwise been overlooked.

        Latency monitors can include a feature that sets thresholds on latency, so alarms will go off when normal conditions are exceeded. The administrator can be made aware of excessive latency before application performance becomes a widespread issue, allowing him or her to make necessary adjustments to the network proactively. This type of proactive latency monitoring allows the administrator to detect and correct problems in the network and applications before users even notice a slowdown.

          10 Cool Things You Can Do With OmniPeek

          Posted on by .

            By Joe Habib, WildPackets Director of Global Services

            OmniPeek is the first protocol analyzer to offer both expert diagnostics and frame decoding in real time, during capture. WildPackets’ OmniPeek has been carefully designed to help IT Professionals analyze and diagnose increasingly diverse volumes of network data, providing precise, contemporary analysis of the problems facing today’s networks.

            Here are ten cool things you can do today with OmniPeek!

            10_cool_things_OmniPeek-1.png

            Main window of OmniPeek, showing the Expert view


            1. Capture and view packets

            Though you can see global network statistics without capturing packets, for most analysis sessions you’ll want to capture packets. One of OmniPeek’s many strengths is in its flexibility. This is immediately apparent in the packet list display, where you can easily customize the display.

            10_cool_things_OmniPeek-2.png

            1. Click the New Capture button on the Start Page, or pull down File…New from the menu bar. Click OK in the Capture Options Dialog, then click the Start Capture button.

            2. OmniPeek provides several ways to customize your view of traffic with a single click.

            a. The auto scroll option (Ctrl-K) allows you to see packets as they come in, which is useful if you’re looking for particular information in the Summary or Expert columns.

            b. The Show Packet List, Show Decode View, and Show Hex View buttons allow you to pick the content you want to see in real-time. Most other analyzers show all three (and not usually in real time) and do not allow you to toggle between the different views.

            c. Right-click on one of the columns in the packet list to customize the column display.

            10_cool_things_OmniPeek-3.png

            2. Capture filtering the easy way

            OmniPeek ships with a set of common, pre-defined filters, but its real power is in the ease with which you can create your own filters.

            1. As packets are coming in, choose something you might want to filter on, such as a protocol or source IP address. Right click on that packet (stop the scrolling, if necessary, by keying Ctrl-K).

            10_cool_things_OmniPeek-4.png

            2. Click on Make Filter – notice that the information on the packet you chose is already set up in the Edit Filter dialog for you.

            10_cool_things_OmniPeek-5.png

            3. Name the filter and make modifications. You may wish to restrict the data to only one attribute, for example, source IP address or protocol.

            4. Click OK to save the filter.

            5. Click on the Filters tab and find your new filter in the list. Check the box next to the filter. You will now be capturing only packets that meet the criteria of that filter. (Alternatively, you can reject only those packets matching the filter criteria by clicking on the Reject Matching button.)

            10_cool_things_OmniPeek-6.png

            6. Go back to the packet list in the Packets view. You should only be seeing packets that meet your filter criteria. Notice the packet counts differ in the upper left corner (Packets Received vs. Packets Filtered)

            3. Advanced filtering

            OmniPeek’s advanced filtering is easier to use than most analyzers’ simple filtering. Filtering on specific protocol decode fields, for example, can be accomplished in just a couple of mouse clicks. Suppose you wanted to view all packets with a Time To Live (TTL) of under 128. A packet with a TTL of less than 128 indicates the packet has most likely traversed a router. When dealing with network slowdowns, it’s interesting to understand where the packet came from and where it’s going. TTL helps us understand the packet’s path. Here’s how to build the filter:

            1. Click on the Packet List Tab.

            2. Choose any IP packet and double-click to open the decode view.

            3. Right click on the TTL field under the IP Header section and select k.

            4. An advanced filter is already made for you!

            5. Double click on the Value box.

            6. Change the Operator to ‘<’ (other LAN analyzers do not have this capability).

            10_cool_things_OmniPeek-7.png

            7. Name the filter and click OK.

            8. Go to the Filters Tab and click on the filter you just created. Packets will then be filtered on the fly!

            9. The same filter can be used for post capture analysis, too. OmniPeek doesn’t force you to define filters in multiple places.

            4. Who are the Top Talkers?

            Top Talkers is a common troubleshooting statistic. The network is slow, for example, and you may want to see which stations are using the most bandwidth.

            1. Click on the Nodes Tab. By default, the View Type is Hierarchical, where logical addresses and symbolic names are nested beneath their physical addresses along with their transmit and receive statistics. However, you can easily change the default view to one showing the Top Talkers.

            2. Pull down IP from the View Type, then click on a column to sort on Bytes or Packets sent or received. Right click on the columns to customize the display.

            3. To view a subset of the talkers, choose a value in the Display Top drop down.

            10_cool_things_OmniPeek-8.png

            4. Note that you can see the top 5 or 10 or 100, etc. Double click on a host to view its protocols in Detail Statistics.

            10_cool_things_OmniPeek-9.png

            Make a filter, build an alarm, or construct a graph for any of these hosts with just a click of the mouse. OmniPeek provides you with an amazing amount of flexibility. Other analyzers decide for you what statistics will be displayed.

            5. What protocols are on your network?

            Perhaps, instead of wanting to know what users are using the most bandwidth, you want to know what applications are using up bandwidth. Are any protocol ratios too  high? Are there any protocols that shouldn’t be on the wire?

            1. To see which applications are on your network, click on the Protocols Tab.


            10_cool_things_OmniPeek-10.png

            2. Double click on a protocol to see the % of usage by each host.

            10_cool_things_OmniPeek-11.png

            3. Return to the Protocols Tab, right click on a protocol, and Select Related Packets.  This is a two-click method of choosing all the packets in the packet list that are talking this protocol. Select Related is available throughout the program: the nodes stats, expert problems, Peer Map, etc.

            6. Make multiple graphs

            OmniPeek’s extensive graphing ability enables you to correlate useful statistics. For example, are broadcast packets a significant portion of your utilization?

            1. Click on the Summary Tab.

            2. Right click on Total Broadcast, and choose Graph… Name the graph in the Graph Data Options dialog and click OK.

            10_cool_things_OmniPeek-12.png

            3. Right click on Average Utilization and choose Graph… Name the graph in the Graph

            Data Options dialog and click OK.

            4. Minimize the capture window and then choose Window…Tile Horizontally. (OmniPeek is one of the few analyzers that complies with MS Windows conventions, which is certainly helpful here!)

            10_cool_things_OmniPeek-13.png

            7. Find problem packets through “Select Related”

            OmniPeek Expert Tab is by far the easiest to use and most up-to-date on the market. Problems are arranged by conversation, rather than by OSI model level. ProblemFinder tests and settings are just one right click away, as are problem descriptions and possible remedies. Other analyzers force you to hunt and peck for the information you need. OmniPeek delivers this information to you automatically. It pinpoints the packets related to a network communications issue, tells you why it’s probably happening, and suggests ways to fix the problem.

            1. Click on the Expert Tab.

            2. Right click on the first host, and choose Expand All.

            10_cool_things_OmniPeek-14.png

            3. Scroll down until you find a particular problem you’d like to look at.

            4. Right click and choose Select Related Packets by Source and Destination or by

            Conversation.

            5. Alternatively, you can go straight to the Problem Summary Log and select all packets  related to a particular problem. Note how the conversations having this issue are  highlighted when you return to the Packets view.


            8. Determine Application Response Time

            Application Response Time is available via the Expert Tab, where detailed round trip analysis of command-response packets is available, showing you the
            best, worst, and average delay. In a similar fashion, throughput is also analyzed.

            1. Click on the Expert Tab

            2. Click on the Node Details tab.

            10_cool_things_OmniPeek-15.png

            3. Right click on the first host in the conversation tree, and choose Expand All.

            4. Walk down the tree until you see an interesting analysis in the Delay and Throughput

            Analysis display in the lower right.

            9. Visualize your network with expert mapping

            The Peer Map is a great way to get a visual perspective of your network. Not only can you select related packets from hosts on the map, but you can easily create ad hoc filters or look at Top Talkers.

            1. Click on the Peer Map.

            10_cool_things_OmniPeek-16.png

            2. Choose IP Map from the Map Type pull down in the upper right hand corner.
            3. In the Node Visibility Criteria area just below, you can choose Top Talkers via absolute number or, say, top The amount of traffic through a node is represented by the size of a dot. If there is one host, such as a mail server or a web server that is skewing the results you are looking for, you can drag that node into the

            10. Find that slow web server fast

            With OmniPeeks’ Expert System, you can easily spot slow servers. Here’s an example of how to troubleshoot a slow web server.
            1.    Start a new capture.
            2.    OmniPeek ships with many standard filters, including HTTP. Click on the Filters Tab
            and check the HTTP filter to immediately activate it.
            3.    Go back to the Packets view. Enable Scrolling (Ctrl-K) so you can see incoming packets. Verify that they are HTTP packets.

            10_cool_things_OmniPeek-17.png

            4.  You may see HTTP access from web traffic not associated with your web server, so you will need to add a new filter. Open a browser and go to your web server. You should see your web server in the packet list display. Create a filter by right-clicking on that packet and choosing Make Filter.

            5. Go to the Filters Tab and enable the filter you just created.

            6. Now go to the Expert Tab. Click the Problem Summary pane and check for TCP Resets or HTTP Slow Response time diagnoses.

            10_cool_things_OmniPeek-18.png

            7. The Problem Log pane provides more detail, including actual time delay between specific packets (e.g. 13 seconds from Packet 908).

            8. Right click and select related packets by See Packet.


            10_cool_things_OmniPeek-19.png

            9.    Go to the Packets view and your problem packet is highlighted!

            10. From there, try and figure out what sort of packet it is. Does it say Data in the Summary Column? What is this packet in response to? Click through the packets that preceded the bad packet.

            Unlike other analyzers, OmniPeek’s expert system ignores the ACKs when determining the HTTP Slow Response diagnosis. Look to see if the ACK was received right away. If so, and it was the data packet that triggered the diagnosis, you know it was the web server that was slow and not the network.

            ———————

            Joe Habib

            Director of Global Services

            I like to follow the KISS principal (KEEP IT SIMPLE STUPID) and welcome problems as they provide me with daily challenges.