Category Archives: 100g

Where to Capture Packets in High-Speed and Data Center Networks

Network analysis changes dramatically as network speeds grow (10G, 40G, and up to 100G). From more packets to capture, to changing traffic patterns like East-West traffic among servers, network analysis strategies must adapt as new technologies are introduced. We’ve written in the past about best practices for network monitoring on high-speed networks. However, we have never gone into detail on how to position capture points to see inside areas that might be neglected by previous monitoring methods.

Below we go into where you should set up your capture points to get the most visibility on your high-speed network.

Capturing Data on the Network
Typically, if you are connecting directly on the network you are going to collect data through traditional SPAN ports, mirror ports, or taps. This is a well known method that is used frequently to obtain a passive feed from a network, so the process and any associated configuration should be familiar. One challenge does arise when virtualization is in use, as you will miss intra-host traffic when capturing only on the physical network. Don’t worry, below we will explain how you can capture this traffic as well.

In this video, you will see where to capture traffic if you are in the data center, a corporate campus, or remote office.

Capturing Data on vSwitch
Data on virtual servers pose a unique challenge, as oftentimes much of the data never leaves the virtual server – for example, communication between and application and a database running on the same virtual machine. In this case, capturing data off the span port of the virtual switch or hypervisor allows you to get visibility into intra-host traffic. To do so, you either need to have network analysis software running directly on the server, or you need a “virtual tap” (a piece of software) that can perform the function of a traditional hardware tap and copy network traffic off to a separate physical tap which can then be utilized in a traditional fashion. If you’re running the network analysis software directly on the local VM, remember to allocate enough memory, IO, and disk space to accommodate your network analysis needs.

Capturing Packets in the Cloud
Cloud computing comes in many shapes and forms. If you are trying to capture data in a private cloud, the practice and procedure will be similar to that of capturing on your vSwitch. If you control the infrastructure, you can sniff anywhere. If you are a service provider, you need to carefully consider data access, data separation, and customer privacy issues.

If you are using a third-party cloud service, the ability to capture and monitor traffic is going to depend on the implementation. If you are running software-as-a-service (SaaS) from a provider, it will be hard to have sniffing rights, so your last point of knowledge about your traffic will be at WAN link. This will still allow you to obtain valuable analytics, like round trip latency, which will provide a good indication of the overall user experience. However, if users are experiencing latency and you think that it might be an application performance problem and not an overall network problem, then it will be difficult to analyze the situation. For example, a database connection issue or database contention may be very difficult to troubleshoot. But then again, isn’t that why you’re paying your SaaS provider?

If you are employing infrastructure-as-a-service then you will have the ability to sniff your own traffic by installing a network analysis software probe on the hosted virtual server to see all the traffic on the virtual  server, thereby restoring your ability to analyze application issues that may otherwise be hidden.

If you are working within another environment and would like tips on capturing data, please leave us a comment.

Best Practices for Managing Colossal Networks

40G is more than just a bigger pipe; it introduces significant new challenges in monitoring and analyzing data traversing the network. You can no longer employ the “break/fix” or “point and shoot” troubleshooting techniques used in the past after problems have already been reported. These high-speed networks require proactive, ongoing network monitoring and analysis to keep them performing as designed. And of course your tools must evolve just as rapidly as your network, which is certainly not always the case.

Monitoring and analysis of 40G networks requires updated tools as well as new strategies and approaches. Let’s take a look at some of the key, though perhaps not so new, strategies that must be employed when considering how to monitor, analyze, and troubleshoot a 40G network.

Capturing All of the Data – All of the Time
Performing on-the-fly analysis or trying to recreate problems that you missed the first time around is no longer feasible on high-speed networks. It is essential to capture network data 24×7, and store as much detailed data, down to the packet level, that you can. By doing so, you have a recording of everything that happened on the network, and you can rewind the data at any time to analyze a specific period of time, usage of a specific application, activity on a particular subnet, or even the details of a specific network flow. To do this effectively, we suggest purchasing a purpose-built network forensics solution, one that is specifically designed for high-speed networks, and that also includes a rich set of real-time statistics. This will help keep all of your data into a single repository for easy post-capture analysis.

Your network forensics solution may not be the only appliance that needs access to the 40G network stream. One way to simplify the collection of 40G network data for detailed analysis is by using an aggregation tap instead of connecting an appliance directly to the 40G network via a dedicated tap. This will provide significant flexibility when dealing with the 40G stream. You can just replicate the 40G feed to multiple network tools, or you can use the built-in filtering to send subsets of the traffic to different network tools, depending on your data analysis needs.

Storage capacity is a primary concern when performing network recording. Let’s say your average usage on your 40G link is 25%, or 10Gbps. At this data rate, assuming a network recording appliance with 32TB of storage, you can record 7 hours of network data. An aggregation tap can also help here, allowing you to split the data stream among multiple network recorders to achieve higher overall storage rates. Another option is to connect your network recorder to a SAN for additional data storage.

Understanding What is Normal
Knowing how you expect your network to be perform is all the more critical when trying to analyze colossal networks. In advance of an investigation, you’ll want to establish clear base lines of your network. If you’re already embroiled in a complex network analysis firefight it is too late to realize that your ability to assess “normal” conditions on the network may be lacking.

Analyzing the Essentials
When faced with an issue on your network, you’ll want to first analyze the essentials. The temptation is to try to capture and analyze everything, especially when the source of the problem is not immediately known. You do, however, know certain things about your network, which allows you to be selective in the analysis options you choose. Often a variety of conditions can be immediately ruled out, and using these clues to limit the collection and analysis to only what is necessary dramatically improves network analysis performance. For example, if you’re looking at a 40G network link, you’re probably not capturing wireless traffic, so you can turn off the wireless analysis. Turning off analyses that aren’t relevant to your investigation refines your search, making it more specific, and increases the processing power and throughput of the appliance you’re using.

Knowing the Limits
Even after analysis has been streamlined to only essential areas of the network, data capture for network analysis on 40G networks generates a great deal of data quickly, and managing the data becomes a significant challenge. Effective analysis requires that you know the limits of your tools, not just the available space for storage, but the processing limits of your appliance as well as how many users can access the appliance concurrently and perform analysis.

Moving from 1 to 10 to 40G introduces new challenges that are still being worked out in the industry, especially when it comes to support for network monitoring, analysis, troubleshooting, and security tools.

If you are in the midst of an upgrade or are thinking about upgrading to 40G, be sure to include the correct tools in the upgrade plan and budget, including solutions for establishing network baselines, capturing and storing the data 24×7, and performing network forensics as needed. It’s easy to continue to treat these networks like 1G, but they’re vastly different and require a new strategies for analysis.

Distributed Networks: Best Practices for Selecting Your Analysis Options

Distributed networks, which include pretty much any corporate network today, require distributed analysis, the collection of network data across multiple key points in the network, 24/7. This is in stark contrast with the portable analysis approach where you capture data only after a problem has been reported, and do so by moving around to different points in the network with a laptop or other mobile device running network analysis software. With today’s high speed and highly distributed networks, this approach is often too little too late, though it does remain a viable option in some smaller wireless LAN (WLAN) infrastructures.

Nobody’s network is the same, and the topology depends on many factors, but most networks have similar characteristics which can be used to help you plan for a holistic solution that will best monitor and analyze your entire distributed environment. Below are several common network characteristics in distributed environments, and the network analysis solutions best suited for each situation.

The Heart of Your Network: The Network Operations Center (NOC)

Though not necessarily located in the center of your corporate campus, it is typically the center of your network, and is therefore a key point to monitor. The NOC typically includes most of the core routing resources, and acts as the hub for network traffic, especially traffic between users and key resources like application servers and data centers. The NOC is essentially a wired environment, so no need for wireless monitoring and analysis. But you will certainly have 1G and 10G network links in the NOC, so employing a network analysis solution specifically designed and optimized for 10G is a must, like TimeLine. And it’s highly recommended that you have a solution that allows for storing detailed network data (packets) over time, providing a complete record of your network transactions. As we’ve been known to say, “packets don’t lie.” At 10G, the last thing you want to do is try to reproduce spurious issues – it’s a whole lot easier to just play the packets back and analyze the original report. Plan for as large an appliance (in terms of disk storage) as you can so you can maximize the historical record you can save.

Where Virtualization Lies: Server Farms

In server farms you often find virtual servers with virtualized applications and data storage. It’s critical to have a solution in place that is able to see the traffic that is traversing within the virtual server, sometimes referred to as “hidden traffic” since it never traverses a physical NIC. These solutions can be software only (like WildPackets OmniVirtual), but may also include additional hardware if you wish to tap into the virtual data for multiple purposes.

Where Wireless Technologies Come Into Play: Remote Offices and Widespread Campuses

Remote offices and campuses have different networking requirements. Remote offices can often get by with wireless-only solutions, especially with the new 802.11n speeds, offering significant savings over wired networks. But keep in mind that all wireless data eventually becomes wired network traffic, with all data being sent back to the corporate data center, so wired network monitoring and analysis is often also required. Given that the max wired network speed at a remote office is likely to be 1Gbps or less, you can usually get away with a less expensive wired analysis solution, like our Omnipliance Edge.

Widespread campuses, with universities being excellent examples, demand mobility with very dense yet ephemeral usage, so wireless again becomes an excellent choice. Capturing wireless data requires a “point of presence”, typically within a 300ft radius of the problem. But this does not mean you need to be physically located within 300ft. Remote sensors, like a WildPackets OmniEngine with wireless adapters, or usage of access points themselves as remote packet collection devices, is very effective in performing remote wireless analysis.

Most importantly, you need to have visibility into all key areas of your network, whether local, remote, or virtual, and to leverage the best technologies on the market to keep your network running smoothly.