Tag Archives: Virtual Networks

Where to Capture Packets in High-Speed and Data Center Networks

Network analysis changes dramatically as network speeds grow (10G, 40G, and up to 100G). From more packets to capture, to changing traffic patterns like East-West traffic among servers, network analysis strategies must adapt as new technologies are introduced. We’ve written in the past about best practices for network monitoring on high-speed networks. However, we have never gone into detail on how to position capture points to see inside areas that might be neglected by previous monitoring methods.

Below we go into where you should set up your capture points to get the most visibility on your high-speed network.

Capturing Data on the Network
Typically, if you are connecting directly on the network you are going to collect data through traditional SPAN ports, mirror ports, or taps. This is a well known method that is used frequently to obtain a passive feed from a network, so the process and any associated configuration should be familiar. One challenge does arise when virtualization is in use, as you will miss intra-host traffic when capturing only on the physical network. Don’t worry, below we will explain how you can capture this traffic as well.

In this video, you will see where to capture traffic if you are in the data center, a corporate campus, or remote office.

Capturing Data on vSwitch
Data on virtual servers pose a unique challenge, as oftentimes much of the data never leaves the virtual server – for example, communication between and application and a database running on the same virtual machine. In this case, capturing data off the span port of the virtual switch or hypervisor allows you to get visibility into intra-host traffic. To do so, you either need to have network analysis software running directly on the server, or you need a “virtual tap” (a piece of software) that can perform the function of a traditional hardware tap and copy network traffic off to a separate physical tap which can then be utilized in a traditional fashion. If you’re running the network analysis software directly on the local VM, remember to allocate enough memory, IO, and disk space to accommodate your network analysis needs.

Capturing Packets in the Cloud
Cloud computing comes in many shapes and forms. If you are trying to capture data in a private cloud, the practice and procedure will be similar to that of capturing on your vSwitch. If you control the infrastructure, you can sniff anywhere. If you are a service provider, you need to carefully consider data access, data separation, and customer privacy issues.

If you are using a third-party cloud service, the ability to capture and monitor traffic is going to depend on the implementation. If you are running software-as-a-service (SaaS) from a provider, it will be hard to have sniffing rights, so your last point of knowledge about your traffic will be at WAN link. This will still allow you to obtain valuable analytics, like round trip latency, which will provide a good indication of the overall user experience. However, if users are experiencing latency and you think that it might be an application performance problem and not an overall network problem, then it will be difficult to analyze the situation. For example, a database connection issue or database contention may be very difficult to troubleshoot. But then again, isn’t that why you’re paying your SaaS provider?

If you are employing infrastructure-as-a-service then you will have the ability to sniff your own traffic by installing a network analysis software probe on the hosted virtual server to see all the traffic on the virtual  server, thereby restoring your ability to analyze application issues that may otherwise be hidden.

If you are working within another environment and would like tips on capturing data, please leave us a comment.

Announcing Our Upcoming Webinar Series

Searching for information on 802.11ac? Having problems with blind spots in your virtual network? Looking for general tips and tricks on how to analyze and troubleshoot common network problems? We’ve got some great topics coming up to answer these questions in our webinar series. Check them out and RSVP today!

802.11ac – Wireless Gigabit Speeds Driving Changes in Wireless Analysis
Wednesday, April 17, 2013, 8:30 am PDT

802.11ac is the very latest technology to be introduced as part of the 802.11 family of specifications. Join us as we dive into 802.11ac, and demonstrate what we believe will be the new standard in wireless packet capture and analysis.

The Blind Spot in Virtual Networks: Seeing with Network Analysis
Wednesday, May 15, 2013, 8:30 am PDT

Virtualization provides tremendous efficiencies, reducing the cost of equipment, management, and even utilities. But as with most technological shifts there are consequences, especially in network analysis, that must be addressed. Virtualization, regardless of the “flavor,” creates a blind spot – a loss of visibility into traffic between virtual applications or virtual systems – when using traditional network analysis products and techniques. In this webinar, we will dissect this problem and demonstrate ways to overcome these network blind spots.

WildPackets Tips and Tricks
Wednesday, June 19, 2013, 8:30 am PDT

In this web seminar our team of customer-facing engineers will highlight some of their favorite problems, including how they uncovered the problem, what they recommended to solve it, and how they verified that the proposed solution was successful. Please join us for this highly informative, real-world demonstration of what really goes wrong in enterprise networks. And get your questions answered too!

Q&A: How IT Trends Are Affecting the Network

We get a lot of questions from our customers about how they should prepare themselves for different technologies (cloud, 10G, etc.). For this blog we wanted to answer some of the common questions that we receive from our customers – mainly network engineers/administrators – regarding both specific networking trends as well as larger technology trends.

We have Jay Botelho, Director of Product Marketing at WildPackets, answering these questions. He’s been in the networking business for over 25 years. If you have any additional questions for our networking guru please be sure to let us know and we’ll address them in our next Q&A blog.

Q: What are some best practices for troubleshooting in a virtual environment?
A: Virtualization creates “blind spots” in your network, making it difficult to monitor with traditional techniques, i.e. spanning a switch port from a physical Ethernet switch to collect packet-based data for complete root-cause analysis.

Here’s a common scenario. A user is experiencing abnormally long delays while working with a specific application. You know from your network architecture that the app is running on a VM, but you’re not the application engineer so you’re not familiar with all of the nuances of the application’s operations (what data sources it accesses, under what conditions, etc.). You’re able to start a packet capture session on a switch just upstream from the virtual server running the app, and after filtering and watching the user connection to the app you can confirm long delays for some operations, but it is clear the delay is NOT between the user and your capture point. The delay is within the VM, in your blind spot, where communication between the application and the database are virtualized on the same VM.

To address this issue, you must collect data from  the virtual switch(es) within the VM in order to get visibility between the application and the database. There are several techniques to achieve this. First, you can use OmniVirtual, a packet-capture software probe specifically designed to run on VMs. You will need to allocate space on the VM to run OmniVitual, just as you would any other application. Once running, OmniVirtual will have access to all data crossing any of the virtual switches on the VM.

A second alternative is to use a virtual tap, available from several tap vendors. These virtual taps install at the VM layer, acting like a traditional tap, providing access to data crossing virtual switches on the VM to a host of solutions, including network analysis and troubleshooting solutions like the Omni Distributed Analysis Platform.

Q: How will migrating to a public Cloud affect my job?
A: There’s some very good data available from industry analysts that predict you will be busier as your company migrates to the Cloud, so don’t worry about your job! But your role will change from managing not only your own infrastructure, but overall service availability and performance in the Cloud as well. Cloud computing merely shifts your application servers from your facility to a third party. Issues like bottlenecks, bandwidth hogs, and unauthorized protocol usage will still adversely affect application traffic. Thus, more diligence must be applied in monitoring application performance and making sure that your service provider is living up to its promises.

Q: Why do issues with VoIP continue to persist?
A: This is a question that we consistently get from both potential customers and customers looking to deploy VoIP. The core issue is that networks are really not that friendly to real-time data (RTP, or real-time protocol, which is used by VoIP and video). Most networks are optimized to carry TCP/IP data traffic, which is much more tolerant of latency, packet loss, and jitter than is RTP. To compensate for this, networks require additional configuration for VoIP, including the use of Quality of Service tagging – QoS (at a minimum) – or even dedicated VLAN or MPLS segments to segregate and give priority to RTP traffic. If you either have or are planning a transition to VoIP, be sure you are using a network analysis solution that treats VoIP like any other data type on the network, since that’s exactly what it is. Often times VoIP problems spike during times of heavy network usage, so you need a solution that can see everything at once and allow you to correlate the activity of all traffic on your network simultaneously.

Q: How will my network analysis needs change as we roll out 10G?
A: 10G is a game-changer for network analysis and troubleshooting. The still oft-used “break/fix” method of network troubleshooting – where packet-based network analysis is only performed after a problem is reported – is no longer effective at 10G. With more data consolidated through fewer resources, the number of problems per segment increases, and the increased network speeds make it far more challenging to try to reproduce problems, or wait for them to happen again. At 10G you need to monitor and record packet-level data on an ongoing basis, arming yourself with a recording of all activity on these highly-utilized segments. If real-time monitoring indicates a negative trend, or if problem reports are rolling in, you can simply “rewind” the network to the troublesome period of time and analyze exactly what was going on. No waiting for it to happen again, and no need for Herculean efforts to reproduce the problem. You have all the data you need to solve the problem – immediately.

This wraps up our first Q&A session. Please keep your questions coming. We’re always up for a challenge, and let’s face it, we picked the softballs this time around…