В этой части рассматриваются различные “сети”, с которым вы взаимодействуете в среде VMware, а также представлены рекомендации по конфигурации с двумя pNICs.

В архитектуре VMware существует 5 видов IP-сетей:

• VMware Management Network. This is the network that connects vCenter Server to the ESX Service Console. Since ESXi doesn’t have a Service Console, the ESXi Management Network is terminated at the vmkernel.

• VMotion Network. This network interconnects the various ESX/i (reminder, ESX/i is my shorthand notation for ESX and/or ESXi) hosts within a VMware cluster and enables VMotion among those nodes.

• NFS Network. The Network File System (NFS) Network is an IP Storage network that provides the interconnect between ESX/i hosts and one or more NFS servers that provide storage for virtual machine and ancillary support (i.e. .iso files).

• iSCSI Network. The Internet Protocol Small Computer Systems Interface (iSCSI) Network is an IP Storage network that provides the interconnect between ESX/i hosts and one or more iSCSI targets that provide storage for virtual machine and ancillary support (i.e. .iso files).

• Virtual Machine Network(s). One or more networks that support VM to access and provide services.

Figure 1. The Various Networks

You may have additional networks in your environment, and I’ll mention some of them as I get through this article. One of these that I’ll mention right now is the out-of-band (OOB) management network. While not required to make your VMware environment work, this network is present in many, if not most, enterprise VMware environments.

The out-of-band management network provides connectivity to your out-of-band management interface (i.e. HP iLO, Dell DRAC, IBM Director, Sun LOM, etc.). Users who have access to this network essentially have the keys to the kingdom. With proper authentication (or by hacking a username/password), they have direct console access, the ability to connect and disconnect virtual I/O devices (CD-ROM, floppy), the ability to power the host on or off, and pretty much anything else you could do if you were sitting in front of the system. This network needs to be protected at all costs! It should never be placed into a DMZ network. Another way to view the OOB network is as the door to your datacenter. You wouldn’t leave your datacenter door propped open to a back alley, so don’t leave your OOB network exposed to unnecessary risks, either!

VMware Management Network

Much like the Out-of-Band Network, the VMware Management Network is critical to the security of your virtual infrastructure. This network provides the management interface to the vmkernel – either through the service console (for ESX) or directly (with ESXi). This is the network where vCenter Server (a.k.a. Virtual Center) lives, as well as the path for ssh, web access, and third-party tools access, see Figure 2.

Figure 2. VMware Management Network

Again, in much the same manner as your OOB network, the Management Network can be viewed as a door into your datacenter. It needs to be protected very carefully and should never be exposed in a DMZ, and will frequently live behind a firewall with minimal ports exposed.

VMotion Network

The VMotion Network is a special-purpose network with only one use – the live migration of a running virtual machine from one physical ESX/i host to another with no interruption of service to clients of the VM. The VMotion interface is a vmkernel interface that is flagged as being used for VMotion. Figure 3 shows the VMotion network.

Figure 3. VMotion Network

All VMotion Network interfaces should be within the same IP broadcast domain to ensure that the hosts can find each other. No function other than VMotion needs to be able to access the VMotion network – in fact, for a two-node cluster, you can use a direct cable with no intervening switch to support VMotion.

It is important to know that the data sent across the VMotion network (on port TCP 8000) is not encrypted in any manner. This means that anyone who can connect their PC to the VMotion network will be able to listen in and intercept that data, which contains whatever happens to be in the virtual machines vRAM at the time of the VMotion. The information could include usernames, passwords, credit card numbers, you name it…it could be there.

With that knowledge, it should come as no surprise that I’m recommending that you protect the VMotion Network very carefully!

NFS Network

The NFS Network is a vmkernel network that supports access to Network File System v3 (NFSv3) shares over the Transmission Control Protocol (TCP). NFS is a file sharing protocol (much like Server Message Block [SMB] or the Common Internet File System [CIFS], the common Windows file sharing protocols (see http://en.wikipedia.org/wiki/Cifs). NFS was originally developed by Sun Microsystems in 1984 for use with Unix systems (see http://en.wikipedia.org/wiki/Network_File_System_(protocol)). Figure 4 shows a typical NFS network configuration.

Figure 4. NFS Network

Since NFS can be used to store virtual machines and/or utility files such as .iso images, it is very common for the NFS server to have multiple connections to the network. Depending on the NFS server or appliance that you are using, there are a variety of ways that these connections can be aggregated to improve the performance of your NFS network. Regardless of how the NFS server is connected to the network, the ESX host is bound by the rules of the vSwitch that’s used to support NFS traffic, as discussed in Part 3.

From a security perspective, the NFS protocol is not encrypted or otherwise secured on the wire. This means that anyone who has access to the NFS network has the ability to intercept data that represents the on-disk information stored in virtual machine files. Obviously, this is a significant risk that needs to be mitigated with appropriate configuration and management actions.

iSCSI Network

The iSCSI Network is quite similar to the NFS network. The primary difference between NFS & iSCSI is that iSCSI is a “block oriented” protocol whereas NFS is a “file oriented” protocol. What does that mean? Basically, it means that with NFS, it is the NFS server that is responsible for managing the filesystem and individual blocks of data on the disk. The ESX Server doesn’t care if the disks are formatted with NTFS, ZFS, ext3, or Ken’s File System – it never sees the structure on the disk. It is the NFS Server’s responsibility to read from, write to, and manage access of all information on the disk.

Conversely, an iSCSI Target (that’s what a single instance of an iSCSI server process is called), the ESX server is intimately knowledgeable about the on-disk structure (unless you happen to be using a Raw Device Map (RDM)) because ESX will communicate with the iSCSI Target using standard SCSI commands, exposing the actual on-disk blocks of data to the ESX host. In many, if not most, cases, the logical unit number (LUN) exposed by the iSCSI Target will be formatted by the ESX host as a VMware Filesystem (VMFS) volume. Figure 5 shows a typical iSCSI configuration.

Figure 5. iSCSI Network

As with NFS, it is not uncommon to see an iSCSI Target configured with multiple uplink connections into the network. And just like NFS, iSCSI is bound by the rules of the vSwitch for load balancing and failover.

I hate to be redundant, but this is worth stating: From a security perspective, the iSCSI protocol is not encrypted or otherwise secured on the wire. This means that anyone who has access to the iSCSI network has the ability to intercept data that represents the on-disk information stored in virtual machine files. Obviously, this is a significant risk that needs to be mitigated with appropriate configuration and management actions.

Virtual Machine Network(s)

Here’s where things get interesting! When you start talking about virtual machines, you’re talking about all the servers that live in your datacenter. These servers provide and consume services of all types – from other virtual machines, from physical servers in the datacenter, and from resources of all types on the Internet and other external networks. Obviously, this means that there may be the need to connect to more than one network to support all of these different communications paths. Figure 6 provides a view of some of the possible connectivity that needs to be supported by the VM Network(s).

Figure 6. Virtual Machine Networks

Notice that I’ve included an NFS Server and an iSCSI server in this diagram. You might ask “Why?” Well, it’s simple. The guest operating system inside a VM can directly mount an NFS volume and it can also use an iSCSI Initiator to connect to an iSCSI Target.

Some of these networks may need to be separated from each other. Depending on the level of sensitivity of each network, you may be able to use a single network divided only by IP subnets; you may need to use VLANs to provide a logical separation among the networks; and you may need to use totally separate vSwitches with dedicated pNICs to provide the required separation. I can’t help you with these decisions…that’s between you, your network team, your application team, your management team, and your security officer (good luck!).

Also notice that, for the first time in my initial six diagrams (other than the ESX Service Console), the network connections into the ESX/i hosts is not through a vmkernel port. In this case, the connection is via a vSwitch that is configured for Virtual Machine connectivity. That’s really just a technicality, a configuration setting. In reality, all vSwitches are owned and managed by the vmkernel.

Best Practices

There are some general guidelines that I like to use when designing a network architecture. Based on the discussion above, you can see that there are quite a few networks that need to / should be protected or isolated. I recognize that not everyone shares the same views on security and data protection – and that’s perfectly fine (as long as you understand the consequences!). I’ll try to accommodate as many positions as I can…on with the show!

There are two primary considerations when deciding how to carve up your networks: security and performance. The third consideration is manageability, and we’ll talk about it in a moment. The key to picking the optimal configuration based on the number of pNICs you have in your hosts is to understand the level of risk associated with mixing the various networks. I’ve created a couple tables that shows my personal assessment of the risk associated with each pairing. Table 1 shows the security implications while Table 2 shows the performance ramifications.

Table 1. Security Impact of Mixing Networks

My rationale for assigning these levels of security risk are as follows:

• The Management Network should be isolated as much as possible. When it is not possible to give it a dedicated vSwitch and associated pNICs (best case scenario), it is typically a “Medium Risk” to mix it with either VMotion or one of the IP Storage networks. The reason I chose medium risk is that the personnel granted access to the Management network are typically the most trusted in your organization. While it is never a good idea to allow users of any level of trust to access an unsecured storage or VMotion network, if you have to do it due to constraints, do it with your administrative users on your Management network segment.

• For the VMotion Network, there is “Low Risk” associated with the IP Storage networks. The reason for this is that there shouldn’t be any users on any of these three networks (VMotion, NFS, iSCSI), so there’s not much chance of someone intercepting data on the wire.

• The VMotion logic above applies to the NFS and iSCSI networks as well.

• The Virtual Machine Network(s) are always considered “High Risk”. This is because you have unknown/untrusted users connecting via potentially uncontrolled systems. There may be cases where you have specific VM Networks that are not high risk (for example, you may run vCenter Server as a VM. That particular VM would live on the Administrative Network which is a Medium Risk network).

Your assumptions about the level of risk associated with each network may be different than what I’ve suggested here. It doesn’t matter, simply substitute your values into the calculations and you’ll be set.

The second consideration I want to discuss is performance. Similarly to security, there are performance impacts to combining the various networks onto the same set of pNICs. Table 2 shows the matrix of performance impacts from mixing the various networks.

Table 2. Performance Impact of Mixing Networks

If you evaluate each network individually, you can develop an understanding of the traffic patterns that exist for each.

• Management Network: There is typically not a lot of traffic on this network. It is used for management functions such as vCenter Server operations (host configuration and management, virtual machine configuration and management, performance monitoring, etc.) and access by third-party applications (configuration management, resource monitoring, etc.). These are typically low-impact applications. The exceptions to the low-impact “rule” are the deployment of templates and the use of service console-based backup utilities.. Each of these functions has a significant impact on management network utilization.

• VMotion Network: This network sits idle except during a virtual machine migration; however, when a VMotion migration is taking place, you want to have as much bandwidth as possible available to enable the migration to complete as quickly as possible. If you do not use VMotion, you don’t need to worry about this network.

• NFS & iSCSI Networks: These are your IP Storage networks. Their utilization fluctuates wildly depending on what is happening in your virtual environment. During steady-state operations, there is typically a “moderate” level of activity; however, when virtual machines are being powered on, resumed, or backed up across these networks, there is significant activity.

• VM Network(s): This is a total crap shoot. In some environments, these networks sit almost idle, while in others, they are hit very hard. You will have to judge for yourself how significant your workloads are – although I will say that it is the exception rather than the rule where the traffic on these networks is “significant”.

Next, I’ve taken the two considerations and combined them. The result is a matrix that shows the overall “risk” of combining the various networks (see Table 3).

Table 3. Overall Impact of Mixing Networks

Again, these are not set-in-stone, has-to-be-this-way recommendations, but rather a tool to be used to help you make your decisions.

Oh, I nearly forgot – I promised to talk some about manageability. Basically, I recommend that each of these networks be separated into individual port groups, even if you’re not using VLAN tagging. If possible, create separate vSwitches for each of the major networks (Management, VMotion, IP Storage). For the VM Network(s), I recommend at least one vSwitch (depending on separation requirements) with a separate port group for each different network. Figure 7 shows two possible configurations.

Figure 7. vSwitch Options

Существует различное количество коммутаторов vSwitch в двух конфигурациях, одинаковых порт групп. Это логическое разделение позволяет легко управлять вашей средой – можно даже разные хозяева с разными конфигурациями коммутатора, еще одной группы портов конфигурации, что поддержка vmotion между ними. Это также позволяет легко масштабировать среду.

В следующей части вы рассмотрим рекомендации для различного количества pNICs в ваших хостах.