vSwitch to pSwitch Inconsistency

Switch Port Mismatch

Recently while rigging up a network config in vSphere, we ran into a strange scenario. It was not a greenfield site from a networking or vSphere perspective and the config was subject to certain constraints which I won’t go into here.

We had reason to have the following network configuration, using vSphere standard switches. A Green link indicates and Active adapter for the vmkernel port or virtual machine port group. An Amber link denotes standby. This environment did not have Enterprise Plus, so no dVS.

Blog Logical Design Trunk and Access

As you can see, the plan was to isolate vMotion on it’s own active vmnic.

This is a deterministic logical design, where we are not using LAG/LACP/PortChannel/Etherchannel.

The main reason for isolation of vMotion was due to the fact that it was being used for a large amount of migrations of VMs from an older cluster to a newer cluster. The number of uplinks was not ideal, and in fact this configuration has been modified completely in terms of port groups and uplinks/VLANs, to anonymise the customer setup.

So vmnic0 is setup as an access port on the switch. All other physical ports are setup as trunk ports, as they carry multiple VLANs. Most people know that’s the definition of a trunk port – the ability to carry multiple VLANs.

VMkernel port vMotion is configured to use vmnic0 as it’s active path and vmnic1 as it’s standby.

And because the Management port group is also using VLAN670, rightly or wrongly, and uses vmnic0 as it’s standby, only VLAN 670 is required on vmnic0. That’s why Port 0 is configured as an Access port.

[ Aside: If you’re going to use VLANs, don’t use the same VLANs for Management and vMotion vmkernel ports. Thats not good practice but this was a legacy configuration ]

The problem

The problem occurs immediately after installation of ESXi (5.1U1 or 5.1U2), version doesn’t matter. Once the management IP address is configured and the VLAN is set, all is good. We can ping the gateway, and lookup DNS.

Once we add the second vmnic (vmnic1) to the management configuration on the DCUI, the server disappears off the air. We were unable to ping it from another system on the same subnet.

We restarted the management agents (not the network) using the troubleshooting submenu on the DCUI.

More weirdness ensued…..

  • Firstly, we noticed the hostd daemon was continually crashing on the host in question.
  • We also noticed that sometimes when we performed a “Test Management Network” on the host, it worked the first time.
  • The next time we did it immediately after, everything failed (ping gateway, DNS servers and resolve DNS name).
  • When we googled it, we found the following link to a KB describing the creation of a file called /etc/hosts.backup in this scenario. There is a resolution and that was in the following KB article:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2047511

As suggested, we removed the file and could again test the management network and ping the host. However, when we tested the management network, the hosts.backup was created again and the next time a “Test Management Network” failed.

And round and round we go…..

Diagnosis

Initially we suspected we had a hardware issue.

  • We had 2 CCIE’s on site who could see nothing wrong with the network configuration.
  • We had a second host which was working ok, with no visible problems. This was the main focus initially so we tried to rule out hardware issues or a possibly bad install by rebuilding the host, and switching physical switch ports between the server. None of this resolved it.

It was actually one of my colleagues who figured it out. Despite the reassurances from our network comrades it actually turned out be simple.

We had to change switch Port 0 from an access port to a trunk port. Still with a single VLAN defined.

All of a sudden it all worked.

Why ?

If you think about it, when the Management vmkernel port is on vmnic1, VLANs are tagged and the trunk port works as expected. When we pull the vmnic1 cable and it switches to vmnic0, we are then using an access port. The VLAN is not tagged, so to speak, anymore, at least by the pSwitch. This causes the port group to mis-function.

If we apply the same logic to the vMotion port group. The starting position must be to leave it untagged on the vswitch. Otherwise it also malfunctions from the get-go. It works fine until we pull vmnic0. Then it switches to Port 1 which is a trunk port. With no VLAN tagging, it immediately gets into trouble.

So we have an inconsistent model. Even though the access port is set for VLAN670 and should work properly, it doesn’t.

Now …. from the Cisco website:

Configuring Access and Trunk Interfaces

Configuring a LAN Interface as an Ethernet Access Port

You can configure an Ethernet interface as an access port. An access port transmits packets on only one, untagged VLAN. You specify which VLAN traffic that the interface carries. If you do not specify a VLAN for an access port, the interface carries traffic only on the default VLAN. The default VLAN is VLAN1.

The VLAN must exist before you can specify that VLAN as an access VLAN. The system shuts down an access port that is assigned to an access VLAN that does not exist.

Anyway this was an issue, that due to suspicion of host-specific issues distracted us from the real issue. It’s incredible how something like this can lead to arbitrary inconsistent behaviour.

 

 

 

 

 

 

 

7 thoughts on “vSwitch to pSwitch Inconsistency

  1. Very good catch, and another indication that a vSphere architect needs to understand the full stack not just VMware products.

    That said, I am a little disapointed that the issue was not spotted by your CCIE’s though, Access and Trunk port definitions are Networking 101 in the CCNA handbook 🙂

    1. Hey Tom.
      Thanks for your comments. I agree and probably should have known this myself. My colleague wasn’t happy about it. I’ve done the CCNA and am not sure why the VLAN ID was configured on the access port definitions if it was going to be dropped. The use case was clearly described to the CCIE in question so I assumed all was correct. You’re right – this is an example of VCDX level knowledge that is required. End-to-End knowledge with ownership of the full technical solution. However it’s also a case where we didn’t look at this initially as the other server stayed up. Through a process of elimination we eventually got there.

      Thanks again.
      Paul

  2. Great Post Paul. I was several times in that situation.

    Please correct me if I’m wrong but when removing the 670 VLAN tags from the vSwitch PG and set 670 as native VLAN on TRUNK Port 1 than there would also be no problem. It’s still no recommended configuration but it would be a workaround.

    Was there a reason why the configuration look like this? Requirements?

    Cheers
    Fred

    1. Hey Fred,
      I left some stuff off the design. In reality it also had two iscsi port groups. Originally it was intended that direct attached storage would be used (not within scope of my design). The cables never arrived so I was focused on trying to support iSCSI and vMotion separately to facilitate a data migration using storage vmotion. This was being configured in light of the storage solution changing for DAS to iSCSI. That’s why vmotion was isolated like that, and yes I discussed with my CCIE comrades and asked them many times why the network issues were present. Another reason to try and introduce separation was that the existing design was all active uplinks and all port groups spread everywhere with iSCSI traffic too.

      Obviously it is more normal to have all uplinks configured as trunk ports. Also we wanted to change vMotion and Management PGs to different VLANs but didn’t want to introduce that change in the middle of our migration window. There were also two other clusters in place which we will potentially need to use vMotion with to move VMs onto this new cluster. If the VLAN change was made for Management we would had to have changed the port to a trunk port. As we all know that’s normal which is why this design was a bit strange. Most of my designs I alternate the vMotion and Management like that, which normally requires trunk ports to support both VLANs.

      So best to configure all ports as trunk ports. I still think you should not allow all VLANs – just get ones you need. They can be added later on the fly.

      Talk soon,
      paul

  3. Hi Paul,

    Thank you for the article. I have couple of doubts that I wanted to verify with you

    While creating the port groups for vMotion 670 and Management 670; did you specify the vlan id at the Virtual Switch level? My doubt is if the vlan ids were configured at the Virtual Switch level then the packets were being sent for the specific vlan id and were tagged before they reached the pNIC.

    If yes; then will access or trunk port cause issues.

    I understand that Packets sent to an access port are untagged and packets sent to a trunk port are tagged.

    Please pardon me for the silly question as I do not have depth knowledge about them (access and trunk ports).

    Thanks in advance

    1. Hi Vaibhav,
      Thanks for your comment.
      Yes. The VLAN was set at the port group / vmkernel port level in the vswitch. So the packets were tagged before reaching the access and trunk ports. What happens is that the vswitch does not differentiate between an access or a trunk port. So when the packet reaches the access port the VLAN tag is not respected and is dropped. That’s why we ended up having a problem. So in theory the configuration is supported but will not work properly for the virtual switch design, especially when you start pulling cables to test vmkernel port failover, that’s when the the pswitch/vswitch inconsistency appears.

      As I mentioned in another reply, it is unusual to have this configuration where both Management and vMotion vmkernel ports use the same discrete VLAN, at least in my designs. That was a legacy configuration which was in place on two other clusters, so we didn’t want to change it in the data migration window. In a “normal” setup, all ports would need to have been trunk ports anyway, as we would typically setup the management and vMotion vmkernel ports with VLANs.

      There’s no such thing as a silly question. I ask them all the time ;-). It was also to demonstrate that sometimes networking people maybe have a different understanding/perspective to vSphere admins/designers.

      Best,
      Paul

  4. When realising it worked via one NIC but not the other, the first thing to look at should be the obvious configuration differences. The most fundamental difference is that the primary port is an access port but the standby is a trunk. Temporarily making these two configurations consistent (I.e. both trunks or both acess) ought to be the first troubleshooting step if one can’t identify the problem from logical analysis alone. Still, there’s been many a time I’ve figured out what I *should* have done after spending many wasted hours.

Leave a Reply

Your email address will not be published.