Recently, a customer had an issue where, upon reboot, Windows-based VMs hosted on vSphere servers would detect a duplicate IP address and then assign themselves 169.254.xxx.xxx IP addresses. These VMs had been operating for years without issue so why is this taking place now? The client had an interesting observation….that the problem started as soon as they migrated their vSphere hosts to Cisco Catalyst 3850 network switches.
The issue they were experiencing is basically identical to the one described in the VMware KB1028373 and the primary workaround is to turn off gratuitous ARP in the guest operating system using an ArpRetryCount registry key. That workaround was attempted by the client before I was engaged but to no effect.
In speaking with network engineers, they were of the belief that IP Device Tracking was the culprit, not Gratuitous ARP but why? Two reasons it seems, first Microsoft introduced (with Vista and above) a new mechanism to detect duplicate IP addresses and the Cisco 3850 switch is aimed more at the access layer level (so I’ve been told), and not necessarily configured out of the box for vSphere/VM functionality, though it can certainly be configured to serve as core/datacenter switches in smaller environments.
From Cisco TAC Article 116529:
Cisco IOS uses the Address Resolution Protocol (ARP) Probe sourced from an address of 0.0.0.0 in order to maintain the IP device-tracking cache when IP device tracking and a feature that uses it is enabled (such as 802.1x) on a Cisco IOS switch.
Duplicate IP Address Cause
If the switch sends out an ARP Probe for the client while the Microsoft Windows PC is in its duplicate-address detection phase, Microsoft Windows detects the probe as a duplicate IP address and presents the user with a message that a duplicate IP address was found on the network for 0.0.0.0. The PC does not obtain an address, and the user must either manually release/renew the address, disconnect and reconnect to the network, or reboot the PC in order to gain network access.
After some discussion, the decision was made to disable the IP Device Tracking feature altogether using the following commands….the example assuming (2) 48-port switches:
- config t
- int range g1/0/1-48, g2/0/1-48
- nmsp attach suppress
- copy run start
False duplicate IP messages have not appeared since making this change.