Replacing a failed 6248XP Fabric Interconnect

To date, I’ve not had to replace a running 6120XP Fabric Interconnect but just days ago, I had to replace a 6248XP Fabric Interconnect that had been configured as part of a FI Cluster. Not long after configuring my FI Cluster, FabricA started rebooting itself anywhere every 60-90 minutes. I called Cisco TAC and after a bit of troubleshooting, it was determined that the 6248 needed to be replaced. I was told that the field engineer would install the new fabric interconnect and re-create the FI cluster, but…when the field engineer arrived he said, “I don’t do configurations…”, thus began this post.

Replacing a Fabric Interconnect is not hard really, you just need to make sure its done right to make the changeover go smoothly. The high-level steps are as follows:

1. connect the new FI to the network (do not connect the L1 and L2 cables)

2. console into the new FI and run through the setup wizard configuring the new FI as a “standalone” fabric interconnect

3. Assuming the new fabric interconnect is not at the same code level as the current FI, open your internet browser and connect to the IP address assigned to the new FI, login as admin, and upload the UCS code that is installed on the cluster member.

4. Still connected to the new FI, update both UCS Manager and the Fabric Interconnect to the code running on the existing cluster member.

5. Once the upgrades are complete, verify the Running and Startup versions match those of the existing cluster member.

6. Console into the new fabric interconnect and run the following commands:
:connect local-mgmt
:erase configuration
:yes (to reboot)

7. Connect the L1 and L2 cables to the existing fabric interconnect

8. Console into the new fabric interconnect and run through the setup wizard. When the setup wizard detects the presence of a peer Fabric interconnect, type y to add the new FI to the existing cluster. Save the configuration and reboot the new FI.

9. Login to UCS Manager or use the command line to verify the cluster state.

10. Test failover. In this example, I use the command line on Fabric-B as Fabric-A was the “new” fabric interconnect. From the command line of Fabric-B, enter the following commands:
:connect local-mgmt
:show cluster state (verify B is still the primary)
:cluster lead a (makes Fabric-A the primary)
:show cluster state (if done quickly, you’ll see the status of SWITCHOVER IN PROGRESS)

11. Login to UCS Manager once again to verify the failover is seen in the GUI and you are done.


Filed under Cisco

2 responses to “Replacing a failed 6248XP Fabric Interconnect

  1. Anonymous

    Gonna try it right now. Love concise explanations. Basically what I already knew but it's nice to see it laid out so clearly rather than pulling it from the dregs of Cisco's documentation.

  2. Anonymous

    Hi, thanks for this post. A question about the procedure…is it disruptive? When you replaced the new FI, all services were up on blade servers connected?

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s