**First and foremost, thank you Michael Kolos for the assist on this blog!!!**
If there is anything I’ve learned over the past four years with VMware Cloud on AWS (also applies to other hyperscalers) it’s that spinning up the SDDC is easy. It’s connecting on-premise and other cloud providers together that’s the hard part. In order to simplify these connections, VMware engineered a little bit of magic with VMware HCX™. In summary, VMware HCX™ is an application mobility platform designed for simplifying application migration, workload rebalancing and business continuity across datacenters and clouds. As a part of an HCX deployment, a few appliances are deployed. HCX Manager, WAN Interconnect (IX)m, WAN Optimization (WAN-OPT), Network Extension (L2C), and a Proxy Host. More details can be found here. For this post, I am going to focus on enabling MON via the HCX Network Extension and why you need to understand policy routing. I’ve run into this a few times with customers. Hopefully, this helps!!
So what is Mobility Optimized Networking?
Mobility Optimized Networking improves routed connectivity patterns for multi-segment applications and virtual machines with inter-VLAN dependencies as those virtual machines are migrated into the cloud. Without MON, HCX Network Extension expands the on-premises broadcast domain to the cloud SDDC while the first hop routing function remains at the source. The network “tromboning” effect is observed when virtual machines connected to different extended segments communicate. More information can be found here.
Scenario: Customer has a Layer 2 network stretched via HCX L2C. In order for the customer to allow their teams to deploy and manage workloads on VMC on a secondary domain, they want to add a Windows Domain Controller as an Identity Source to the VMC SDDC vCenter. This DC is deployed on the same VLAN where the network is stretched. Firewall rules have been validated similar to my previous blog which details the rules needed for the domain controller to communicate with the vCenter. When trying to add the following error appears. F So what gives?!!!!!
Here’s my list of assumptions:
- A source VM, on an HCX L2E network with MON enabled, and the VM set to use the local (MON) gateway – not the source(on-prem) L3 gateway.
- Default policy routes (RFC1918) in place.
- VM trying to reach the vCenter of the local SDDC (i.e. the one where the MON gateway is, not the source L3 gateway)
In this scenario, MON will only optimize the path for other VMs on the MON enabled networks whose gateway is also the MON network, or for routed segments in the same SDDC (connected to the same T1 Router aka Compute Gateway). The SDDC’s management CIDR (which vCenter is a part of) is not connected to the same T1 Route as the vCenter is connected to the Management Gateway. As a result, there is not a matching T1 route. Without this route, the traffic decision then moves to the policy routes. Since the vCenter is using private IP, which is in the default Policy routes for MON, the traffic will be sent back over the L2E to the source L3 gateway. Depending on the routing configuration, this path may or may not work.
To resolve this, we need to modify the default policy routes. Depending on what the desired path is (e.g. what destinations are reachable on the on-prem side), we can update them to match that. The easiest way to do this is to add a DENY route with the SDDC’s management CIDR to the Policy Routes.
In my lab, the SDDC uses 10.62.0.0/16 as the management CIDR, this is how you can add the Policy Route. Go to HCX Manager via the HCX Center Plug-in or HCX Manager URL and select Network Extension > Advanced > Policy Routes.
Enter in the SDDC Management CIDR as a deny rule.
Once this change is made, the Domain Controller is able to communicate with vCenter and is able to be added as an identity source.
Street2Clouds 
