Category Archives: Networking

Policy Routing with HCX Mobility Optimized Networking (MON)

**First and foremost, thank you Michael Kolos for the assist on this blog!!!**

If there is anything I’ve learned over the past four years with VMware Cloud on AWS (also applies to other hyperscalers) it’s that spinning up the SDDC is easy. It’s connecting on-premise and other cloud providers together that’s the hard part. In order to simplify these connections, VMware engineered a little bit of magic with VMware HCX™. In summary, VMware HCX™ is an application mobility platform designed for simplifying application migration, workload rebalancing and business continuity across datacenters and clouds. As a part of an HCX deployment, a few appliances are deployed. HCX Manager, WAN Interconnect (IX)m, WAN Optimization (WAN-OPT), Network Extension (L2C), and a Proxy Host. More details can be found here. For this post, I am going to focus on enabling MON via the HCX Network Extension and why you need to understand policy routing. I’ve run into this a few times with customers. Hopefully, this helps!!

So what is Mobility Optimized Networking?
Mobility Optimized Networking improves routed connectivity patterns for multi-segment applications and virtual machines with inter-VLAN dependencies as those virtual machines are migrated into the cloud. Without MON, HCX Network Extension expands the on-premises broadcast domain to the cloud SDDC while the first hop routing function remains at the source. The network “tromboning” effect is observed when virtual machines connected to different extended segments communicate. More information can be found here.

""

Scenario: Customer has a Layer 2 network stretched via HCX L2C. In order for the customer to allow their teams to deploy and manage workloads on VMC on a secondary domain, they want to add a Windows Domain Controller as an Identity Source to the VMC SDDC vCenter. This DC is deployed on the same VLAN where the network is stretched. Firewall rules have been validated similar to my previous blog which details the rules needed for the domain controller to communicate with the vCenter. When trying to add the following error appears. F So what gives?!!!!!

Here’s my list of assumptions:

  • A source VM, on an HCX L2E network with MON enabled, and the VM set to use the local (MON) gateway – not the source(on-prem) L3 gateway.
  • Default policy routes (RFC1918) in place.
  • VM trying to reach the vCenter of the local SDDC (i.e. the one where the MON gateway is, not the source L3 gateway)

In this scenario, MON will only optimize the path for other VMs on the MON enabled networks whose gateway is also the MON network, or for routed segments in the same SDDC (connected to the same T1 Router aka Compute Gateway). The SDDC’s management CIDR (which vCenter is a part of) is not connected to the same T1 Route as the vCenter is connected to the Management Gateway. As a result, there is not a matching T1 route. Without this route, the traffic decision then moves to the policy routes. Since the vCenter is using private IP, which is in the default Policy routes for MON, the traffic will be sent back over the L2E to the source L3 gateway. Depending on the routing configuration, this path may or may not work.

To resolve this, we need to modify the default policy routes. Depending on what the desired path is (e.g. what destinations are reachable on the on-prem side), we can update them to match that. The easiest way to do this is to add a DENY route with the SDDC’s management CIDR to the Policy Routes.

In my lab, the SDDC uses 10.62.0.0/16 as the management CIDR, this is how you can add the Policy Route. Go to HCX Manager via the HCX Center Plug-in or HCX Manager URL and select Network Extension > Advanced > Policy Routes.

Enter in the SDDC Management CIDR as a deny rule.

Once this change is made, the Domain Controller is able to communicate with vCenter and is able to be added as an identity source.

Native AWS VPC Connecting to VMware Cloud on AWS (Part 2) – Route Tables and Transit Gateway Attachments

Welcome to Part 2 of VPN connectivity to VMC via AWS VPN. For this post, I will walk through how to attach an AWS VPC to an AWS Transit Gateway in order for route tables to be learned by the VMC SDDC and AWS VPC that will ultimately allow communication from workloads in AWS and workloads in VMC. If you immediately try to ping across the VPN from an EC2 instance to a VMC VM, it will fail as there is more work to do in order to create communication pathways in the form of adding a VPC to the Transit Gateway as well as configure some NSX rules.

In the diagram above I created two EC2 instances with an AWS VPC. For the sake of testing and validation, I first had to create a jump VM with access to the internet in order to remote into the workload VM. This is because, in order for the workload VM to ping across the VPN, 0.0.0.0/0 must be routed through the Transit Gateway and not an Internet Gateway. The goal here is to have a VM on the 10.0.10.32/27 subnet ping a VM on the 10.250.10.0/24 subnet.

To start, the first thing to check is the routing table for the Transit Gateway. This is done by selecting “Transit Gateway Route Tables” on the sidebar with the VPC.

You will then see a list of learned routes from the VMC SDDC. For the sake of this blog, I have deployed AWS EC2 Instances on a subnet that I plan on attaching to the Transit Gateway (10.0.10.0/24). However, notice below that only the learned routes for the VPN to VMC are listed below. While the 10.250.10/0/24 subnet is learned, in order to open communication via the AWS VPC, we first need to attach the VPC to the Transit Gateway so the 10.0.10.0/27 subnet is included. Here are the steps to create the VPC attachment to the Transit Gateway.

This image has an empty alt attribute; its file name is image-18-1.png

Step 1: Select “Transit Gateway Attachments” then “Create Transit Gateway Attachment”. Find correct Transit Gateway ID.

This image has an empty alt attribute; its file name is image-20-1.png

Step 2: Add tag for attachment name.

This image has an empty alt attribute; its file name is image-17-1.png

Step 3: Select the correct VPC and ensure you have the correct subnet selected. Select “Create Attachment”

This image has an empty alt attribute; its file name is image-19-1.png

As you can see, now that the VPC is attached to the TGW, the 10.0.10.0/24 subnet is now listed. This will now allow for the routing of traffic from my EC2 instance to my VM on VMC.

This image has an empty alt attribute; its file name is image-25.png

The final step is to create groups and add inbound and outbound firewall rules within the Network & Security section of the SDDC console. Make sure to apply the rule to the VPN Tunnel Interface and NOT “All Uplinks“. The reason for this is that a firewall rule applied to All Uplinks does not apply to the VPN Tunnel Interface (VTI), which is a virtual interface and not a physical uplink. The VPN Tunnel Interface must be specified explicitly in the Applied To parameter of any firewall rule that manages workload VM communications over a route-based VPN. In addition, you can see how groups are created on an earlier blog here.

Now that the routing table is complete and Compute Gateway Firewall rules are in place, you can test!

EC2 Windows VM
Ping results from EC2 VM to VMC VM

VMworld 2021 – 11 Sessions I’m Excited About!!!!

While I’m disappointed I can’t see my coworkers and customers again this year at VMworld this October 5-7, I’m still looking forward to all the great content that will be shared. One of the benefits of virtual is again this year is that it’s free for all attendees! Here’s my Top 11.

VMware vSAN – Dynamic Volumes for Traditional and Modern Applications [MCL1084] – Duncan Epping and Cormac Hogan are at it again presenting their deep knowledge of VSAN in both traditional and modern application use cases. I’m looking forward to see their take on VMware vSphere container storage interface (CSI) in Kubernetes! 

William Lam – App Modernization Deep Dive with VMware Cloud on AWS and VMware Tanzu [MCL2290] – Is VMworld even VMworld without William?! I have been waiting for a long time to talk VMWonAWS and Tanzu!! For those of you who want to see modern apps with Tanzu on VMWonAWS, this session is a must!

Achieving Happiness: The Quest for Something New [IC1484] – Those of you who have met Amanda Blevins know that she’s not only about technology but is also passionate about personal development and brand building. Many things have changed over the past 18 months with our day to day profession and I’m anxious to see what insights Amanda and Steve Athanas (CIO UMass Lowell) will have for us!

A Guide to vSphere with VMware Tanzu: Day 2 Operations for the VI Admin [APP1718] – No doubt Dean Lewis and Simon Conyard will bring their technical acumen and British wit to the session as they cover basic Kubernetes architecture in a way that makes sense for the VI Admin. Kubernetes is a fun word to say, but it’s a completely different thing to say AND do in the enterprise….at the end of the day, you still need to manage the application. These two gents will show you how!

An End-to-End Demo of Day 0 to Day 2 VMware Tanzu Management with vRealize [APP1586] – Matt Bradford and Sam McGeown always create great demos for their sessions. This is a must see for those on the Tanzu and modern application path and want to see how the vRealize suite is making Day 0-2 a cinch.

A Guide to the Cloud Operating Model [MCL1115] – Clouds are becoming the new silos. SaaS can grow your environment exponentially at a rapid pace and before you know it, you’re in the same siloed chaos you were in before cloud. Taruna and Martijn walk you through VMware’s multi-cloud approach when creating a consistent cloud operating model. It’s great to leverage multiple clouds based on specific use cases but it’s important to know how to best manage them.

Design Principles: Cloud Architecture Design and Operations [MCL2151] – Without a doubt these two Principal Architects are some of the smartest people I know at VMware. Mitesh and Andrea have been designing Enterprise VMWonAWS deployment since the service has been available. If you want to know how best to design VMWonAWS for production, this session is #1!

Automate VMware HCX Workload Migration to VMware Cloud on AWS [MCL1050] – This session would be #2 to the one above. Now that you have the VMWonAWS SDDC deployed, it’s time to migrate! Phoebe and Asaf bring their VCDX (between them, they have four!) and HCX knowledge to show you how to automate your migrations.

Cloud Workload Security and Protection on VMware Cloud [SEC1296] – While you’ve migrated workloads to VMWonAWS, you still need to secure them. Being in the cloud does not remove you from needing to protect the asset. Using the security features of NSX on VMWonAWS is a great start. To be even more secure, this panel will show how you can leverage Carbon Black on VMWonAWS.

A Guide to Application Migration Nirvana [MCL1264] – Bottom line….application migrations can be HARD! vRealize Network Insight has quickly become one of the main tools used to help customers understand applications and how to effectively plan for migrations to VMware’s Cloud. Martijn Smit has a wealth of experience to share do be sure to add this to the schedule!!

VMware DRaaS – Combine Two Services for Comprehensive Disaster Recovery Plans [MCL1202] – This session should be awesome! It’s not just about Site Recovery Manager (SRM) anymore. If you haven’t taken a look at VMware Cloud Disaster Recovery (VCDR aka Datrium) yet, you should. This session will cover both solutions and how we’re allowing customers to recover from ransomeware attacks, outages, and more. It’s all about flexibility and this session will give you the information you need to make those critical business continuity and disaster recovery decisions.

I admit that most of these sessions are cloud and application based but that’s where my passion lies and that’s where my customers are headed! Don’t forget to register today and Enjoy VMworld 2021!!!!

Native AWS VPC Connecting to VMware Cloud on AWS (Part 1) – Native VPC Connectivity via VPN

With the release of VMware Cloud on AWS 1.12, we delivered additional connectivity options with the addition of transit connect (VMware managed AWS Transit Gateway). Over the past few weeks, I have been working on a project that has specific success criteria as well as challenges that prevent certain connectivity options so I thought what better way to show how we got around the issue than to show how we did it as Transit Connect and AWS Direct Connect were out of scope. The plan is to have this be a multiple part blog series that details how to setup and test a route based VPN, attach to native AWS EC2 workloads via ENI in the VMC VPC as well as leverage a Transit connect that makes connectivity easier to configure, manage and maintain. The diagram below is the overall architecture for those already leveraging AWS native services via a Direct Connect and Transit Gateway but can only connect to VMC via IPSEC VPN for certain reasons (shout out to fellow VMC Architect Will Lin for the assist!). While Direct Connect provides much better speeds and feeds, you may still be able to accomplish what you need depending on your application requirements.

STEP 1: To get things started, we need to either identify or create a VPC that we want to communicate with the VMware Cloud on AWS SDDC. If you already have a VPC configured you can skip to Step 2. To create a new VPC, log into your AWS account go to Services > VPC> Create VPC. Similar to creating your VMC SDDCm it is paramount that plan your CIDR range appropriately so can assign subnetting correctly based on what you are trying to accomplish. ** Make sure you are selecting the correct AWS Region based on your requirements! **

Next, create a subnet that you will assign to the VPC. This is why understanding the CIDR ranges is important as you cannot have any overlap between your VPC CIDR and VPC subnets so keep that in mind as you build things out.

Creating a subnet assigned to the AWS VPC

Step 2: Deploy a AWS Transit Gateway (TGW). For reference, an AWS Transit Gateway connects VPCs and on-premises networks through a central hub. Keep in mind the default ASN number for the AWS TGW is 64512. If you are going to use an existing TGW, get the correct number from your team. For the sake of this blog, I setup the TGW with all the default settings.

Step 3: Create a Customer Gateway (CGW). A CGW is a resource that you create in AWS that represents the customer gateway device in your on-premises network, or VMC in this situation. 

In order to configure the CGW, you need to enter the VPN Public IP listed in the Networking & Security section of the SDDC console.

Step 4: With your transit and customer gateways configured, it’s time to create the VPN connection from the AWS side. Go to Services > VPC > Site-to-Site VPN Connections > Create VPN Connection. Name the VPN connection and select “Transit Gateway” as the gateway type and add your CGW. Take note that you can also add a new CGW didn’t as a part of the previous steps. We want to set the routing options to dynamic so we take advantage of BGP. My personal preference is to define the CIDR and per AWS documentation, this needs to be a /30 within a certain range. I also created a basic preshared key rather than have AWS create the key randomly.

AWS VPN Configuration

Step 5: Download the VPN configuration as a generic file. This is where you can validate the configuration and use it for configuring the VPN from the VMware Cloud SDDC Console.

Step 6: Configure the VPN on the SDDC. Go to the VMC SDDC Console > Networking & Security > VPN > Route Based > Add VPN. Here you want to take the Virtual Private Gateway Outside IP and enter that into the Remote Public IP Field. Take the Customer Gateway Inside IP Address and enter that number as the BGP Local IP. Next, take the Virtual Private Gateway Inside IP and enter that into the BGP Remote IP field. All that’s left is to enter the BGP ASN number that your configured earlier as a part of the TGW creation. It should also be listed in the config file you downloaded. If all numbers are correct, you should see the VPN Tunnel and BGP come up in a matter of minutes! If the VPN comes up and BGP does not, check your IPs and ASN numbers. Additional help can also be found here!!!!

You now have a working VPN from AWS to VMC!!! While the tunnel is up, there is more work to do so you can fully test traffic. In Part 2 I will cover routing tables and SDDC Gateway rules to enable two way communication. Stay tuned!!!!

Day 2 VMware Cloud on AWS SDDC Scale Up…in Four Clicks or Keystrokes!!!!!

As customers continue to build their cloud strategy with a combination of VMware products and services, one thing has been heard loud and clear…”Make Day 2 Operations easy!” As customers continue to move and increase their footprint in VMWonAWS, the SDDC’s demand for management resources will increase. While the VMC Sizer is a great tool to help understand the recommended size of an new SDDC, there will be times when SDDC growth is too big for the management VMs to handle after the SDDC is deployed…kind of like the time when Sheriff Brody realized he was going to need something much larger to catch a Great White shark.

When an SDDC is created, two resource pools are created. One named “Compute- ResourcePool” and one named “Mgmt-ResourcePool”. Mgmt-ResourcePool (MRP) is VMware managed and is comprised of vCenter, 2 NSX Edges, and 3 NSX Managers by default. In order ensure uptime and performance, all resources in this MRP have reservations assigned so these appliances always have what they need.

For more information, Product Manager Vish Kalsi wrote a quick blog on choosing the correct SDDC deployment. In short, medium management appliances require 34 vCPU and 116GB memory to run vCenter, NSX Manager and other management appliances. Large management appliances require 68 vCPU and 240GB memory. Large SDDCs are ideal for addressing a larger density of workloads . Large SDDCs support enhanced network throughput on the NSX Edge appliance. VMware recommends large-sized deployments with more than 30 hosts or 3000 VMs, or if the resources (CPU or memory) are oversubscribed in the management cluster.

Previously, a VMware support ticket needed to be opened in order to convert a regular aka medium SDDC to large. This method was obviously not preferred by most as this is the opposite of a self-service cloud operation model. However, begging with VMWonAWS 1.10, you can now upscale your SDDC to large with just a few clicks….or keystrokes!!!

Start by logging into the Cloud Services Portal, select your SDDC and go to Settings > SDDC > Management Appliance. You will see your SDDC as well as the “Upsize” option listed as seen below.

Upsize Option within the Cloud Services Portal

The only thing left to do is accept the addition of hosts if necessary and understand that you can never go back to a regular size SDDC. Once Upsize is selected, the process takes about 2 hours to complete and you will lose connectivity. It is recommended to do this during a maintenance window.

Once complete, the Management appliances will reflect as a “Large”

Once, in vCenter, you will see that the NSX Edges have gone from 4 CPU x 8 GB RAM to 8 CPU x 32 GB RAM and vCenter has gone from 8 CPU x 28 GB RAM to 16 CPU x 37 GB RAM (only 12 of the 16 CPUs are reserved in this configuration). You can check the before and after in the VM summary as seen below.

Regular SDDC vCenter
Large SDDC vCenter

Now that the SDDCs have been upscaled, it’s onto bigger and better things for your VMWonAWS SDDC!

Deploying a VMware Cloud on AWS SDDC End to End

For those of you who are ready to deploy your first Software Defined Data Center (SDDC) on VMware Cloud on AWS, there is a little bit more than meets the eye when it comes to the initial deployment. As a part of the VMware TAM Lab series, I demonstrate how to deploy an SDDC from start to finish, including the configuration of the VPC in AWS.

**SHAMELESS PLUG** – Subscribe to the TAM Lab YouTube channel. We are covering all VMware Technologies and use cases….including how to go about building your own home lab. Check it out!!!

Dive in!!!!! Learning all about VMware Cloud on AWS and HCX

Lately, I’ve been asked by peers and customers alike “How can I learn more about VMware Cloud on AWS?!” Many of us are finding ourselves in front of screens much more than normal these days so what better way to fill in some time gaps than by learning more about VMware Cloud on AWS and HCX?! While search engines are helpful, I hope my “definitive list” helps!!! If you need more, feel free to reach out!!! Happy Learning!!!

VMware Cloud on AWS

YouTube

VMware Cloud on AWS Customer Success YouTube Channel

VMware Cloud YouTube Channel

VMware Cloud on AWS Blogs

Nico Vibert – https://nicovibert.com/

Gilles Chekroun – http://www.gilles.cloud/

Ryan Kelly – http://www.vmtocloud.com/

William Lam – https://www.virtuallyghetto.com/

Tom Twyman – https://occasional-it.com/

Dustin Spinhirne – https://dspinhirne.github.io/vmcbook/

VMware Cloud Blog – https://cloud.vmware.com/community/blog/

Community Sites

VMware Cloud on AWS Blog Community – https://cloud.vmware.com/community/vmware-cloud-on-aws/

VMTN Forum –https://communities.vmware.com/community/vmtn/vmc-on-aws/overview

VMware Cloud on AWS VMUG- https://community.vmug.com/communities/community-home169?CommunityKey=df5b4c52-4f7b-48dc-b5ad-ea0be799e128

Documentation

VMware Cloud on AWS Sizer and Workload Profiles – https://vmc.vmware.com/sizer/workload-profiles

VMware Cloud on AWS Documentation – https://docs.vmware.com/en/VMware-Cloud-on-AWS/index.html

Configuration Maximums – https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws-operations/GUID-10A0804B-04F4-4B8A-9EBA-85169F533223.html

Getting Started – https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/vmc-on-aws-getting-started.pdf

Operations Guide – https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/vmc-aws-operations.pdf

Feature Walkthrough – https://featurewalkthrough.vmware.com/t/vmware-cloud-on-aws/

Hands on Labs

HOL-2052-01-ISM – VMware Horizon on VMware Cloud on AWS – https://labs.hol.vmware.com/HOL/catalogs/lab/6542

HOL-2087-01-HBD – VMware Cloud on AWS – Getting Started – https://labs.hol.vmware.com/HOL/catalogs/lab/6593

HOL-2087-91-HBD – VMware Cloud on AWS – Lightning Lab- https://labs.hol.vmware.com/HOL/catalogs/lab/6053

Hybrid Cloud Extension (HCX)

Blogs

Gabe Rosas – https://hcx.design/ THIS IS A ONE STOP SHOP FOR HCX!!!!!

Emad Younis – https://emadyounis.com/

Communities

https://cloud.vmware.com/community/vmware-hcx/

Documentation

HCX Overview – https://docs.vmware.com/en/VMware-HCX/services/user-guide/GUID-A7E39202-11FA-476A-A795-AB70BA821BD3.html

Hands on Labs

HOL-2081-01-HBD – VMware HCX – Getting Started with Cross-Cloud Mobility- https://labs.hol.vmware.com/HOL/catalogs/lab/6352

NSX Inventory Groups and Memberships to Manage Edge Firewall Rules in VMware Cloud on AWS

I have been using this feature of NSX heavily as of late so I wanted to highlight the power and flexibility of inventory groups that allow you to configure firewall rules that best fit your organization. The grouping of VMs and network services within VMware Cloud on AWS allows you put VMs that have common characteristics such as databases, web servers, operating systems, IP addresses and tags together to more easily publish firewall rules. By default, VMware Cloud on AWS creates two primary management groups and a service inventory when the SDDC is created. The two “parent groups” as I will call them are the Management Group and Workload Group respectively. The Management groups are system defined groups of infrastructure components such as ESX hosts, vCenter, NSX Manager or any other management appliances such as HCX. Workload groups are user defined groups of Virtual Machines or IP addresses.

Network and Security Inventory Groups

As you can see above, I added sub groups to the Workload Group based on function (Web, SQL, etc.). Recently, I have been testing the Membership Criteria option as this feature leverages tags and can also group based on Virtual Machine names that contain or start with a defined string. To add a new group, select Add Group, name your group, and then choose one of three membership types, Virtual Machine, IP Address or Membership Criteria. For this example, I used Membership Criteria. Once Membership Criteria is selected, pick the member VMs based on one of the two criteria (VM Name or Tag). For VM name you can choose contains or equals to categorize your VM grouping. The tag criteria can only be leveraged by having the tag name be equal to the VM(s) that is defined. In order to leverage the tag membership criteria, the the VM must be already tagged.

**Public Service Announcement** vCenter Tags and Attributes are NOT manifest in Networking & Security in the SDDC. These tags can only be added via Virtual Machines under Networking & Security > Inventory > Groups > Virtual Machines. Right click on the VM, select Edit and add your tag.

For the sake of this blog, I have created several VMs with different names but the same tag to show how tagging is leveraged.

VMs with their respective tags

Once VMs are tagged correctly, verify tagging is working by going to the Workload Groups by selecting the group > View Members.

Web Server members based on tags.

Now that VMs are properly grouped and tagged, Compute Gateway rules can now be configured and published. Go to Networking & Security > Compute Gateway > Add New Rule > Name the rule and select your source and destination. You should see your newly created group in the selection.

To verify that the rule is applied correctly, go to Networking & Security > Inventory > Groups > Workload Groups > Select the group that was added to the edge firewall rule > Select View Reference.

Below is a demo showing how VMs (tagged “Web”) on different Network segments can all access the internet with one rule.

Refer to the Networking & Security Guide what was updated on January 27,2020. Lastly, if you want to see how to leverage the Distributed Firewall (DFW) to protect 3 Tier Applications be sure to check out Michael Armstrong’s latest blog!!!