Native AWS VPC Connecting to VMware Cloud on AWS (Part 2) – Route Tables and Transit Gateway Attachments

Welcome to Part 2 of VPN connectivity to VMC via AWS VPN. For this post, I will walk through how to attach an AWS VPC to an AWS Transit Gateway in order for route tables to be learned by the VMC SDDC and AWS VPC that will ultimately allow communication from workloads in AWS and workloads in VMC. If you immediately try to ping across the VPN from an EC2 instance to a VMC VM, it will fail as there is more work to do in order to create communication pathways in the form of adding a VPC to the Transit Gateway as well as configure some NSX rules.

In the diagram above I created two EC2 instances with an AWS VPC. For the sake of testing and validation, I first had to create a jump VM with access to the internet in order to remote into the workload VM. This is because, in order for the workload VM to ping across the VPN, 0.0.0.0/0 must be routed through the Transit Gateway and not an Internet Gateway. The goal here is to have a VM on the 10.0.10.32/27 subnet ping a VM on the 10.250.10.0/24 subnet.

To start, the first thing to check is the routing table for the Transit Gateway. This is done by selecting “Transit Gateway Route Tables” on the sidebar with the VPC.

You will then see a list of learned routes from the VMC SDDC. For the sake of this blog, I have deployed AWS EC2 Instances on a subnet that I plan on attaching to the Transit Gateway (10.0.10.0/24). However, notice below that only the learned routes for the VPN to VMC are listed below. While the 10.250.10/0/24 subnet is learned, in order to open communication via the AWS VPC, we first need to attach the VPC to the Transit Gateway so the 10.0.10.0/27 subnet is included. Here are the steps to create the VPC attachment to the Transit Gateway.

Step 1: Select “Transit Gateway Attachments” then “Create Transit Gateway Attachment”. Find correct Transit Gateway ID.

Step 2: Add tag for attachment name.

Step 3: Select the correct VPC and ensure you have the correct subnet selected. Select “Create Attachment”

As you can see, now that the VPC is attached to the TGW, the 10.0.10.0/24 subnet is now listed. This will now allow for the routing of traffic from my EC2 instance to my VM on VMC.

The final step is to create groups and add inbound and outbound firewall rules within the Network & Security section of the SDDC console. Make sure to apply the rule to the VPN Tunnel Interface and NOT “All Uplinks“. The reason for this is that a firewall rule applied to All Uplinks does not apply to the VPN Tunnel Interface (VTI), which is a virtual interface and not a physical uplink. The VPN Tunnel Interface must be specified explicitly in the Applied To parameter of any firewall rule that manages workload VM communications over a route-based VPN. In addition, you can see how groups are created on an earlier blog here.

Now that the routing table is complete and Compute Gateway Firewall rules are in place, you can test!

EC2 Windows VM

Ping results from EC2 VM to VMC VM

Advertisement

Day 2 VMware Cloud on AWS SDDC Scale Up…in Four Clicks or Keystrokes!!!!!

As customers continue to build their cloud strategy with a combination of VMware products and services, one thing has been heard loud and clear…”Make Day 2 Operations easy!” As customers continue to move and increase their footprint in VMWonAWS, the SDDC’s demand for management resources will increase. While the VMC Sizer is a great tool to help understand the recommended size of an new SDDC, there will be times when SDDC growth is too big for the management VMs to handle after the SDDC is deployed…kind of like the time when Sheriff Brody realized he was going to need something much larger to catch a Great White shark.

When an SDDC is created, two resource pools are created. One named “Compute- ResourcePool” and one named “Mgmt-ResourcePool”. Mgmt-ResourcePool (MRP) is VMware managed and is comprised of vCenter, 2 NSX Edges, and 3 NSX Managers by default. In order ensure uptime and performance, all resources in this MRP have reservations assigned so these appliances always have what they need.

For more information, Product Manager Vish Kalsi wrote a quick blog on choosing the correct SDDC deployment. In short, medium management appliances require 34 vCPU and 116GB memory to run vCenter, NSX Manager and other management appliances. Large management appliances require 68 vCPU and 240GB memory. Large SDDCs are ideal for addressing a larger density of workloads . Large SDDCs support enhanced network throughput on the NSX Edge appliance. VMware recommends large-sized deployments with more than 30 hosts or 3000 VMs, or if the resources (CPU or memory) are oversubscribed in the management cluster.

Previously, a VMware support ticket needed to be opened in order to convert a regular aka medium SDDC to large. This method was obviously not preferred by most as this is the opposite of a self-service cloud operation model. However, begging with VMWonAWS 1.10, you can now upscale your SDDC to large with just a few clicks….or keystrokes!!!

Start by logging into the Cloud Services Portal, select your SDDC and go to Settings > SDDC > Management Appliance. You will see your SDDC as well as the “Upsize” option listed as seen below.

Upsize Option within the Cloud Services Portal

The only thing left to do is accept the addition of hosts if necessary and understand that you can never go back to a regular size SDDC. Once Upsize is selected, the process takes about 2 hours to complete and you will lose connectivity. It is recommended to do this during a maintenance window.

Once complete, the Management appliances will reflect as a “Large”

Once, in vCenter, you will see that the NSX Edges have gone from 4 CPU x 8 GB RAM to 8 CPU x 32 GB RAM and vCenter has gone from 8 CPU x 28 GB RAM to 16 CPU x 37 GB RAM (only 12 of the 16 CPUs are reserved in this configuration). You can check the before and after in the VM summary as seen below.

Regular SDDC vCenter

Large SDDC vCenter

Now that the SDDCs have been upscaled, it’s onto bigger and better things for your VMWonAWS SDDC!

Effectively Planning VMware Cloud on AWS SDDC Upgrades

One of the questions I am often asked is now that I am using VMware Cloud on AWS, how do I go about managing my SDDC life cycle? The answer…..VMware has you covered! As of March 2020, we have made some significant enhancements to the Notification Gateway (NGW) that give you several options to receive updates from VMware Cloud Services regarding maintenance activities such as certificate replacements and SDDC upgrades to new releases. While the NGW can be leveraged in several different areas, my preferred integrations are with Slack and Microsoft Teams. Setting up these integrations are fairly straightforward. Look no further than William Lam’s blog for details.

Even if you have Webhook integrations setup, you will still get a notification email similar to the image below letting you know when your SDDC is scheduled for an upgrade.

Notification email from Notification Gateway detailing each phase of the SDDC upgrade.

It is imperative that you take note of the dates and times your SDDC is scheduled for each phase as your times will all be in UTC timezone so do your time conversions accordingly. When you login to your SDDC console and go to the maintenance tab and you will see each phase listed along with recommendations for each phase.

Each phase of the SDDC is highlighted below as well as details around SDDC accessibility during the upgrade. For detailed information, read my associate Tom Twyman’s blog and the SDDC upgrade notes found here. We continue to improve upgrade processes in the background so check back often!! There are additional considerations to make when integrating with HCX, Site Recovery and Horizon so be sure to understand the impacts listed in the read me!! Keep in mind that during Phase 1 your vCenter certificate will be updated and the NSX certificate will be updated during Phase 3. If you have other products and services that depend on vCenter, you will need to take the proper steps to accept the new certs.

While there are time estimates for each phase, mileage may vary during the upgrade. To make things a bit easier for you. I have included a simple excel spread sheet to help you plan your SDDC upgrade.

sddcupgradescheduleDownload

After going through several customer upgrades over the past two years, my top 5 things to do are

1. Don’t forget about certificate validation afterwards!
2. Plan your outages around each phase and best to be conservative. Allot for the full estimated time.
3. Setup integrations with the NGW. While emails are nice, it has been my observation that people get too many emails these days and these notifications are often ignored. Pick a delivery method that will get your attention!
4. Read the release notes as well as upgrade notes before your scheduled upgrade.
5. Don’t panic! For some, giving VMware the keys to the car (SDDC) is unnerving, and they want to watch and be involved. Remember this is a service, we have you covered. Sit back and relax!

VMware Cloud on AWS – SDDC to SDDC VPN and MS AD Replication

This is part two of my blog on how to leverage Microsoft Active Directory as an Identity Source and have AD replicate between two VMware Cloud on AWS SDDCs. Now that I have Active Directory running in US East, I will setup a route based VPN between my US East SDDC and US West SDDC. For my lab, I am using a Route Based VPN to replicate Active Directory Traffic. To add Route Based VPNs to both SDDCs, take note of your SDDC Public IPs on your Management Networks, determine what you want your Autonomous System Numbers (ASN) to be, and determine your IPs for both BGP local IPs. To keep the BGP IP scheme simple, I chose 169.254.x.x/30 to only allow for two available IP addresses. FYI, There are two different number ranges for Public and Private ASN numbers. Public is 1-64,511 and Private is 64,512-65,535. Route based VPN makes things simple in this scenario since we are leveraging Border Gateway Protocol (BGP) where both SDDCs are able to exchange routes and leverage BGP peering. For a deeper dive into BGP peering specifically around AWS Direct Connect and VMware Cloud on AWS, check out Nico Vibert’s Blog. It will not disappoint!

SDDC Public IP Info

Once you have the ASN and SDDC Public IP information, you can add your route based VPN by going to Networking & Security tab -> Network -> VPN -> Route Based -> “Add VPN”. For my lab, I have kept all the defaults for the tunnel and IKE settings. You may need to make changes here based on your security requirements. You must, however, select a pre shared key that will be used for both VPN connections to establish a secure connection. I have also left the Remote Private IP field blank. Once you click “Save”, you will see the status of the VPN and BGP Remote IP go to a yellow status as the negotiations take place. If successful, you should see both Remote BGP IP and VPN status turn green.

SDDC to SDDC VPN once completed

The next step in the process is to deploy a second Domain controller inside the second SDDC. Before you can promote the second DC, you need to first deploy a Windows Server VM in SDDC #2. Once the VM is deployed, you will then need to establish two-way communication across the VPN tunnel to be able to add the Windows Server to the domain and promote it. Although the VPN is up, you still need to configure additional Gateway Firewall rules in order for Domain Controllers to talk to each other across networks. Go back to Networking & Security -> Security -> Gateway Firewall -> Compute Gateway -> Add New Rule. For two-way communication, add two rules that allow traffic to and from the Domain Controller. Make sure that you have this traffic go over the VPN Tunnel Interface and NOT the Internet Interface. Make these rules for both SDDCs.

Gateway Firewall Rules for VPN Tunnel Interface

Before promoting your soon-to-be Domain Controller, make sure you can ping across the VPN via IP and DNS FQDN. The next step in the process is to deploy a second Domain Controller inside SDDC #2. I will not go through the process in this blog but the steps are similar to setting up the first DC in that you need to promote the server to a Domain Controller. There are several blogs out there on how to do this but here’s one just in case. Once added, you can verify Active Directory is syncing across SDDCs and Domain Controllers by running “repadmin /replsummary” via the Command Prompt. You can now add users, GPOs, etc to either side and both SDDCs will have the same info. To take things even further, add your new Domain Controller as an identity source to the new SDDC. This will allow users to login to either vCenter as long as they have an account on the domain. If you missed my blog on setting up AD as an identity source with VMWonAWS, click here.

Bye Bye Spreadsheets! Hello (New) VMCSizer!!!! Part 2

In a previous blog, I highlighted Workload profiles and how they should be used in right sizing your VMWonAWS environment. Since my last blog, the sizer has been updated not only with a new URL but with several new features. One of which is that you can now choose either i3 or R5 instances depending on your workload needs. You will notice that when you select an r5 instance, you are automatically assigned 15 TB of AWS Elastic Block Storage (EBS) aka Elastic VSAN. For more information regarding Elastic VSAN, click here.

r5 instance type

Similar to the previous version, you will be able to see the results of your workload inputs. Another new feature is ribbon across the top that allows you get into the data!! Information is key when sizing your environment and this section of the sizer gives you everything you need.

Recommendation buttons that allow you to go deep into your data inputs and results

As a part of the recommendation, you can see below that the sizer has identified my SDDC to be storage bound due to my storage requirements. This gives me a good idea where I will need to grow going forward.

SDDC Recommendation Dashboard

With the continued interest and adoption of VMware Cloud on AWS come two topics that always come to the forefront once you get passed how cool it is…..HOW MUCH DO I NEED? and HOW MUCH IS IT GOING TO COST?! To get the full picture, you will need to capture the details of your environment. There are several tools available and luckily enough, Bill Roth from VMware highlighted these tools in a blog a few weeks ago. In addition to his mention of RVTools, which is very popular, I would also encourage you to reach out to your….shameless plug…VMware Technical Account Manager. They have an additional toolset that can help you right-size the environment. Take a test drive and size today!!

VMware Cloud on AWS – Identity Access with Microsoft AD

For years I have been Window (see what I did there) shopping Intel NUC, HP Microservers, Mac minis, and others to setup my home lab v 2.0. However, with the onslaught of Cloud Services, I continue to balk at the thought of dropping thousands of dollars every few years for new hardware, as well as the electric bill and management overhead that comes with it. With VMware Cloud on AWS, I wanted to see how I could create a lab environment and continue to use Active Directory for vCenter authentication. Due to not having an vCenter on prem, Hybrid Linked Mode (HLM) wasn’t an option for an identity source. VMware has existing documentation where you can see the options for Identity Sources. This blog will walk you through the setup and configuration steps I took to get AD working within VMWonAWS vCenter. Like with all things in Public Cloud, it’s critical to have your networking straight before you begin adding subnets, etc.

• Create your subnet via SDDC > Networking & Security > Network > Segments > Add Segment

• Login to vCenter with the cloudadmin account.  We can see the network segment is added in vCenter. Note that we cannot add networks from vCenter. We must use the SDDC Console to add logical networks

Networking View from vCenter

One of the great things about vSphere 6.7 and later is the additional functionality built into the Content Library. I have already loaded several OVF Templates and will deploy my Domain Controller from a Win2016 Std OVF template. For more content library goodness, check out William Lam’s blog here. I’m a huge fan and I recommend you use Content Libraries!!

During OVF deployment, place the VM on the correct network

With the Network Segment selected and IP assigned, the new Domain Controller will be able to communicate with the SDDC vCenter after a few more configurations.

Now that we have the DC on the proper network segment, we need to allow traffic to flow between the SDDC Management Gateway and the DC. To do this we need to create a Management Group. This is done by going to the SDDC Console > Networking & Security > Inventory > Groups > Management Group > Add Group. Add your domain controller to the Management Group with its assigned IP.

Once the Management Group assignment has been configured, we can now add a Gateway firewall rule to allow the domain controller to talk to the SDDC vCenter. To enable communication, go to SDDC Console > Networking & Security > Gateway Firewall > Management Gateway > Add New Rule. This is where adding the user defined group comes into play as we need to be able to select the group to add as the destination for the firewall rule.

We now need to allow communication via the DNS settings on the management gateway. We must remove the default DNS settings and add the domain controller(s) IPs so LDAP/AD can communicate with the SDDC vCenter. If we don’t change the IPs from default, we will get an LDAP error that the URL cannot be reached. Here’s a video that ties together the final piece of adding the DNS server and assigning the GlobalCloudAdmin role to the user I want to login to vCenter with the s2c.local domain credentials. In addition, you can read Nico Vibert’s blog that shows how to use AWS Directory Services as an identity source. Enjoy!!

AWS re:Invent 2018 – Day 4

Last full day at re:Invent for me but it ended on a really good note. The morning was spent attending Werner Volgels’ Keynote that covered new database services, serverless, and more! I highly recommend watching.

The next session I attended put me on my heels. My background is in systems administration and operations. I am not a developer but my main goal in attending re:Invent was to stretch myself and learn more about what Andy Jassy refers to as “builders”. I believe that Artificial Intelligence (AI) and Machine Learning (ML) are going to be major disruptors in all industries so I jumped at the chance to learn more about them. I attended a session on the newly announced AWS Deep Racer. This was a 2.5 hour workshop where I learned about Reinforcement Learning (RL). This is the main type of machine learning behind Deep Racer. The standby line to get into the session was at least 100 people so I’m lucky I pre-registered for this one. This session was attended by developers, robotic specialists, ML scientists, and those who simply wanted to learn more about AI. The surprise of the session was that each of us was given a Deep Racer for attending!!! The irony was that we had to pick up the car and then take it to the FedEx store to have it shipped to our homes if we didn’t want to carry it on the plane. I’m pretty sure AWS could have leveraged someone who’s really good at shipping things to my door….but who cares….I got one!!!!

My last session for the conference ended on a high note. ENT215-R1 – Top Strategic Priorities You Can Tackle with VMware Cloud on AWS. With yesterday’s announcement of AWS Outposts, this was a highly attended session. Well-known VMware technologists such as William Lam, Kyle Ruddy, Emad Younis, and Alan Renouf were all in attendance. AWS VP Sandy Carter and VMware VP Mark Lohmeyer along with Emad covered more uses cases for VMWonAWS and introduced AWS Outposts. This is a must watch if you are interested in Hybrid Cloud.

After sending my Deep Racer off for home delivery, it was time for some R&R at the hotel before re:Play. Re:Play is the party held on the last night of the conference. The only time I have seen so many people in tight spaces have been at major sporting events or amusement parks. PEOPLE EVERYWHERE!! Even the line for the men’s restroom was insane! The laser show and dodgeball were entertaining. It was great to see all the excitement after a long week of sessions. After about an hour of bumping into people, I decided to call it a night. Day 4 = 20,545 steps (10.18 miles)

 

AWS re:Invent 2018 – Day 3

Day 3 I attended a breakfast to celebrate the great things that VMware and CloudHealth are doing with our partners and customers. I’m excited about the multi-cloud functions the service has and how it will help customers get their arms around better managing their public cloud instances from security to costs. Here’s a link to VMware CEO Pat Gelsinger and CloudHealth CEO Tom Axbey discussing the acquisition and strategy going forward. During breakfast, we watched the live steam of Andy Jassy’s keynote. The next 2.5 hours of announcements were announced at an insane pace as I struggled to keep track. Once Andy started telling the story of Hybrid Cloud, I knew something cool was coming. Low and behold, Pat Gelsinger (VMware CEO) joins him on stage to announce AWS Outposts!!! There are so many exciting things about this announcement. In a nutshell, we are letting users choose between on-premises servers and storage, which can be ordered in quarter, half, and full rack units. AWS Outposts can be upgraded with the latest hardware and next-generation instances to run all native AWS and VMware applications. A second version, VMware Cloud on AWS Outposts, will let customers use the VMware control plane and APIs to run the hybrid environment. After the keynote, I headed back to the Expo Hall to see what kind of attention the AWS Outposts message was getting and it was fairly packed! There’s a lot of interest around this technology. Very exciting! I spend a few hours there talking to several other VMware attendees at our booth and on the floor. It was awesome to see all the customer meetings. VMware and AWS are going to continue to innovate together, that much is clear.

My last session of the day was ENT313-S Running Production Workloads in VMware Cloud on AWS. VCSA and Hybrid Cloud Extension (HCX) all-pro Emad Younis and VMWonAWS Director Alex Jauch presented. Alex and Emad focused on the deep partnership between VMware and AWS that makes this service possible. If you want to know more about use cases, how the service is built, and how to quickly migrate workloads between on-prem and VMWonAWS, look no further than this session.

Day 3 = 14,509 steps (7.18 miles)

 

AWS re:Invent 2018 – Day 0

Being my first year at re:Invent I wanted to give my insights regarding the conference. First and foremost….like most conferences this size….WEAR COMFORTABLE SHOES! With 50K+ attendees spread across 5 different venues up and down the strip, you will definitely hit your step goals for the week. Day 0 count = 16,230 (8.03 miles).

Check-In 

Compared to some conferences I have been to, AWS pulled registration off beautifully by allowing you to register at Terminals 1 and 3 at McCarran International Airport plus the Aria and Venetian. I only had to wait about 5 minutes at the Aria. The SWAG pick up at the Venetian was a snap and there was even a place to try on the famous AWS re:Invent hoodie beforehand. No more guessing on the fit.

Midnight Madness

Next, it was back to the hotel for some rest before the night’s activities. I can attest that while at the conference, use the shuttles!!! Saves your legs and feet, you may even have some interesting conversations with others along the way. Once back at the Venetian, I waited in line for what I thought was going to be a cheap easy way to get a free meal…..the Tetonka Challenge! AWS re:invent was at it again with trying to break last year’s Guinness World Record for the largest chicken wing eating competition.  400+  waited in line to compete to see who would take home the crown. I met some great people as I waited in line but I am sorry to say that after only 22 wings I bowed out. Something about soggy-ish wings didn’t quite hit the spot. The winner ate 70+ which is absolutely insane! I even got my one second of fame. You can see me standing behind the man in the green jacket off to the left when they awarded the winner.

Coupled with the Tatonka Challenge were Portlandia’s Carrie Brownstein and Fred Armisen’s best attempt at live comedy. In my opinion, it fell pretty flat but at least they tried. They were the on-stage cheerleaders for two more world record attempts in the largest Air Drum Ensemble (Phil Collins’ “In the Air Tonight” was selected for the Air Drumming) and Most People Lighting Glow Sticks Simultaneously. I have no idea if we broke all three but I think we did!! It was a good time but if it weren’t for the Tetonka Challenge I would not have missed much by not attending. Let’s see what Day 1 brings!! All I know is that I will not be eating wings for the rest of the week!!!

 

VMware Cloud on AWS Connection Options

Happy New Year!!! This is going to be an exciting year for VMware Cloud on AWS and I wanted to kick off 2018 by highlighting the way in which you are going to connect into and out of VMware Cloud on AWS.

First of all, VMware Cloud on AWS is optimized (VMware Cloud Foundation) to run on dedicated, elastic bare metal infrastructure at a very high level inside Amazon’s data centers. For security purposes, the VMware Cloud on AWS SDCC is bifurcated to the components that manage the SDDC itself such as ESXi, VSAN, NSX, and vCenter.

Here’s a simple explanation of how you can setup the connectivity framework.

The first thing you need to setup is a connection to the management components of the SDDC.  You will first need to create a Management VPN and choose a set range of IP addresses that will be used by management components such as the ESXi hosts and vCenter. This range will be in the form of a simple CIDR block. We recommend using a /20 CIDR block for management purposes. After you connect the management portion of the SDDC, you will then need to setup an IPSec VPN between your on-prem data center and management components. This VPN can be setup over the Internet or AWS Direct Connect (DirectX). After this connection is established, you can then build firewall rules on the VMware Cloud on AWS Console. With these rules you can control access to the  vCenter from your on-prem data center.

There is an optional connection you can setup if you need access to your vCenter Server directly from the Internet. A public IP is automatically provided during the provisioning process. It is important to note that all access to this IP is restricted. To provide access, you will need to configure firewall rules in the VMware Cloud on AWS console to allow this direct type of Internet access.

The second VPN you will need to setup is between your compute workloads and your on-premise data center. Several logical networks are required to provide the IP addresses for the workloads you plan on migrating or build in VMware Cloud on AWS. This VPN secures these workloads and allows them to connect back to your on-prem data center. This can be an IPSec VPN or L2VPN. The L2VPN advantage is that you can stretch a single L3 IP space from on-prem to the cloud and is also required for live migrations. This VPN can go over the Internet or AWS DirectX. You can again create firewall rules as needed to access on-prem workloads.

The next connection is between your SDDC workloads and your Amazon VPC. This is automatically configured and built during the SDDC provisioning process. Once you select the Amazon VPC subnet that will be associated with your VMware Cloud on AWS SDDC an elastic network interface (ENI) will be created allowing traffic to flow between both environments.  In order to control security, you will need to configure AWS IAM policies as well as firewall rules on the VMware Cloud on AWS side to allow access between both. Lastly, you will likely need to give direct public internet access to some of your SDDC workloads. To make these accessible to the Internet, you will need to leverage AWS elastic IPs along with NAT and firewall configurations to allow this type of access.

That’s it! Now you are ready to leverage your SDDC on VMware Cloud on AWS!

Also, here’s a video that covers the content discussed above.

-SL