Welcome to my first blog post on my VMware Nested Home lab series.

VMware Nested Home Lab Series

• VMware Nested Home Lab Part 1 - Overview
• VMware Nested Home Lab Part 2 - Management and Workload Domain Design

Overview

Whether you work in support, engineering, consulting or are simply an enthusiast, homelabs have always been a brilliant way of troubleshooting and keeping up to date with new technologies.

Compared to your work, or customer’s production environment, homelabs are a safe space, which is there to be broken at any given time. This allows you to understand the consequences of your actions and enables you to learn to fix the issue. I have used my homelab for many scenarios, such as working on automation, integrating with new products, simulating an ongoing issue in a client’s production, and most common of all, keeping the environment up to date with new features, patches/upgrades.

Homelabs can be costly, but with the right budget set in mind and an understanding of what you would like to achieve, it can be affordable. There are many great blogs out there written by individuals with different setups ranging from a single server to a full-blown home rack-mounted data center. As I won’t be going into detail on what hardware choices are available, I would strongly recommend checking out William Lam’s homelab series for more details. https://williamlam.com/home-lab.

Instead, I’ll go through what my setup looks like, and why I believe this works just fine for me.

Homelab Hardware

When I first started off wanting to invest in a homelab, after a bit of googling, there was so much hype regarding Intel NUCs. Each NUC could be spec’d up to 64GB RAM, and the CPU is pretty decent, depending on how deep your pockets go. They were small form factor, taking up less space, and weren’t loud, meaning I would not get an earful from my wife.

After owning 3 x i5 Intel NUCs, each with 64 GB RAM, 1TB NVMe and 1TB SSD, I realised the CPU was becoming a bottleneck when running multiple resource-intensive appliances. The other limitation was the physical networking kit, lab VLANs would need to be trunked on the physical switch for packets to transverse between the physical servers. Finally, if your lab nested ESXi hypervisors were participating in a vSAN cluster, and were spread across the multiple physical hosts, then your traffic would be limited to 1Gbit, unless you upgrade your network kit to 10Gbit. I still love the Intel NUCs, I currently use them as my dev kit for my homelab automation, which hopefully I will cover soon.

Following some careful consideration, and saving, I decided it was worth investing in a single physical ESXi hypervisor, which would allow me to run at least 2-3 labs (workload domains) at the same time.

So, here is my Bill of Materials (BOM)

Physical Server

The above BOM allows room for expansion for more RAM, a further 4 x 64GB can be added, providing a total of 512GB. For now, 256GB is sufficient. 12TB of HDD allows me to archive my labs when not needed in the short term, where live labs will be running on a mixture of NVMe and SDD drives.

Network Gear

The router and switch are personal choices. In this case, the switch does not matter much as a single physical server will be used. However, I have chosen the USG as my home router as it can configure BGP dynamic routing and DNS forwarding. Allowing me to peer with my lab-specific vyOS routers, saving me from creating static routes for each lab network.

Nested Homelab Design

The meaning of a “nested homelab” is where we nest a VMware ESXi Hypervisor as a Virtual Machine within a physical ESXi Hypervisor server. For that reason, this is a very popular choice for all “homelabers”, where one could build a very resource-heavy physical server, load VMware ESXi hypervisor as the Operating System, and deploy many virtual nested VMware ESXi hypervisors for a particular lab. This nested layer could then contain one or more virtual machines.

The homelab consists of the following physical components:

• Home router - to create the management domain networks, static/dynamic routes towards homelab environment.
• Laptop - to install, configure and manage the homelab environment.
• Server running VMware ESXi Hypervisor - to host the management and nested workload domain of the homelab environment.

There are two common ways of creating VMware nested homelabs.

Option 1

Each lab VMware appliances run within its lab nested VMware ESXi hypervisor VMs (Layer 2 - Nested Hypervisor).

Option 2

Each lab VMware appliances run within the management VMware ESXi hypervisor physical server (Layer 1 - Physical Hypervisor)

Design Choice

After testing both of the above options, I have opted to use Option 2 for the majority of all lab deployments. Unless I am deploying a VCF lab, then option 1 would be my choice.

From experience, the reasons for option 2 follows:

Better performance, powering on and off a workload domain (lab) is much quicker, achieved by powering up all the lab VMs hosted on the physical ESXi hypervisor, directly from your management domain vCenter. Take into consideration that VMware appliances such as vCenter Server Appliance is resource hungry, running that on a nested ESXi hypervisor VM could have a performance impact if not sized correctly.

This leads to the next reason, better resource and capacity management. When spinning up multiple labs, for different scenarios, whether you are creating a lab that consists of the following products; VMware NSX-T lab, VMware NSXALB (AVI) lab, VMware Aria, VMware Cloud Director, etc. Careful resource and capacity planning are required when sizing your VMware nested ESXi VMs if using Option 1. Therefore, with Option 2, only the bare minimum resources can be allocated to accommodate multiple VMware NSX-T Edges, lightweight workload VMs and vSAN. Leaving the resource and capacity management of the Lab VMware appliance VMs to the physical ESXi hypervisor server. This would allow for quicker planning and deployment, and the ability to run multiple labs in parallel, of course depending on the physical resources available.

Management and Workload Separation, similar to some production environments, the management VMs, whether for the Management Domain, or Workload Domain, are hosted on the same cluster of ESXi hosts, in this case, a physical ESXi hypervisor. Each lab is therefore considered, as a Workload Domain, which would host the nested NSX-T Edges and workload VMs which are used for testing. This can be easily managed within the Management vCenter using folders, with the use of careful naming conventions.

vSAN, the easiest storage solution for nested ESXi hypervisors. Unfortunately the more resources you give ESXi, the more vSAN likes to consume. In a homelab, resources are precious, unless you have deep pockets. This ties back to the better resource and capacity management point, option 2 would avoid wasted resource allocation. Many “homelabers” have battle scars from vSAN, a great product in production, but when trying to stretch your resources to the maximum in a homelab, could lead to inaccessible vSAN objects. Therefore, as with Option 2, leaving the lab VMware appliances out of the nested vSAN hypervisors would save a lot of time and frustration. I sure would rather prefer to rebuild the vSAN cluster compared to rebuilding the whole lab.

Summary

In this article, we have discussed the use cases of owning a homelab, my choice of hardware and the different designs for a nested homelab.

The next part of the series will cover in detail the design for the management and workload domain before we set off to deploy the homelab.

VMware Nested Home Lab Series

• VMware Nested Home Lab Part 1 - Overview
• VMware Nested Home Lab Part 2 - Management and Workload Domain Design