ESXi 5 nested install

Here is my LAB:

1. GA-970A-D3 MB with built-in Realtek NIC chipset and IOMMU support (no video - I used some old PCI video card), AMD 6 core 3.2 GHz CPU, 16 GB of GSKILL RAM, 400 w SPI power supply (I had some generic ATX case already), no HDDs - booting from some old 1GB USB stick ; bought in Memory Express (www.memoryexpress.com) for $426 including taxes. Not so bad.

							PRICE MATCH
Description			Qty	Unit Price	Price	Qty	Unit Price	Price
Gigabyte GA-970A-D3 w/ Dual DDR3 1866, 7.1 Audio, Gigabit Lan, PCI-E, CrossFireX			1	$109.00	$109.00	1	$103.74	$103.74
AMD Phenom™ II X6 1090T Black Edition 3.2GHz w/ 9MB Cache			1	$179.99	$179.99	1	$167.12	$167.12
GSKILL Ripjaws X Series 8GB PC3-10666 Dual Channel DDR3 Kit (2 x 4GB)			2	$49.99	$99.98	2	$47.44	$94.88
SPI ATX-400PN 400 Watts ATX12V 2.01 Power Supply w/ 120mm Fan, Noise Killer			1	$39.99	$39.99	1	$39.99	$39.99
				Subtotal	$428.96		Subtotal	$405.73
				Taxes	$21.45		Taxes	$20.29
				Total	$450.41		Total	$426.01

2. Shared NFS storage on a Linux box.


Installation/configuration process

• Install and configure ESXi 5 as usual.
• Configure vSwitch (or VM Network Port Group) with Security Policy Exception set to Promiscuous Mode - Accept  (this is very important, otherwise you'll get network issues for your nested VMs - e.g. in particular they would be able to get DHCP address, UDP and some other protocols like SIP etc would work but there won't be any TCP connectivity!)
• If you want to use VLANs in nested hypervisors, create additional port group, name it e.g Trunk and set VLAN ID to 4095
• Enable Hardware virtualization on physical ESXi 5 host - add vhv.allow = "TRUE"  to /etc/vmware/config of your physical ESXi 5.0 host using vi or simply run

  echo 'vhv.allow = "TRUE"' >> /etc/vmware/config 

• Create virtual ESXi 5 on your physical ESXi 5 host, use version 8 hardware and Red Hat 5 64bit OS, make sure the SCSI adapter is LSI Logic Parallel (default), assign 1 NIC from regular VM network ports and several NICs from Trunk port group.
• Edit settings of the newly created ESXi 5 VM - change OS type in "Options" to Other-> ESXi 5.0
• Select "Intel EPT/AMD RVI for MMU virtualization" in Options -> CPU/MMU virualization
• Attach the ESXi 5 iso CD image, power on and install as usual.
• If you have only 1 shared datastore and use HA cluster you'll receive the following error when you try to start a VM in cluster : "The operation is not allowed in the current state" or something like that. You'll also get the following warning message in the summary tab on the hosts: "The number of heartbeat datastores for host is 1, which is less than required: 2". This is because new advanced HA was introduced in ESXi 5 which relies on datastore heartbeat. Default numbers of hearbeat datastores is 2 (could be changed from 2 to 5). To disable sufficient datastore condition checking go to Cluster HA advanced options, add the following setting: das.ignoreinsufficienthbdatastore  and set the value to be True if you want, on the other hand, to change the number of heartbeat datastores required, add the following setting: das.heartbeatdsperhost and set the value to be anything from 2 (default) to 5