Setting up ESXi 5.1 on a 2011 Mac Mini Server

One thing I’ve wanted to do for a while is virtualizing my home server setup, as well as making it easier to stand up (and take down) test servers as needed.

I’ve been doing a lot of work with VMWare Fusion on my Mac and could have gone that way, but I wanted to do the virtualization with VMWare’s free ESXi software. I hadn’t previously set up a dedicated hypervisor, so I wanted to learn how to do that.

I have a 2011 Mac Mini Server, which is fortunate because setting up ESXi on that Mini model has been well-documented in a number of places. The latest available version as of this date is ESXi 5.1, so I decided to install that. After some work, I now have ESX 5.1 running on my Mini Server. See below the jump for the details.

One thing to keep in mind is that ESXi is not officially supported on the Mac Mini. It works, but getting it running will take some work. Specifically, you will need to add the following ethernet driver to the ESXi install ISO file:

VMware ESXi 5.0 Driver CD for Broadcom NetXtreme I Gigabit Ethernet including support for 5717/5718/5719/5720

The way I found out that this driver is needed is that I first tried installing the stock ESXi 5.1 install without slipstreaming the driver into the installer ISO. What happened? Everything installed fine, but networking didn’t work.

Building the ESXi installer

For the second round, I started up a Windows 7 VM and downloaded a copy of ESXi-Customizer  as well as a copy of the following files from VMWare’s site:

VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso

VMware ESXi 5.0 Driver CD for Broadcom NetXtreme I Gigabit Ethernet including support for 5717/5718/5719/5720

Once I had the VMWare installer files downloaded, I then used ESXi-Customizer to slipstream using the following procedure:

1. Launched ESXi-Customizer by double-clicking on the ESXi-Customizer batch file script.

2. In the Select the original VMWare ESXi ISO: section, selected VMware-VMvisor-Installer-5.1.0-799733.x86_64.iso

3. In the Select an OEM.tgz, a VIB file or an Offline Bundle: section, selected tg3-3.123cv50.1-offline_bundle-841079.zip (corresponds to the Broadcom ethernet driver.)

4. In the Select the working directory (needs to be on a local NTFS-formatted drive): section, selected the Documents directory of my account.

5. Checked off the Create (U)EFI-bootable ISO (ESXi 5.0 only) checkbox, in order to enable the new ISO to boot the Mini.

6. Clicked the Run! button.

I received a warning that the (U)EFI-bootable ISO could not be further customized. That was fine, so I clicked the OK button.

ESXi-Customizer went to work and prompted me about replacing the ethernet driver. I wanted to do this, so I clicked the Yes button.

Once finished, the newly-created ISO was available in my account’s Documents directory as an ISO file named ESXi-5.x-Custom.iso.

My next action was to copy the ISO file from my Windows VM back into my Mac. From there, I used Disk Utility to burn the ISO file to a CD.

Installing ESXi 5.1 on the Mini

From there, I hooked up an Apple USB SuperDrive to the Mini, popped the newly-burned CD in and rebooted the Mini. When rebooting, I held down the Option key to allow the various boot drive options to appear, then selected the CD. The CD showed up with EFI Boot and Windows partitions, so I selected EFI Boot.

NOTE: All screenshots of this process are from ESXi running inside of VMWare Fusion, but the Mac Mini install process was identical.

Once booted from the CD, I was asked to select the ESXi 5.1 installer. Once selected, the ESXi installer boot process began.

1. When asked to begin the installation process, I hit the Enter key.

2. I hit F11 to accept the license agreement.

At this point, the installer took a few minutes to scan the machine.

3. Once scanning completed, I was asked to select the drive I wanted to install on. I selected the drive I wanted and hit the Enter key.

4. I selected US Default for the keyboard layout.

5. When prompted, I set a password for the root account. This is the user account that you’ll initially use to log into your ESXi server.

At this point, the installer took a few minutes to scan the machine again.

6. The installer then confirms that you want to install using the options you’ve selected. You’re also warned that the disk will be repartitioned.

NOTE: Repartitioning will wipe everything on the drive. If you have data you need to get off of this drive, hit the Escape key to back out at this point.

7. ESXi 5.1 will then install. When finished, the installer will request that you remove the installation disc and then reboot. Once you’ve removed it, hit the Enter key to reboot.

Booting ESXi 5.1 on the Mini and changing network settings

On reboot, the Mini should now boot from the newly-created ESXi boot drive. Once up at the server display screen, you should see that it’s picked up a dynamic IP from your DHCP server and has the hostname of localhost.

Here’s how to change to a static IP and a new hostname:

1. Click the F2 key.

2. Log in as the root account, using the password you set during the install process.

3. You should be now at the System Customization screen. Select Configure Management Network.

5. In the IP Configuration settings, Select Set static IP address and network configuration: and hit the space bar to make that option active.

5. Set the static IP, subnet mask and gateway address.

6. In the DNS Configuration settings, set the address(es) of your DNS server(s) and also set your desired hostname. In my case, I set a fully qualified DNS name.

7. In the Custom DNS Suffixes settings, you may need to set a DNS suffix. In my case, my domain was already filled in.

8. Once you’ve made all of your changes, hit the Escape key to exit back to the System Customization screen. You’ll be asked to confirm the changes you’ve made to your network settings and warned that there will be a brief network outage for the ESXi server and any VMs running on it. Hit the Y key to apply the new settings.

9. At the System Customization screen, you should now see your static IP and hostname displayed under Configure Management Network.

10. Hit the Escape key to exit to the server display screen. It should now be displaying your hostname and your static IP address.

Logging to your ESXi server

To log into your ESXi server, you will need to have access to a Windows VM or a Windows PC.

1. Install the vSphere client on the Windows box.

2. Launch the vSphere client and enter the following information:

IP Address / Name: IP or DNS address of your ESXi server

User name: root

Password: the password you set during the install process

3. You’ll be warned that an untrusted SSL certificate is installed. Click the Ignore button.

If everything worked right, at this point you should be in! You’ll be warned that you’re using a 60 day evaluation license.

If you have a VMWare account, you can log in to the VMWare website and download a free ESXi license to use with your Mini ESXi server.

Wrap-up

Overall, once I got all the pieces I needed, I found the installation and network configuration process to be fairly intuitive (which is not something you can say for all enterprise-grade software.)

There were two main issues I ran into. The first was the Broadcom network driver, which the stock ESX 5.1 install does not include. I was able to fix that by slipstreaming the driver into the ESXi installer.

The second problem was outside the ESXi server and was an issue I had with my 2012 Airport Express. Here’s the details on this issue:

I have my boxes in a couple of places. The first location is over by my TV, where the cable modem is. I have a TiVo, a home theater Mini and my main 802.11n dual-band wireless access point located there.

My office is in another room. The machines there are hooked into a gigabit switch and use an Airport Express connected to the switch to act as a wireless bridge back to the internet connection provided by the cable modem. After I set up two VMs on the ESXi server, there were now three IP addresses associated with the Mini. The Airport Express started only accepting two IP addresses from the ESXi Mini, and dropping the other.

I was able to identify the Airport Express as the problem because I could ping the third address from other machines located on that same switch. After trying to fix it a few ways, the ultimate fix was to get the Airport Express out of the loop. I did this by relocating the ESXi Mini to be stacked underneath the home theater Mini in the TV cabinet. Once the Airport Express was no longer providing the network connection, the ESXi server’s three IP addresses were all available.