I’ve recently been building a small set of CentOS server virtual machines with various settings preconfigured, and packages preinstalled. These were built from the ‘minimal’ CentOS-6.5-x86_64-minimal.iso distribution, as you don’t need a GUI to administer a Linux server. Initially these VMs were built manually, following a build document, but after several additions to the VMs, and documenting these updates in the build document, I decided to automate the whole process. This post describes how I achieved this – I had some problems, hope this helps…

UPDATED: The need to specify an IP adderss for the remote_host property has been fixed in Packer’s GitHub repo, and should be in a release coming soon!

I decided to use Mitchell Hashimoto’s excellent Packer system. I’m running it on an Ubuntu Linux 12.04 desktop VM. Eventually this will be changed to run under Jenkins, so that changes to the configuration can be checked into source control, and the whole process can be fully automated. Until then, I’ve automated it using Windows 7 as my main system, with VMware Player 6.0.1 running the Ubuntu Linux desktop. I also have an instance of VMware ESXi 5.5.0 also running under VMware Player. The Ubuntu VM with Packer creates the new CentOS VMs inside this Nested ESXi. If you haven’t seen the film Inception, now might be a good time to watch it…. Both the Ubuntu and ESXi VMs use bridged networking, and are on the same IP network.

On the ESXi system, I have:

  • installed the VMware Tools for Nested ESXi
  • configured remote SSH access and the ESXi Shell (under Troubleshooting Mode Options) – Packer currently requires SSH access to ESXi, rather than using VMware’s API; this may change in the future
  • enabled discovery of IP address information via ARP packet inspection. This is disabled by default, and is enabled by SSH using esxcli system settings advanced set -o /Net/GuestIPHack -i 1
  • allowed Packer to connect to the VNC session of the VM being built, so that it can provide the early boot commands to the CentOS installer (specifically, giving KickStart a specific configuration file, served by a small web server – more on this later). To enable VNC access, I used the vSphere client to visit the server’s Configuration/Security Profile settings, and under Firewall/Properties…, enabled gdbserver (which enables ports in the range VNC requires, 5900 etc.) and also SSH Client and SSH Server (I forget some of the other things I tried… sorry!)
  • configured a datastore called ‘vmdatastore’ which is where I want Packer to build the VMs.

On the Ubuntu system, I have a directory containing:

  • The CentOS minimal .ISO
  • A Kickstart file. This was taken from a manual installation’s anaconda-ks.cfg, and modified using a CentOS desktop’s KickStart Configuration tool. See below for its contents.
  • The Packer .JSON script. See below.
  • A script to launch a webserver to serve this directory – Packer needs to get the .ISO and KickStart file over the network, and this is how it’s served. Nothing complex: python has a simple one-line server which I use here.
  • A script to run packer.
  • A script to run on the built VM after the OS has been installed. This isn’t the hard part, so this just echoes something: in reality, this installs the packages I need, configures all kinds of stuff.

So let’s see some scripts. They are all in my ~/packertemplatebuilding directory. The Ubuntu desktop VM’s IP address is 192.168.0.1, and the ESXi VM’s IP address is 192.168.0.2; root SSH access to ESXi is used, and the password is ‘rootpassword’. (Of course these are not the real settings!)

The webserver launching script:

#!/bin/sh
python -m SimpleHTTPServer &

The Packer launch script:

#!/bin/sh
# export PACKER_LOG=enable
packer build base-packer.json

The Packer script – one of the problems I had was that the IP addresses you see in here were initially given as hostnames, and set in DNS. This didn’t work, as Packer (0.5.1) is using Go’s net.ParseIP(string-ip-addr) on the remote_host setting, which yielded the error “Unable to determine Host IP”. Using IP addresses isn’t ideal, but works for me:

{
  "builders": [
    {
      "type": "vmware-iso",
      "iso_url": "http://192.168.0.1:8000/CentOS-6.5-x86_64-minimal.iso",
      "iso_checksum": "0d9dc37b5dd4befa1c440d2174e88a87",
      "iso_checksum_type": "md5",
      "disk_size": "10240",
      "disk_type_id": "thin",
      "http_directory": "~/packertemplatebuilding",
      "remote_host": "192.168.0.2",
      "remote_datastore": "vmdatastore",
      "remote_username": "root",
      "remote_password": "rootpassword",
      "remote_type": "esx5",
      "ssh_username": "root",
      "ssh_password": "rootpassword",
      "ssh_port": 22,
      "ssh_wait_timeout": "250s",
      "shutdown_command": "shutdown -h now",
      "headless": "false",
      "boot_command": [
        "<tab> text ks=http://192.168.0.1:8000/ks.cfg<enter><wait>"
      ],
      "boot_wait": "20s",
      "vmx_data": {
        "ethernet0.networkName": "VM Network",
        "memsize": "2048",
        "numvcpus": "2",
        "cpuid.coresPerSocket": "1",
        "ide0:0.fileName": "disk.vmdk",
        "ide0:0.present": "TRUE",
        "ide0:0.redo": "",
        "scsi0:0.present": "FALSE"
      }
    }
  ],
"provisioners": [
    {
      "type": "shell",
      "script": "ssh-commands.sh"
    }
  ]
}

Note that this need for IP addresses has been fixed and will be in a future Packer release.

The ssh-commands.sh script:

#!/bin/sh
echo Starting post-kickstart setup

And finally, the Kickstart file ks.cfg, note the hashed value of the VM’s root password has been redacted. Use the Kickstart Configuration tool to set yours appropriately:

#platform=x86, AMD64, or Intel EM64T
#version=DEVEL
# Firewall configuration
firewall --enabled --ssh --service=ssh
# Install OS instead of upgrade
install
# Use CDROM installation media
cdrom

rootpw  --iscrypted insert-hashed-password-here
authconfig --enableshadow --passalgo=sha512

# System keyboard
keyboard uk
# System language
lang en_GB
# SELinux configuration
selinux --enforcing
# Do not configure the X Window System
skipx
# Installation logging level
logging --level=info

# Reboot after installation
reboot

# System timezone
timezone --isUtc Europe/London
# Network information
network  --bootproto=dhcp --device=eth0 --onboot=on
# System bootloader configuration
bootloader --append="crashkernel=auto rhgb quiet" --location=mbr --driveorder="sda"

# Partition clearing information
zerombr
clearpart --all  --drives=sda

# Disk partitioning information
part /boot --fstype="ext4" --size=500
part pv.008002 --grow --size=1
volgroup vg_centos --pesize=4096 pv.008002
logvol / --fstype=ext4 --name=lv_root --vgname=vg_centos --grow --size=1024 --maxsize=51200
logvol swap --name=lv_swap --vgname=vg_centos --grow --size=3072 --maxsize=3072

%packages --nobase
@core

%end

And that’s it! You’ll have to adjust the timings of the various delays in the Packer .JSON file to match your system. Have the appropriate amount of fun!