Setting up a Cloud at Cost node with a fresh Ubuntu image

This file documents how to install a fresh ubuntu image on a CaC node. The ubuntu image is taken directly from ubuntu.com. However, they only provide images over http (not https).

It also documents how to set up full-disk encryption with remote unlocking via ssh.

I used several good references to develop this set of instructions. They are listed at the bottom of the page.

I would also note that none of this material is particularly CaC specific. Rather, the approach will apply to all modern Ubuntu server installations.

Install a fresh Ubuntu image

Get the mini.iso for the desired distribution:

wget -c "http://archive.ubuntu.com/ubuntu/dists/<desired_dist>/main/installer-amd64/current/images/netboot/mini.iso"

(here <desired_dist> is e.g. yakkety)

Mount the mini.iso as a loopback on your local system.
```
sudo mount -o loop mini.iso /mnt
```

Copy /mnt/linux and /mnt/initrd.gz to a webserver someplace.

cp /mnt/linux /var/www/linux
cp /mnt/initrd.gz /var/www/initrd.gz

On the CaC node (built according to standard procedures), download the linux and initrd.gz files to the /boot partition.
```
wget -c "https://<your_web_server>/linux"
wget -c "https://<your_web_server>/initrd.gz"
```
Open up a console to the CaC node from panel.cloudatcost.com. Steps 6-11 need to be performed on the console.
Reboot the CaC node (click the alt-ctl-del button in the upper right of the console screen) and hit escape several times to stop at the grub2 screen on the console. Then enter "c" to get to the grub prompt.

At the grub prompt, enter the following:

grub> linux /linux
grub> initrd /initrd.gz
grub> boot

This will boot you into the ubuntu installer. Follow the installation process using the node's network parameters from the CaC panel.
When prompted to repartition the disk, go ahead and repartition using full disk encryption. You will need to enter a password for the FDE. Go ahead and enter a throwaway password. This will be replaced later once the secure FDE boot process is established.
At this point, install only the system utilities packages (already marked for installation) and the openssh-server packages.
Complete the installation. Reboot the node, type the FDE password into the console to decrypt the disk.

You now have an original ubuntu image installed. You can ssh into the node and do work. However, you will have to decrypt the FDE disk at boot time via the (unencrypted) console each time you reboot. This is problematic for two reasons: 1) console access is easily lost, and 2) the FDE password is sent in cleartext over the network.

Set up remote FDE decryption

To set up secure FDE decryption via ssh at boot time, follow these steps on the CaC node:

Install the dropbear and busybox packages.
```
sudo apt-get install dropbear busybox
```
(required on ubuntu yakkety, maybe others) Copy the /bin/busybox binary to /usr/lib/initramfs-tools/bin. This is required because the binary in /usr/lib/initramfs-tools/bin does not work with the current unlock script.
```
sudo mv /usr/lib/initramfs-tools/bin/busybox /usr/lib/initramfs-tools/bin/busybox.orig
sudo cp /bin/busybox /usr/lib/initramfs-tools/bin/busybox
```

Copy the system's ssh server keys to /etc/dropbear-initramfs, converting them to dropbear format in the process. Also, remove old dropbear keys.

sudo /usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_ecdsa_key /etc/dropbear-initramfs/dropbear_ecdsa_host_key
sudo /usr/lib/dropbear/dropbearconvert openssh dropbear /etc/ssh/ssh_host_rsa_key /etc/dropbear-initramfs/dropbear_rsa_host_key
sudo rm /etc/dropbear/*key

Create a /etc/dropbear-initramfs/authorized_keys file containing the public key(s) that you want to authenticate to the CaC node with when decrypting the FDE.
```
sudo vi /etc/dropbear-initramfs/authorized_keys
...
```
Edit /etc/initramfs-tools/initramfs.conf and add the following line after the DEVICE= line (using params from the CaC panel for this node).
```
IP=<server ip address>::<gateway ip address>:<netmask>::<ethernet device name>:none
```
e.g
```
IP=192.168.122.192::192.168.122.1:255.255.255.0::eth0:none
```
Update the initramfs.
```
sudo update-initramfs -u -k all
```

Reboot the node and test the ssh connection:

ssh -i /path/to/your/private/key root@<cac_ip_or_fqdn>

You should be prompted to type "cryptroot-unlock" and enter the FDE password.
Once the decryption is successful, type "exit" to allow the CaC node to reboot.

Change the FDE password

At this point, you can reboot your CaC node securely. However, you should change the FDE password. To do this:

Add a new key to the luks partition.
```
sudo cryptsetup luksAddKey /dev/sda5
```
Delete the old key from the luks partition.
```
sudo cryptsetup luksRemoveKey /dev/sda5
```
Next time you reboot, you can unlock the FDE with a fresh password that hasn't been sent to the console over the network in the clear.

Important Reminders

The dropbear ssh private key is stored unencrypted in the initramfs. As a result, it is possible for anyone who has physical access to the disk to copy the ssh key and spoof the node.
While the node is running, the luks disk encryption key is stored in ram. As a result, it is possible for anyone who has physical access to the machine to copy the encryption key and decrypt the disk image offline. The only way to assure this does not happen is to maintain physical security over the node hardware at all times. This is a problem with all vps installations, and not just with CaC.