Google cloud services (computing instance) offer encryption by default for disk storage, the customer can provide its own key with the feature customer supplied encryption (detailed here also).

How can we apply a disk encryption with cryptsetup without giving the encryption key to google?

On Linux, when using an encrypted disk (luks), it is unlocked at boot time with a password the idea here is to encrypt the disk with cryptsetup on top of the google system (default encryption) and get an earlier access to the instance to be able to unlock the drive at the boot time. This can be implemented with the help of the remote serial console feature (note that an opensuse VM was used for this howto, the different steps should not differ for other distros).

Con:

• Disk speed will be reduced by 10~15%.
• At boot/reboot time the encryption password need to be introduced over a serial session.
• Reboot on failure/migration may require manual password introduction (if not automated).

Pro:

• Data access is truly limited to the customer/owner.
• Customer privatized disk encryption at rest.
• Customer privatized disk encryption for snapshot backups and images.
• Google can not access the disk’s data (or at least it would be very hard).
• The disk encryption key is not given to Google.
• Customizable encryption method, algorithm and key size.
• Data protection improvement etc.

Implementation summary:

• 0 Have an existing VM instance
• 1 Enable instance/VM serial access
• 2 Create an additional (target) encrypted hard drive
• 3 Copy the current hard drive to the encrypted one
• 4 Replace the old drive with the new one
• 5 Start the VM and introduce the password over the serial console

Current google implementation

<div style="position: relative; margin: 1.5em 0; padding-bottom: 56.25%;">
    <iframe style="position: absolute;" src="https://www.youtube.com/embed/Svz2KHE1mdM" width="100%" height="100%" frameborder="0" allowfullscreen></iframe>
</div>

How to encrypt the disk of a VM (gcloud, cryptsetup):

• Prerequisite:

  • Create a new temporary VM on the target zone (this need to be a copy of the target machine, you can backup the disk with a snapshot then restore the snapshot to a new disk)
  • Create a new empty disk (to be encrypted later), this is the target disk, its size needs to be at least 256 MB larger than the source disk (for the new /boot partition), also the disk can be more larger if you want to expand the space.
  • Mount both disks to the temporary VM and start it.

      Temporary VM Config:
      /dev/sda : main disk, copy of the original source disk 
      /dev/sdb : new empty disk larger than /dev/sda
    

• Serial console:

  • Summary: enable serial port connection, and connect to the machine over serial port (doc1, doc2, doc3)
  • Under metadata add serial-port-enable with the value TRUE (and enable serial port on the instance options)
  • Connect to the serial port with ssh/gcloud; gcloud example:

      gcloud compute --project=prj-name connect-to-serial-port vm-name --zone=us-central2-b
    

• Update grub config:

  • Summary: enable serial console (makes grub accessible with the serial console)
  • Add/edit this to /etc/default/grub

      # ...Enabling serial console... 
      GRUB_TIMEOUT=15
      GRUB_TERMINAL="serial"
      GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
      # ...End
    

  • Apply the settings with grub2-mkconfig -o /boot/grub2/grub.cfg
  • Reboot and check if grub is accessible on the serial console.

• Format the target disk:

  • Summary: we need to create 2 partitions one for /boot (un-encrypted) and an other for the system / (encrypted), it does not matter if you already had a separate partition for /boot or not on your old system, on the new one we would need to have it for the system to boot grub and ask for the password to unlock the encrypted partition.
  • Format the empty disk and create 2 partitions one sized at least 256 MB (to contain /boot) and the other one with the remaining space; Use fdisk /dev/sdb to do so, create the first one with 256 MB or more as a primary partition and enable the bootable flag; then create a second primary partition sized with the remaining space (note that if you were using a partition for the swap on the old system you need to create that partition as well)

      /dev/sda :
      /dev/sdb :
            |- sdb1 : 256 MB primary and bootable (empty)
            |- sdb2 : xx GB primary (empty)
    

  • Set the file system to ext4 for the boot partition with mkfs.ext4 /dev/sdb1
  • Wipe the second partition for an additional security with

      dd if=/dev/urandom of=/dev/sdb2 bs=4096 status=progress
    

  • Setup the encryption for sdb2 with

      cryptsetup -y -v --cipher aes-xts-plain64 --key-size 256 --pbkdf-force-iterations=100200 --hash sha256 --type luks1 --label=linux --use-random luksFormat /dev/sdb2
    

    Change the cipher, the key size, encryption type etc. to match your needs

  • Check the encrypted partition with cryptsetup luksDump /dev/sdb2
  • Unlock the encrypted partition with cryptsetup luksOpen /dev/sdb2 crypteddisk
  • Set the encrypted mapped partition to ext4 with mkfs.ext4 /dev/mapper/crypteddisk
  • Close the encrypted partition cryptsetup close /dev/mapper/crypteddisk

• Clone the source disk:

  • Summary: we need to copy our system / to the new encrypted partition /dev/mapper/crypteddisk (sdb2) for this step you can use different tools, in this howto we will be using dd, important note here, for dd to copy a disk without any error the source partition need to unmounted. If you want to avoid any data loss you can create an other temporary VM and attach to it the source and target disk as additional disks to perform this step… Otherwise to avoid a too long migration process i closed most of the running process and used dd without un-mounting the source partition (not recommended), then used fsck to correct any error due to the fact that the source was not un-mounted.
  • Unlock the encrypted partition with cryptsetup luksOpen /dev/sdb2 crypteddisk
  • Copy the source partition to the target with the following (supposing / is /dev/sda1)

      dd if=/dev/sda1 of=/dev/mapper/crypteddisk bs=4096 status=progress
    

  • Check and fix the new encrypted partition with fsck /dev/mapper/crypteddisk
  • Check the disks structures with fdisk -l (don’t pay attention to crypteddisk size, we are fixing that later)

• Partitions UUID:

  • Get all the UUID of all the partitions and keep that info.

      blkid /dev/sda 
      blkid /dev/sda1
      blkid /dev/sdb
      blkid /dev/sdb1
      blkid /dev/sdb2
      blkid /dev/mapper/crypteddisk
    

• Resize the disk:

  • Summary: in this step we will expand the size of the new encrypted partition.
  • Expand the encrypted partition with

      cryptsetup resize crypteddisk -v
      e2fsck -f /dev/mapper/crypteddisk
      resize2fs /dev/mapper/crypteddisk
    

• Backup the MBR:

  • This is not required but it may be useful

      dd if=/dev/sdb of=/backup/location/sdb.mbr count=1
      dd if=/dev/sda of=/backup/location/sda.mbr count=1
    

• Setup the boot partition:

  • Mount the new boot partition and copy /boot content to it.

      mkdir /tmp/boot
      mount /dev/sdb1 /tmp/boot
      cp -a /boot/* /tmp/boot/
      ls -l /tmp/boot/*
      umount /tmp/boot
      rmdir /tmp/boot
    

  • Remove the old /boot folder content and leave it as a mount location.

      mkdir /tmp/system
      mount /dev/mapper/crypteddisk /tmp/system
      rm -rf /tmp/system/boot/*
      ls -l /tmp/system/boot/*
      umount /tmp/system
      rmdir /tmp/system
    

• Chroot and setup the new system:

  • Load the new encrypted disk as the current one (chroot) and setup it to apply the new system config, encryption etc. (also make sure /mnt is empty before proceeding).

      mount /dev/mapper/crypteddisk /mnt/
      mount /dev/sdb1 /mnt/boot
      for i in sys dev proc; do mount --bind /$i /mnt/$i; done
      chroot  /mnt
    

    From now on we are on the new system (keep in mind that sdb need to be considered sda on the configs files)

  • Update /etc/fstab config (make sure to use sdb1 uuids not sda1), here is cat /etc/fstab output:

      # Main Partition ----------------------
      # Entry for /dev/mapper/crypteddisk (sda2) :
      UUID=CHANGE-THIS-WITH-CRYPTEDDISK-UUID / ext4 noatime,acl 0 0
            
      # Boot Partition ----------------------
      # Entry for /dev/sda1 :
      #/dev/sda1 /boot ext4 defaults 1 2
      UUID=CHANGE-THIS-WITH-THE-CURRENT-SDB1-UUID /boot ext4 noatime,acl 1 2
            
      # Swap Partition/File -----------------
      /swap/swapfile swap swap defaults 0 0
      # 
    

  • Update /etc/crypttab config (if crypttab file does not exist, create it with -rw-r--r-- permissions), also make sure to use sdb2 uuids not sda1 nor crypteddisk, here is cat /etc/crypttab output:

      crypteddisk UUID=CHANGE-THIS-WITH-CURRENT-SDB2-UUID
    

  • Update the grub config on /etc/default/grub, you only need to change GRUB_ENABLE_CRYPTODISK, GRUB_CMDLINE_LINUX and GRUB_DISABLE_OS_PROBER here is cat /etc/default/grub output:

      GRUB_DISTRIBUTOR=My-Custom-Server...
      # .................................... Command line
      #GRUB_CMDLINE_LINUX=" root=/dev/sda1 disk=/dev/sda resume=swap console=ttyS0,38400n8 quiet"
      GRUB_CMDLINE_LINUX=" root=/dev/mapper/crypteddisk luks_root=/dev/sda2 luks="root" disk=/dev/sda resume=swap console=ttyS0,38400n8 quiet"
      # .................................... Options
      GRUB_DEFAULT=0
      GRUB_HIDDEN_TIMEOUT=0
      GRUB_HIDDEN_TIMEOUT_QUIET=true
      GRUB_GFXMODE=800x600
      GRUB_GFXPAYLOAD_LINUX=keep
      GRUB_THEME=/boot/grub2/theme/theme.txt
      GRUB_BACKGROUND=
      # .................................... Enabling serial console... 
      GRUB_TIMEOUT=30
      GRUB_TERMINAL="serial"
      GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"
      # .................................... Enabling cryptsetup
      GRUB_ENABLE_CRYPTODISK="y"
      # .................................... Ignore other os
      GRUB_DISABLE_OS_PROBER="true"
      # ....................................
    

  • Apply grub changes with:

      grub2-mkconfig -o /boot/grub2/grub.cfg
    

  • Update the init ram disk and force encryption modules to be included with:

      mkinitrd -d /dev/mapper/crypteddisk -f "dm luks"
    

  • Update the MBR and reinstall grub on the new disk to make it bootable

      grub2-install /dev/sdb
    

  • Exit the chroot and un-mount everything

      exit
      cd /
      for i in sys dev proc; do umount /mnt/$i; done
      unmount /mnt/boot
      unmount /mnt
    

    If you have any issue un-mounting a partition use -l option for instance umount -l /mnt/sys

• Apply the magic:

  • Turn off the temporary VM; Detach all the disks; Attach the encrypted disk as the boot disk.
  • Turn on the temporary VM; Connect to it with the serial console and voila! (you will be asked the password to unlock the crypted partition at boot time).
  • Test if everything is fine, then use the new encrypted disk on the production VM and delete the temporary copy disk and the temporary VM.
  • Additional infos are available here and here. Enjoy ;)