Auto-unseal Vault using GCP Cloud KMS

9min
|
Terraform
Vault

When a Vault server starts, it starts in a sealed state. It does not know how to decrypt data, and has limited access. Before you can perform an operation, you must unseal it. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This prevents one person from having full control of Vault. However, this process is manual and can become painful when you have several Vault clusters as there are now different key holders with different keys.

Solution

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, AWS KMS, Azure Key Vault, Google Cloud KMS, and OCI KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

This tutorial demonstrates an example of how to use Terraform to provision an instance that can use an encryption key from GCP Cloud KMS to unseal Vault.

Prerequisites

To perform the tasks described in this tutorial, you need to have:

Google Cloud Platform account
- Cloud IAM Service Account created and the JSON file downloaded
- Cloud KMS API enabled
Terraform v0.15.2 or later installed and basic understanding of its usage

Step 1: Provision the cloud resources

Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.
```
$ git clone https://github.com/hashicorp/vault-guides
```
This repository has supporting content for the Vault tutorials. The content specific to this tutorial is within a sub-directory.
Change your working directory to /operations/gcp-kms-unseal.
```
$ cd vault-guides/operations/gcp-kms-unseal
```
The working directory should contain the provided Terraform files.
```
$ tree
.
├── README.md
├── main.tf
├── terraform.tfvars.example
├── variables.tf
└── versions.tf
```
The main.tf generates the following:
- Service account with Cloud KMS IAM for Compute Engine instances
- Compute Engine instance with Vault installed
- (Optional) Cloud KMS key ring and crypto key

Use the necessary GCP account information in the terraform.tfvars.example and save it as terraform.tfvars. Overwrite the default variable values (variables.tf) as needed.

Example:

gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"

gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"

It assumes that you have a Cloud KMS key ring named, "test" and a crypto key named "vault-test". If your Cloud KMS key ring (key_ring) and crypto key (crypto_key) names are different, be sure to set the correct values in terraform.tfvars.

Example:

gcloud-project = "gcloud-vault-test"
account_file_path = "/Users/james/gcp/gcloud-vault-test.json"

gcloud-region = "us-central1"
gcloud-zone = "us-central1-c"
key_ring = "my-key-ring-name"
crypto_key = "my-crypto-key-name"
# keyring_location = "global"

If you do NOT have a Cloud KMS key ring and crypto key

If you do NOT have a Cloud KMS key ring and crypto key to use, un-comment the key ring and key creation block in the main.tf file (line 77 - 81 and line 84 - 88).

...snip...

# Create a KMS key ring
resource "google_kms_key_ring" "key_ring" {
   project  = "${var.gcloud-project}"
   name     = "${var.key_ring}"
   location = "${var.keyring_location}"
}

# Create a crypto key for the key ring
resource "google_kms_crypto_key" "crypto_key" {
   name            = "${var.crypto-key}"
   key_ring        = "${google_kms_key_ring.key_ring.self_link}"
   rotation_period = "100000s"
}
...snip...

Terraform will create a Cloud KMS key ring named, test in the global location, and a key named, vault-test. If you want to customize the key and key ring names, be sure to overwrite the key_ring and crypto_key values in the terraform.tfvars.

Un-comment line 92 and comment out line 93:

# Add the service account to the Keyring
resource "google_kms_key_ring_iam_binding" "vault_iam_kms_binding" {
   key_ring_id = "${google_kms_key_ring.key_ring.id}"
   # key_ring_id = "${var.gcloud-project}/${var.keyring_location}/${var.key_ring}"
   role = "roles/owner"

   members = [
     "serviceAccount:${google_service_account.vault_kms_service_account.email}",
   ]
}

Perform a terraform init to pull down the necessary provider resources.
```
$ terraform init
```
Run terraform plan to verify your changes and the resources Terraform will create.
```
$ terraform plan
```

If all looks good, perform a terraform apply to provision the resources.

$ terraform apply -auto-approve
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Step 2: Test the auto-unseal feature

SSH into the generated compute instance. SSH

To verify the Vault installation, run the vault status command.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       gcpckms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  1.8.1
Storage Type             file
HA Enabled               false

Notice that Initialized is false.

Note

If you received an error instead, refer to the troubleshooting section.

Run the vault operator init command to initialize the Vault server.

$ vault operator init

Example output:

Recovery Key 1: iz1XWxe4CM+wrOGqRCx8ex8kB2XvGJEdfjhXFC+MA6Rc
Recovery Key 2: rKZETr6IAy686IxfO3ZBKXPDAOkkwkpSepIME+bjeUT7
Recovery Key 3: 4XA/KJqFOm+jzbBkKQuRVePEYPrQe3H3TmFVmdlUjRFv
Recovery Key 4: lfnaYoZufP0uhooO3mHDAKGNZB5HLP9HYYb+LAfKkUmd
Recovery Key 5: L169hHj3DMpphGsOnS8TEz3Febvdx3vsG3Xr8kGWdUtW

Initial Root Token: s.AWnDagUkKNNbvkENiL72wysn

Success! Vault is initialized

Recovery key initialized with 5 key shares and a key threshold of 3. Please
securely distribute the key shares printed above.

The initialization generates recovery keys (instead of unseal keys) when using auto-unseal. Some of the Vault operations still require Shamir keys. For example, to regenerate a root token, each key holder must enter their recovery key. Similar to unseal keys, you can specify the number of recovery keys and the threshold using the -recovery-shares and -recovery-threshold flags. It is strongly recommended to initialize Vault with PGP.

After a successful initialization, check the Vault server status.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.8.1
Storage Type             file
Cluster Name             vault-cluster-23602eff
Cluster ID               0068b53a-42df-73ee-784b-d6753f08ced7
HA Enabled               false

The Vault server is already unsealed (Sealed is false).

In the service log, you should find a trace where Vault fetches the GCP KMS key to unseal the Vault server.

$ sudo journalctl --no-pager -u vault

...
 ==> Vault server configuration:
       GCP KMS Crypto Key: vault-test
         GCP KMS Key Ring: test
          GCP KMS Project: gcloud-vault-test
           GCP KMS Region: global
                Seal Type: gcpckms
                      Cgo: disabled
               Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_re
quest_size: "33554432", tls: "disabled")
                Log Level: (not set)
                    Mlock: supported: true, enabled: false
                  Storage: file
                  Version: Vault v1.8.1
              Version Sha: c19cef14891751a23eaa9b41fd456d1f99e7e856
 ==> Vault server started! Log data will stream in below:
...

Restart the Vault server to ensure that Vault server gets automatically unsealed.

$ sudo systemctl restart vault

Verify the Vault status.

$ vault status

Review the Vault configuration file (/test/vault/config.hcl).

$ cat /test/vault/config.hcl

The configuration file looks as below.

storage "file" {
  path = "/opt/vault"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}

seal "gcpckms" {
  project     = "gcloud-vault-test"
  region      = "global"
  key_ring    = "test"
  crypto_key  = "vault-test"
}

disable_mlock = true

Notice the Vault configuration file defines the gcpckms stanza which sets the GCP Cloud KMS key ring name, crypto key name, and its location as well as your GCP project ID.

The credentials value is not needed. This is because the Vault server running on this Compute Engine instance has it own service account with Cloud KMS role.

Warning

Although the listener stanza disables TLS (tls_disable = "true") for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.

Step 3: Rotating the unseal key

When you use Shamir keys to seal Vault, you have to run the vault operator rekey command to generate a new set of unseal keys.

With Auto-unseal enabled, you can rotate the Cloud KMS key used to unseal Vault. One of the benefits of using Cloud KMS is its automatic key rotation feature which eliminates the need for a manual operation.

Automatic rotation

To enable automatic rotation of a key, set the rotation schedule using the gcloud command-line tool:

$ gcloud kms keys update <KEY_NAME> \
         --location <LOCATION> \
         --keyring <KEYRING_NAME> \
         --rotation-period <ROTATION_PERIOD> \
         --next-rotation-time <NEXT_ROTATION_TIME>

You can also manage it through GCP Console.

From the navigation menu, select Security > Cryptographic Keys.
Select the key ring used for Vault Auto-unseal.
Select the key to rotate.
Select EDIT ROTATION PERIOD.
Select the desired rotation period, and then click SAVE.

Manual rotation

To create a new key version and make it primary using the gcloud command-line tool:

$ gcloud kms keys versions create --location <LOCATION> \
         --keyring <KEYRING_NAME> \
         --key <KEY_NAME> --primary

Using the GCP Web Console, select ROTATE. Key Rotation

To learn more about Cloud KMS key rotation, refer to its online documentation.

Step 4: Clean up

Once completed, execute the terraform destroy commands to clean up the cloud resources.

$ terraform apply -destroy -auto-approve

You can delete the Terraform state files.

$ rm -rf .terraform terraform.tfstate*

Troubleshooting

If vault status command returns an error (For example, Error checking seal status: Get http://127.0.0.1:8200/v1/sys/seal-status: dial tcp 127.0.0.1:8200: getsockopt: connection refused), check the log.

$ sudo journalctl --no-pager -u vault

If the log does not exist, try running the Vault server manually.

$ sudo chmod -R 0755 /opt/vault
$ vault server -config=/test/vault/config.hcl

The output should describe what error has occurred. Verify the values in the seal stanza of /test/vault/config.hcl.

Help and reference

Auto-unseal using Azure Key Vault

Auto-unseal using AWS KMS