Ephemeral Values and Write Only Arguments

Ephemeral resources were introduced in Terraform 1.10, but they were missing a key component. Write-only arguments are a new feature in Terraform 1.11, and they help to complete the goals of ephemeral resources.

Ephemeral Values

I’ve already written a whole post about ephemeral values, but allow me to summarize it here as well. Ephemeral values were created to help address the issue of sensitive values being stored in state and plan files. State data holds the attribute values of all data sources and resources, and some of those attributes might be sensitive data you’d rather not appear in state.

An additional concern is sensitive information found in saved Terraform plan files. Plan files not only contain the entirety of the configuration, but also the planned changes and variable values used to create the plan.

Values that are marked as ephemeral will never be persisted in state or in a plan file. The value is accessed and stored in memory for the period during which it is needed, and then it is flushed. You can get ephemeral values from input variables, output values, or ephemeral resources. Input variables and output values can be marked as ephemeral by setting the ephemeral argument to true. Ephemeral resources are a distinct block type and must be implemented by a provider.

Use Cases for Ephemeral Values

So how can you use ephemeral values? When the feature was released in 1.10, the primary use cases were for provider settings and provisioner credentials. That’s because the values used for providers and provisioners are not persisted in the plan file or in state. What about resource arguments though? By default, all resource arguments are recorded in the plan and state, so you could not use ephemeral values, that is until the introduction of write-only arguments.

Write-Only Arguments

Write-only arguments are a special type of resources argument that is not persisted in state and not written to the plan file. You can write a new value to the resource, but you cannot retrieve the current value through state data. Now you can’t simply mark an argument in a resource as write-only, instead a new argument needs to be added to the resource type in the provider to support the write-only functionality.

For example, version 4.34.0 of the azurerm provider includes a write-only argument for the value of a Key Vault Secret, and a write-only password argument for several database related resources like azurerm_mssql_server. As new releases of the provider roll out, additional resources will have write-only arguments added to them.

You might be wondering, if Terraform doesn’t know the value of the write-only argument, how does it know when that value changes? The answer is that along with the new write-only argument is a version argument to signal to Terraform that the current value should be overwritten.

For instance, consider the following azurerm_mssql_server resource:

resource "azurerm_mssql_server" "db" {
  name                                    = local.name
  resource_group_name                     = azurerm_resource_group.db.name
  location                                = azurerm_resource_group.db.location
  version                                 = "12.0"
  administrator_login                     = "mssqladmin"
  administrator_login_password_wo         = var.db_password_info.value
  administrator_login_password_wo_version = var.db_password_info.version
}

The write-only password argument is called administrator_login_password_wo and the version argument is called administrator_login_password_wo_version. That seems to be the pattern for all write-only arguments I’ve seen. The write-only argument has the suffix wo and the corresponding version argument has the suffix wo_version.

Key Vault Example

Why don’t we walk through a few examples so you can see these write-only arguments in action? If you want to play along, the code for these examples is stored in my Terraform Tuesdays repository.

Consider a configuration where we want to store an Azure Key Vault secret value. Before write-only arguments, the resource block would look like this:

resource "azurerm_key_vault_secret" "normal" {
  name         = "db-password-normal"
  value        = var.db_password_regular
  key_vault_id = azurerm_key_vault.example.id
}

After deploying the configuration, we can inspect the state data for the Key Vault secret, and sure enough, the value of the secret is stored in plaintext.

{
      "mode": "managed",
      "type": "azurerm_key_vault_secret",
      "name": "normal",
      "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "value": "T@COSsoG00dY'All",
...

That’s no good! If someone gains unauthorized access to my state data, they can find out my innermost secrets.

Sidebar: I know that some people want to protect state by encrypting it locally before pushing to the remote backend. I guess that’s fine if your state backend is compromised, but anyone with permissions to run a plan would be able to view the encrypted state data. Better to not have your sensitive data in there to begin with.

Now, let’s check out the same resource type, but with write-only arguments:

resource "azurerm_key_vault_secret" "write_only" {
  name             = "db-password-wo"
  value_wo         = var.db_password_ephemeral
  value_wo_version = var.db_password_version
  key_vault_id     = azurerm_key_vault.example.id
}

Instead of setting the value argument, I’m using the value_wo argument. When you use that argument, you also have to include the vault_wo_version argument. The input variable for the value_wo is marked as ephemeral, so I don’t accidentally use it somewhere else in the configuration.

variable "db_password_ephemeral" {
  description = "The ephemeral database password to be stored in the Key Vault."
  type        = string
  sensitive   = true
  ephemeral   = true
}

variable "db_password_version" {
  description = "The version of the database password to be stored in the Key Vault."
  type        = number
}

If I try and use db_password_ephemeral with a regular argument or in a non-ephemeral context, I’ll get back the error:

Ephemeral values are not valid for "value", because it is not a write-only attribute and must be persisted to state.

In my terraform.tfvars I have the following values set:

prefix                = "sopes"
db_password_regular   = "T@COSsoG00dY'All"
db_password_ephemeral = "T@COSsoG00dY'All"
db_password_version   = 1

Obviously, I wouldn’t want to store the secret values in a terraform.tfvars file, as that would defeat the purpose of hiding them from state. But for the purpose of demonstration, we’ll let that one go. You’d probably want to use an environment variable to pass the value in a production context.

If I check out the state data for my new Key Vault secret:

{
      "mode": "managed",
      "type": "azurerm_key_vault_secret",
      "name": "write_only",
      "provider": "provider[\"registry.terraform.io/hashicorp/azurerm\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "value": "",
            "value_wo": null,
            "value_wo_version": 1,

The value field is empty, and the value_wo has a value of null. Our secret value is no longer stored in state!

Updating Write-only Arguments

Now what if I wanted to update the secret? I’ll change the value for the write-only secret in my terraform.tfvars file:

prefix                = "sopes"
db_password_regular   = "T@COSsoG00dY'All"
db_password_ephemeral = "BUrr!t0sR@lso"
db_password_version   = 1

Then I’ll run a terraform plan:

...

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Because I didn’t change the value of the value_wo_version argument, Terraform has no way of knowing that the value of the secret should be updated. And sure enough, we get back a plan with no changes.

Now I will update the db_password_version to 2, and run a new plan:

Terraform will perform the following actions:

  # azurerm_key_vault_secret.write_only will be updated in-place
  ~ resource "azurerm_key_vault_secret" "write_only" {
        id                      = "https://sopes-ephemeral-83610.vault.azure.net/secrets/db-password-wo/582c7570578c42dd80d0c6badf2a938b"     
        name                    = "db-password-wo"
        tags                    = {}
      ~ value_wo_version        = 1 -> 2
        # (8 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Since the version has changed, now Terraform knows it needs to update the value as well, and the plan we get back has one change listed.

A few quick notes about the value_wo_version argument.

First, the value for the argument must be a whole number greater than zero. So it might make sense to update the input variable with some validation:

variable "db_password_version" {
  description = "The version of the database password to be stored in the Key Vault."
  type        = number

  validation {
    condition     = var.db_password_version > 0 && var.db_password_version == floor(var.db_password_version)
    error_message = "The db_password_version must be a non-negative integer."
  }
}

Second, I thought I would be clever and combine the password value and version into a single input variable:

variable "db_password_ephemeral" {
  description = "The ephemeral database password to be stored in the Key Vault."
  type        = object({
    value   = string
    version = number
  })
  sensitive   = true
  ephemeral   = true
}

But this marks the entire input variable as ephemeral. When you try to use the version value with the value_wo_version argument, Terraform throws an error because the argument is not write-only and the value is ephemeral. So until you can mark individual attributes of an input variable as ephemeral, you’ll need to keep the value and version separate.

Using Ephemeral Resources with Write-only Arguments

The first example showed how we might get a secret value into Key Vault without storing it in state data. What about using the secret? That’s where ephemeral resources come into play.

In a new configuration, I have an ephemeral block for a Key Vault secret:

ephemeral "azurerm_key_vault_secret" "db_password_ephemeral" {
  name         = var.db_password_info.key_vault_secret_name
  key_vault_id = var.db_password_info.key_vault_id
}

The syntax of the block is exactly the same as the azurerm_key_vault_secret data source, but the block keyword is ephemeral instead of data.

To use this ephemeral resource, I have an azurerm_mssql_server using the argument administrator_login_password_wo and administrator_login_password_wo_version:

resource "azurerm_mssql_server" "db" {
  name                                    = local.name
  resource_group_name                     = azurerm_resource_group.db.name
  location                                = azurerm_resource_group.db.location
  version                                 = "12.0"
  administrator_login                     = "mssqladmin"
  administrator_login_password_wo         = ephemeral.azurerm_key_vault_secret.db_password_ephemeral.value
  administrator_login_password_wo_version = var.db_password_info.version
}

Just like when we created the Key Vault secret, we need to include a version here so Terraform knows if it needs to update the login password, since that value isn’t stored in state.

Let’s see what happens when I run a plan:

$ terraform plan
ephemeral.azurerm_key_vault_secret.db_password_ephemeral: Opening...
ephemeral.azurerm_key_vault_secret.db_password_ephemeral: Opening complete after 3s
ephemeral.azurerm_key_vault_secret.db_password_ephemeral: Closing...
ephemeral.azurerm_key_vault_secret.db_password_ephemeral: Closing complete after 0s

During the plan process, Terraform opens the contents of the ephemeral resource in memory. Then it uses that information with the MSSQL server, and once the plan is complete, the ephemeral resource is closed. Terraform will do the same thing during apply, accessing the ephemeral resource, using the value for creation, and closing the resource.

The planned creation of the azurerm_mssql_server.db also shows the write-only nature of the attribute:

  # azurerm_mssql_server.db will be created
  + resource "azurerm_mssql_server" "db" {
      + administrator_login                      = "mssqladmin"
      + administrator_login_password_wo          = (write-only attribute)
      + administrator_login_password_wo_version  = 1

Feel free to deploy this configuration yourself if you want to see the resulting state data.

Unsupported Resources

The addition of write-only arguments is absolutely fantastic and I’m excited to see more resource types implement them. But what if the resource type you want to manage isn’t supported? There is a solution! The AzAPI provider.

My fellow HashiCorp Ambassador, Stu Mace, has written a whole blog post on using the AzAPI provider and its sensitive_body argument. Give it a read and a share!

Conclusion

The introduction of write-only arguments closes the loop on the ideas first introduced with ephemeral values in Terraform 1.10. Between write-only arguments in azurerm resources and the sensitive_body in azapi resources, you can now keep your secrets out of Terraform state entirely. And I think that’s a good thing!

If I had one gripe, it’s the management of the new version argument to signal when the secret value changes. As a community, we’ll need to develop some guidelines and best practices on managing the version value effectively.