Add a HashiCorp Vault secret manager

To store and use encrypted secrets (such as access keys), you can add a HashiCorp Vault Secret Manager.

The video below provides an overview of how to use HashiCorp Vault as an external secret manager.

important

You can only use Harness Built-in Secret Manager to store authentication credentials for access to the corresponding secret manager.

Storing credentials from one secret manager within another can result in complex and challenging situations. Moreover, these configurations might introduce vulnerabilities, posing potential security risks.

The Harness platform has several validations, including the disabling of self-references.

Before you begin

Harness' key concepts

Harness Secret Manager Overview
Store authentication credentials
Make sure that the Harness Delegate can connect to the Vault URL.
Make sure you have View and Create/Edit permissions for secrets.

Step 1: Add a Secret Manager

This topic assumes you have a Harness Project set up. If not, go to Create Organizations and Projects.

You can add a Connector at Project/Organization/Account scope. To do this, go to Project setup, Organization, or Account Resources.

In Connectors, select Connector.

In Secret Managers, select HashiCorp Vault. The HashiCorp Vault Secret Manager settings appear.

Step 2: Overview

Enter a Name for your secret manager.

You can choose to update the ID or let it be the same as your secret manager's name. For more information, go to Entity Identifier Reference.

Enter Description for your secret manager.

Enter Tags for your secret manager.

Select Save and Continue.

Enter Vault URL.

Enter Base Secret Path. The Base Secret Path is used for writing secrets. When Harness reads secrets, it uses the full path.

For more information, go to Vault documentation.

Select the Authentication Type.

Option: App Role

The App Role option enables the Harness Vault Secret Manager to authenticate with Vault-defined roles.

The Vault AppRole method allows you to define multiple roles corresponding to different applications, each with different levels of access. The application's App Role ID and Secret ID are used for authentication with Vault. You need these to log in and fetch a Vault token.

To assign a Secret ID, you can create a new Secret or choose an existing one.

The SecretId should not expire and it should be valid until it is manually revoked. Harness uses the App Role ID and Secret ID you supply to fetch a Vault Auth Token dynamically whenever there is a CRUD operation of secrets related to this vault. For example, when creating a secret in this vault, Harness internally uses this App Role Id and Secret ID and makes a call to vault via the delegate to generate a token. Now, this token is used to make the actual secret creation call to vault. This token is never received on the Harness side. It resides in the delegate and is cached for the duration that is one percent less than the time-to-live (TTL) of the token to prevent the generation of a new token for each request for improved performance. If you don't want to cache the token, you can disable the caching and token will be destroyed after the operation is successfully completed.

For more information, go to RoleID and Authenticating Applications with HashiCorp Vault AppRole in the HashiCorp documentation.

If you encounter errors, setting token_num_uses to 0 can often resolve problems.

Permissions

The Vault AppRole ID or the Periodic Token used in either of the authentication options must have an ACL policy attached so that Harness can use it. Typically, you create the policy first, then create the AppRole or Periodic Token and attach the policy.

important

The AppRole option only works with the default authentication path approle. If you change the authentication path to a non-default value like approle-test, the AppRole authentication will fail.

In the policy examples below: if you've created a Read-only Vault Secret Manager, this secret manager needs only read, and list permissions on Vault. It does not need — and cannot assume — create, update, or delete permissions. If the secrets are in the Secret Engine named “secret”, the policy must have the following permissions.

path "secret/*" {
  capabilities = ["create", "update", "list", "read", "delete"]
}

If the secrets are in a subfolder, such as secrets/harness, the policy will look like this:

path "secret/harness/*" {
  capabilities = ["create", "list", "read", "update", "delete"]
}
path "secret/harness" {
  capabilities = ["list", "read"]
}

These examples apply only to a v1 secret engine. If you are planning to use a secret engine with version 2 (versioned secret engine), then the policies are different as explained here. Go through this link to understand the correct permissions required for your use case. If the Vault Secret Manager needs to renew tokens, the following permissions are needed:

path "auth/token/renew-self" {
 capabilities = ["read", "update"]
}

Option: Token

For Harness, the Token option requires periodic tokens (tokens that have renewal options).

To create a periodic token, make sure to specify a period in the token creation command:

vault token create -policy=harness -period=768h

Next, use the new token with Harness. To do this, perform the below steps:

Select Create or Select a Secret.
The secret settings page appears. Here you can either Create a new Secret or Select an existing secret. If creating a new Secret, enter the token which you created in the Secret Value field.

For detailed steps on creating a secret, go to Add and reference text secrets.

If you have already added a Secret with your token, you can choose the same as shown below:

Select Apply.

If you want to verify the renewal manually, use the command:

vault token lookup <token_id>

Option: Vault Agent

This option enables the Harness Vault Secret Manager to authenticate with the Auto-Auth functionality of the Vault Agent.

To authenticate with Vault Agent, make sure you have configured it on the required environment, with entries for method and sinks. For more information, go to Vault Agent.

In the Sink Path field, enter any sink path you have in your Vault Agent Configuration. This is the path of the encrypted file with tokens. The specified delegate reads this file through file protocol (file://).

Option: AWS Auth

This option provides an automated mechanism to retrieve a Vault token for IAM principals and AWS EC2 instances. With this method, you do not need to manually install or supply security-sensitive credentials such as tokens, usernames, or passwords.

In the AWS Auth method, there are two authentication types:

Harness recommends using the IAM technique for authentication since it is more versatile and complies with standard practices.

To authenticate with AWS Auth, make sure you have configured the vault with entries for Header, Role, and Region. For more information, go to AWS Auth Method in the Vault documentation.

You must add the Server ID Header from Vault as a Harness Encrypted Text Secret and select it for Server Id Header in Harness.

In Role, enter the role you have configured in the Vault.

In Region, enter the AWS Region for the Secret Manager.

Option: Kubernetes Auth

This option uses a Kubernetes Service Account Token to authenticate with Vault. With this method of authentication, you can easily add a Vault token into a Kubernetes pod.

To authenticate with Kubernetes Auth, make sure you have created a role in the vault inside auth/kubernetes/role. This role authorizes the "vault-auth" service account in the default namespace and it gives it the default policy. This is also where you'll find the service account name and namespace that will be used to access the vault endpoint.

For more information, go to Kubernetes Auth Configuration in the Vault documentation.

In Role Name, enter the role you have configured in the Vault.

In Service Account Token Path enter the JSON Web Token (JWT) path. This is the path where the JWT token is mounted. The default path of this token is /var/run/secrets/kubernetes.io/serviceaccount/token.

For more information, go to Service Account Tokens in the Kubernetes documentation.

Step 2: Select Secret Engine and Version

Once you have entered the required fields, you can choose to Fetch Engines or Manually Configure Engine.

Fetch Engines

If you want Harness to automatically fetch secret engines, include this read permission for sys/mounts In the ACL policy.

path "sys/mounts"{
 capabilities = ["read"]
}

Select Fetch Engines.

Harness will populate the Secret Engine drop-down with the list of engines and their versions.

Select the engine you want to use.

Manually Configure Engine

If you don’t want to or cannot add the ACL policy (with read permission for sys/mounts) in the Secret Manager, perform the following steps:

Identify the engine version of the Secret Manager in Vault.
In Secret Engine Name, enter the name of the Secret Engine.
In Secret Engine Version, enter the engine version.

You cannot change the Secret Engine later. Harness blocks editing this setting later since there might be secrets that are created/referenced under this secret engine. Changing the secret engine might break references to those secrets.
In Kubernetes Auth Endpoint, enter the authentication endpoint, if configured. The default value is Kubernetes.

Step 3: Renewal Interval (minutes)

In Renew Interval, you can (optionally) enter how often the Harness Delegate reloads the Vault access token.

You can expect a delay during the Vault renewal. A periodic job runs to check if there has to be a renewal, resulting in a delay of no more than two minutes.

Review: Validating Non-Read Only Vault Secret Managers

To validate a non-read-only Vault Secret Manager, Harness creates a dummy secret in the secret engine.

The path of the secret is as follows:

v2 Secret Engine:

<SECRET_ENGINE_NAME>/data/\<BASE_PATH>/harness_vault_validation#value

v1 Secret Engine:

<SECRET_ENGINE_NAME>/\<BASE_PATH>/harness_vault_validation#value

The secret can fail because of various reasons.

Using the Token/App Role, the Vault authentication is not successful.
The following permission is not available in any of the policies attached to the Token/App Role. If this permission is not available, the user will not be able to fetch the list of secret engines from the customer vault and Harness will show a single option of Secret Engine named “secret” with version 2, which might be incorrect for the customer. Make sure to add the permission to a policy attached to the Token/App Role as follows:
```
         path “sys/mounts”{
            capabilities = ["read"]
            }    
```
The policy attached to the Token/AppRole does not provide the write permission in the specified path. Make sure you update the policies and permissions.

Step 4: Read-only Vault

If required by your organization's security practices, select the Read-only Vault option. This selection authorizes Harness to read secrets from Vault, but not to create or manage secrets within Vault.

Once you have filled out the required fields, select Finish.

Read-only Limitations

If you select Read-only Vault, there are several limitations on the resulting Harness Vault Secret Manager.

Also a read-only Harness Vault Secret Manager:

Cannot be used in the Add Encrypted File dialog.
Cannot create inline secrets in the Add Encrypted Text modal.

Step 5: Test Connection

Once the Test Connection succeeds, select Finish. You can now see the connector in Connectors.

important

Harness tests connections by creating a dummy secret in the Secret Manager or Vault. For the Test Connection to function successfully, make sure you have Create permission for secrets.

The Test Connection fails if you do not have Create permission. However, Harness still creates the connector for you. You may use this connector to read secrets, if you have View permissions.

Reference JSON secrets

Harness allows you to manage the lifecycle of your secrets independently by referencing JSON secrets in the vault.

For example, you can store a secret in vault with the following JSON.

{
  "key1": "value1",
  "key2": {
    "key21": "value21",
    "key22": "value22"
  },
  "key3": {
    "key31": {
      "key311": "value311"
    }
  }
}

Here are sample outputs for the respective JSONPath from the above JSON file:

test-secret (without any # key)

{
  "key1": "value1",
  "key2": {
    "key21": "value21",
    "key22": "value22"
  },
  "key3": {
    "key31": {
      "key311": "value311"
    }
  }
}

test-secret#key1

"value1"

test-secret#key2

{
  "key21": "value21",
  "key22": "value22"
}

test-secret#key3

{
  "key31": {
    "key311": "value311"
 }
}

test-secret#key3.key31

{
  "key311": "value311"
}

test-secret#key3.key31.key311

 "value311"

info

You cannot use a JSON XPath for nested keys in expressions. For example, <+secrets.getValue("account.YOUR_SECRET_MANAGER://myVault/harness/testpath/example")>.

Harness provides limited support for keys that include dots. Keys with dots only work when the key is present at first level in the JSON. For example:

{
  "key.abc": "some-value",
  "key": {
    "nested.key1": "some-value"
  },
  "key.pqr": {
    "nestedKey": "some-value"
  },
  "pqr.xyz": "some-value",
  "pqr": {
    "xyz": "some-nested-value"
  }
}

Here are sample outputs for the respective JSONPath from the above JSON file:

/path/to/secret#key.abc returns some-value.
/path/to/secret#key.pqr returns {"nestedKey": "some-value"}.
/path/to/secret#key.nested.key1 and key.pqr.nestedKey are not supported.
/path/to/secret#pqr.xyz returns some-nested-value and not some-value. (Hierarchical paths take precedence over keys with dots.)

warning

Harness does not recommend using keys that include dots and might deprecate support in future releases.

Before you begin​

Step 1: Add a Secret Manager​

Step 2: Overview​

Option: App Role​

Permissions​

Option: Token​

Option: Vault Agent​

Option: AWS Auth​

Option: Kubernetes Auth​

Step 2: Select Secret Engine and Version​

Fetch Engines​

Manually Configure Engine​

Step 3: Renewal Interval (minutes)​

Review: Validating Non-Read Only Vault Secret Managers​

Step 4: Read-only Vault​

Read-only Limitations​

Step 5: Test Connection​

Reference JSON secrets​

Before you begin

Step 1: Add a Secret Manager

Step 2: Overview

Option: App Role

Permissions

Option: Token

Option: Vault Agent

Option: AWS Auth

Option: Kubernetes Auth

Step 2: Select Secret Engine and Version

Fetch Engines

Manually Configure Engine

Step 3: Renewal Interval (minutes)

Review: Validating Non-Read Only Vault Secret Managers

Step 4: Read-only Vault

Read-only Limitations

Step 5: Test Connection

Reference JSON secrets