Setup Hashicorp Vault in EC2 and Connect to Mulesoft
This blog will take you through to create your own Hashicorp Vault Server in EC2 so that you can push property from this vault to your Mulesoft App
Hashicorp Vault, Vault Project, Mulesoft, Mule Properties, Vault Mule Properties
Mulesoft | Anupam Chakraborty | Jun 27 2020 05:08 PM
Introduction
Today, I will explain how to set up a simple Hashicorp Vault in an EC2 Server and then access it from our computer so that it can provide properties to a Mule Application. There is amazing documentation around this in the Vault Project Website, however, this would be more of a step by step code with exactly what you need to get going.
Prerequisite
- Amazon AWS Account to build an EC2 system. I will use a Basic T2.micro Free tier EC2 System. Take a close look at your AWS account to make sure you are not incurring charges.
- Mule ESB running in Local Computer.
So, without any further ado, let us build our Vault and Connect to Mule.
Step 1: Setting up a t2.micro EC2 Server.
I will not go through the steps for how to bring up an EC2 Server. If you need help around this. Look at my previous blog Amazon EC2 Cloud with LAMP Stack and Loopback CRUD API. I will use the Amazon Linux 2 AMI (HVM), SSD Volume Type with t2.micro Instance, and use the standard 8 GiB Storage. The important part is the security group. We will use the SSH on port 22 along with a custom TCP port 8200 for this box.
Once this is up and running, we will SSH into this instance using an SSH key.
For me this came as
$ ssh -i "SlalomEC2.pem" ec2-user@ec2-3-87-191-61.compute-1.amazonaws.comBecause we are login for the first time, we would update the system by running
$ sudo yum updateStep 2: Install Vault in our EC2
Next, we will download 2 packages: the vault itself and consul for the backend from the respective hashicorp website. They are downloaded as zip files. We will then extract them.
$ wget https://releases.hashicorp.com/vault/1.4.2/vault_1.4.2_linux_amd64.zip
$ wget https://releases.hashicorp.com/consul/1.8.0/consul_1.8.0_linux_amd64.zip
$ unzip vault_1.4.2_linux_amd64.zip
$ unzip consul_1.8.0_linux_amd64.zipat this time, there would be 4 files in my home directory
Then we would move them to the bin folder so that they can be executable. Finally, we will delete the zipped file that was downloaded.
$ sudo mv vault /usr/local/bin/vault
$ sudo mv consul /usr/local/bin/consul
$ rm vault_1.4.2_linux_amd64.zip
$ rm consul_1.8.0_linux_amd64.zipAt this time Vault is running, we can verify this with the command
$ vault version
Vault v1.4.2Step 3: Setup the Vault Config
Now let us set up our Vault so that we can connect it from our computer.
We need to create a file name something like vaultConfig.hcl
$ touch vaultConfig.hcl
$ nano vaultConfig.hclAnd fill up the file with the following text
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
ui = true
storage "consul" {
address = "127.0.0.1:8500"
path = "vault/"
}
disable_mlock=trueyou can make sure if you have set up the file correctly by issuing the following command
$ more vaultConfig.hclStep 4: Bring up the Vault Backend
If you see the above config, we would be using the consul as our storage. We can start vault without any backend, and it will store the data in the memory. However, we would rather use the Consul storage, which is much more reliable and is production-ready.
To start the consul, simply issue the command
$ consul agent -dev &The & at the end will make sure that we are running the consul in backend and hence, even if we break the script using Ctrl+C, the consul will keep running. This can be checked using the following command
$ ps -ef | grep consulStep 5: Bringing up the Vault Server.
With the configuration in place, starting the server is simple, as shown below. Modify the -config flag to point to the proper path where you saved the configuration above.
$ vault server -config=vaultConfig.hcl &
==> Vault server configuration:
Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: consul (HA available)
Version: Vault v1.4.2
==> Vault server started! Log data will stream in below:Yohoo, we have the vault up and running now. But wait, it is a vault. So, we would need to unseal it.
Step 6: Unseal the vault
The next step is to unseal the vault. To do that, we should first set up the Vault Server as an Environment variable and then need to use the following command.
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ vault operator init
Unseal Key 1: qz28vdxHN/3RvJ7KlDInFDpZQCKkr0RgeYIDTQJu3glY
Unseal Key 2: msJbFKKtf7tN0Vb7hs+yEBXE20wz1Syqlirp3iUcGYDA
Unseal Key 3: TAeEbw5XjEhzzBckmkz/DNjWcJ7u551gSowyEfTFdKyO
Unseal Key 4: CUt1uKw+mmbBCImgPC1o8r0N3Z7VhgwKDw7Vo19lu9O5
Unseal Key 5: XRJodDat7J9qkvgQ3fF4IYcfpd1b3gHSMJQgRmVA/Osd
Initial Root Token: s.1PUGwN7lQ9ZGWGTJ5nZKunms
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.We should keep a note of these keys, this would be used to unseal our vault.
For that, we should execute the command vault operator unseal 3 times with 3 of the key above.
$ vault operator unseal
Unseal Key (will be hidden):
Key-Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce 7df52188-ed15-1337-7b39-b73bb3d03489
Version 1.4.2
HA Enabled true
$ vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce 7df52188-ed15-1337-7b39-b73bb3d03489
Version 1.4.2
HA Enabled true
$ vault operator unseal
Unseal Key (will be hidden):
[INFO] core.cluster-listener.tcp: starting listener: listener_address=0.0.0.0:8201
[INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
[INFO] core: vault is unsealed
[INFO] core: entering standby mode
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.4.2
Cluster Name vault-cluster-95c51d21
Cluster ID 25d8d6a8-86d6-8320-3e2d-1545e291d59b
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
[ec2-user@ip-172-30-0-151 ~]$ 2020-06-27T03:00:52.554Z [INFO] core: acquired lock, enabling active operationOur vault is now unlocked and ready to be used.
Step 6. Login to vault and create a Secret Engine
We should remember the token that we received when we used the command vault operator init. This token will be used to login to the vault. So in my case, I would use the following command
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ vault login s.1PUGwN7lQ9ZGWGTJ5nZKunms
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.1PUGwN7lQ9ZGWGTJ5nZKunms
token_accessor n3IvnysY1REGdZAPaHndI1Cj
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]Create a KV-V2 Secret Engine
Now, that I have logged in, I should create a Secret Engine so that I can use the secret engine in my Mule Project.
We have to create a secret engine in the vault to store different secrets. If we use any other secret engine, we would get an error with code 403.
So, let us use the following command to create a secret engine.
$vault secrets enable -path=mySecret -description='This is the demo Secret Engine' kv-v2
Success! Enabled the kv-v2 secrets engine at mySecret/This should create a key value version 2 secret engine, that we would use for our requirement.
You can check all the secret engines in the vault system using the following command.
[ec2-user@ip-172-30-0-151 ~]$ vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_cee084dc per-token private secret storage
identity/ identity identity_40dc37fc identity store
mySecret/ kv kv_ee4a9dce This is the demo Secret Engine
sys/ system system_292962f2 system endpoints used for control, policy and debuggingStep 7: Put and Get Secret from the Engine
Now that we have created a secret engine, let us first create a test secret. I will call it muleMessage.
$vault kv put mySecret/muleMessage welcomeMsg='Welcome to MuleSoft'
Key Value
--- -----
created_time 2020-06-27T03:15:44.972268511Z
deletion_time n/a
destroyed false
version 1You can see this using the following command
$ vault kv get mySecret/muleMessage
====== Metadata ======
Key Value
--- -----
created_time 2020-06-27T03:15:44.972268511Z
deletion_time n/a
destroyed false
version 1
======= Data =======
Key Value
--- -----
welcomeMsg Welcome to MuleSoftYou can also, browse the secret like this:
$vault kv list mySecret
Keys
----
muleMessageYou can get a specific value directly using this command
$ vault kv get -field=welcomeMsg mySecret/muleMessage
Welcome to MuleSoftNow let us create another secret with 2 value.
$vault kv put mySecret/httpListener host=0.0.0.0 port=8081
Key Value
--- -----
created_time 2020-06-27T03:20:47.156553753Z
deletion_time n/a
destroyed false
version 1If you do a get on this now
$ vault kv get mySecret/httpListener
====== Metadata ======
Key Value
--- -----
created_time 2020-06-27T03:20:47.156553753Z
deletion_time n/a
destroyed false
version 1
==== Data ====
Key Value
--- -----
host 0.0.0.0
port 8081Step 8: Accessing the vault from our computer.
To access the vault from our computer, we should install the vault from the hashicorp website: https://www.vaultproject.io/downloads. This has a download for different operating systems. Once we have installed this on our computer. If we execute the following command, we should be able to access the secrets directly from our computer instead of SSH.
$ export VAULT_ADDR='http://<EC2 IP Address>:8200'In my case, I had to do like this
anupam.chakraborty@mbpr ~ % export VAULT_ADDR='http://3.87.191.61:8200'
anupam.chakraborty@mbpr ~ % vault login s.1PUGwN7lQ9ZGWGTJ5nZKunms
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token s.1PUGwN7lQ9ZGWGTJ5nZKunms
token_accessor n3IvnysY1REGdZAPaHndI1Cj
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]Now you should be able to hit all the kv get and put from your pc directly.
anupam.chakraborty@ mbpr ~ % vault kv get mySecret/httpListener
====== Metadata ======
Key Value
--- -----
created_time 2020-06-27T03:20:47.156553753Z
deletion_time n/a
destroyed false
version 1
==== Data ====
Key Value
--- -----
host 0.0.0.0
port 8081Step 8: Accessing the UI
Here comes the best part of this vault. There is a Web-based user interface available for us to use. Simply go to http://<EC2 IP Address>:8200/ui
This gives a nice frontend and you should be able to access all the vaults that you have created using this UI.
Step 9: Connect to the Vault from MuleSoft
Now, we would use the Vault to pass the properties to our mule app.
In order, to do this we would have to use the Vault Property Provider connector. The source code is available here at https://github.com/avioconsulting/mule-vault-properties-provider.
We can clone this and then publish this into our exchange. Then finally add the dependency using the following pom xml
<dependency>
<groupId>${orgId}</groupId>
<artifactId>vault-connector</artifactId>
<version>1.0.0-SNAPSHOT</version>
<classifier>mule-plugin</classifier>
</dependency>And here is the simple flow to use this vault
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:vault-properties-provider="http://www.mulesoft.org/schema/mule/vault-properties-provider" xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd
http://www.mulesoft.org/schema/mule/vault-properties-provider http://www.mulesoft.org/schema/mule/vault-properties-provider/current/mule-vault-properties-provider.xsd">
<vault-properties-provider:config name="Vault_Config" doc:name="Vault Config" >
<vault-properties-provider:token-connection vaultUrl="http://3.87.191.61:8200" engineVersion="v2" vaultToken="s.1PUGwN7lQ9ZGWGTJ5nZKunms" />
</vault-properties-provider:config>
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" basePath="">
<http:listener-connection host="${vault::mySecret/httpListener/host}" port="${vault::mySecret/httpListener/port}" />
</http:listener-config>
<flow name="homeFlow" >
<http:listener doc:name="Listener" doc:id="6a565a15-8dc4-44d4-92b3-4f39c1c7d7e2" config-ref="HTTP_Listener_config" path="/sayHi/{name}"/>
<set-variable value="${vault::mySecret/muleMessage/welcomeMsg}" doc:name="prop" variableName="prop"/>
<ee:transform doc:name="Transform Message" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
msg: "Hi " ++ ( attributes.uriParams.name default "Jon Doe") ++ ". " ++
(if (now().hour >= 12 ) "Good Afternoon" else "Good Morning") ++
". " ++ vars.prop
}]]></ee:set-payload>
</ee:message>
</ee:transform>
</flow>
</mule>Finally Thank you for going through this blog. Let me know if you were able to make this to work. Thank you once again for reading my blogs.
Comments:
Leave a Comment:
Please login to post comment into this page.