How to quickly replace environment variables in a file

We know storing credentials or other sensitive values in a configuration file (e.g. Kubernetes yaml file) is bad, but how can we get values easily replaced without having to do a complicated string substitution or writing a custom Python script?

I often have personal environment variable files for projects that I use to store credentials and configurations in. Before working on a project I would the corresponding configuration file into the shell session. However, these files cannot be stored in git repositories or shared with coworkers or bots. Even worse, sometimes the repositories have files in them that need to be changed, which is dangerous, because it’s easy to accidentally commit these files.

Well as it turns out, there already is a good solution and it is called envsubst. We can use envsubst to substitute environment variable placeholders inside configuration files and we can even pipe it into other commands like Kubernetes' kubectl.

envsubst < config.txt

EnvSubst

The envsubst is part of the gettext internationalization (i18n) and localization (l10n) project for unix. It's usage is quite easy and I hope this will explain it.

Some systems have gettext with envsubst preinstalled. However, if it is missing, you can install it using a package manager. For macOS you can use homebrew:

brew install gettext

Learn more about homebrew in my article about setting about your development machine in one script

Example

Let’s say, we have an existing configuration file, that want to give to someone or use with a bot. Ideally we don’t want to include credentials in that file.

# my configuration file
server: https://gitlab.com/skofgar
username: foo_user
password: mymonkey

1. Create sample configuration file

We don’t want to check that information into a git repository nor do we want it laying around or send it to someone like this , but how can we improve this? Lets replace the information we don’t want in the file with environment variables:

server: $SERVER_URL
username: $USER_NAME
password: $USER_PASSWORD

2. Configure environment variables

Then define these environment variables either by defining them in the shell session with:

export SERVER_URL=https://gitlab.com/skofgar
export USER_NAME=foo_user
export USER_PASSWORD=mymonkey

or save them to a file (e.g. .env) and then loading them into your current shell session by using source .env

Make sure to use export, otherwise your variables are considered shell variables and might not be accessible to envsubst

Read more here: https://ostechnix.com/difference-between-defining-bash-variables-with-and-without-export/

3. Substitution

To run an actual substitution, perform the following commands.

> envsubst < config.txt  
server: https://gitlab.com/skofgar
username: foo_user
password: mymonkey

It is also possible to write your substitution to a new file:

> envsubst < config.txt > confidential_config.txt

Piping substitution into Kubernetes and other tools

It is possible to pipe the output into other commands like less or kubectl for Kubernetes (k8s).

# pipe into less
>  envsubst < config.txt | less

# pipe a deployment "deploy.yml" into kubectl apply
> envsubst < deploy.yml | kubectl apply -f -

Conclusion

This is a great way to improve your Continuous Integration and Continuous Deployment (CI/CD) pipelines or just simplify your own workflow. I use this for a project where we have to replace credentials that are temporarily needed, that we didn’t want to check into the git repository and that a CI pipeline also needs to access.

Let me know in the comments what you think of this solution and if you have come across an alternative approach.

Here’s a summary of all steps combined:

# print content of configuration file
> cat config.txt
server: $SERVER_URL
username: $USER_NAME
password: $USER_PASSWORD

# load environment variables
> source .env

# replace environment variables in file content
> envsubst < config.txt
server: https://gitlab.com/skofgar
username: foo_user
password: mymonkey

# replace environment variables and write to new file
> envsubst < config.txt > confidential_config.txt

# pipe into less
>  envsubst < config.txt | less

# pipe a deployment "deploy.yml" into kubectl apply
> envsubst < deploy.yml | kubectl apply -f -

Sources

Some of the sources I used for this article are:

• envsubst for Kubernetes: https://serverfault.com/a/843883
• gettext documentation: https://www.gnu.org/software/gettext/

Originally published at Skofgar’s Blog.