Best practice: How to store secrets and settings in Python project

Handling credentials, secrets and settings is a very responsible part of any project which must be safe and convinient for local developement at the same time. Any developer knows that such kind of data shouldn’t be exposed by pushing to Git repositories. This article describes the most common ways of working with sensitive data using json configuration file, environment variables and AWS SecretsManager.

Json file

Create secrets.json file in project root directory and add it to .gitingore:

$ cd <project-dir>
$ touch secrets.json
$ echo "secrets.json" >> .gitignore

In order to let other contributors know what exact secrets are required for successful project build and run, you might specify a list of keys without values in project’s documentation:

{
    "DB_HOST": "",
    "DB_PORT": 0,
    "DB_NAME": "",
    "DB_USER": "",
    "DB_PASSWORD": "",
    "PURCHASES_REPORT_KEY": "",
    "APP_AUTH_KEY": "",
    "APP_CLIENT_KEY": ""
}

Next, write a simple function to read the file and import this function from other parts of your project:

import os
import json
def read_secrets() -> dict:
    filename = os.path.join('secrets.json')
    try:
        with open(filename, mode='r') as f:
            return json.loads(f.read())
    except FileNotFoundError:
        return {}
secrets = read_secrets()

Environment Variables

This approach is very similar to the previous one. But instead of keeping secrets in a .json file, we’ll export them to environment variables.

$ cd <project-dir>
$ touch init_env_vars.sh
$ chmod +x init_env_vars.sh
$ echo "init_env_vars.sh" >> .gitignore

Script structure might look like this:

#!/bin/zsh
export DB_HOST="localhost"
export DB_PORT=5432
export DB_NAME="app_db"
export DB_USER="app_user"
export DB_PASSWORD="123Qwerty"
export REPORT_KEY="ZntLAWQH73EGsJQz"
export APP_AUTH_KEY="Basic 4ctCTDK9PMd9fyQcfUPQqKtFedLVLvbbtLUvq3jTbGeH"
export APP_CLIENT_KEY="cf6eS52P7w5GCCspxNu9e6JFGD5NDCyG"

Now you can access listed secrets in your project using os.getenv() or os.environ[]:

import os
DB_HOST = os.getenv('DB_HOST')
...

AWS SecretsManager

SecretsManager is a reliable and secure solution for enterprise applications and services. Along with other AWS services it provides a big variety of configuration options such as permissions, accesses and roles.

First of all you need to create a new Secret entry in AWS dashboard and fill out Key/value form:

At the final step you’ll be provided with all needed credentials and even code examples. So you might end up with something like this:

import os
import boto3
from botocore.exceptions import ClientError
def read_secrets(secret_name: str) -> dict:
    region_name = os.environ['AWS_REGION']  # this variable is always present in AWS environment

    session = boto3.session.Session()
    client = session.client(service_name='secretsmanager', region_name=region_name)

    try:
        get_secret_value_response = client.get_secret_value(SecretId=secret_name)
    except ClientError as e:
        print(e.response['Error']['Code'])
        return {}
    else:
        if 'SecretString' in get_secret_value_response:
            secret = get_secret_value_response['SecretString']
            secret = json.loads(secret)
            return secret
secrets = read_secrets('<secret-name>')

More examples in boto3 official documentation.

Combine

Sometimes for developement and debugging it’s easier to use secrets stored on your local machine while the app in production uses AWS SecretsManager. To not change code every time you can combine methods described above to have several secret handlers that would try to fetch data from multiple sources in specified order

import os
import json
import boto3
from botocore.exceptions import ClientError
class SecretsGateway:
    def get_secrets(self) -> dict:
        return self.from_json() or self.from_aws('<secrets-name>')
    def from_json(self) -> dict:
        filename = os.path.join('secrets.json')
        try:
            with open(filename, mode='r') as f:
                return json.loads(f.read())
        except FileNotFoundError:
            return {}
    def from_aws(self, secret_name: str) -> dict:
        region_name = os.environ['AWS_REGION']
        session = boto3.session.Session()
        client = session.client(service_name='secretsmanager', region_name=region_name)
        try:
            get_secret_value_response = client.get_secret_value(SecretId=secret_name)
        except ClientError as e:
            print(e.response['Error']['Code'])
            return {}
        else:
            if 'SecretString' in get_secret_value_response:
                secret = get_secret_value_response['SecretString']
                secret = json.loads(secret)
                return secret
secrets_gateway = SecretsGateway()
secrets = secrets_gateway.get_secrets()

Tip: good practice is to separate secrets from settings and keep them in two different storages for having more granular permissions configuration. For example, these are settings:

{
    "REGION": "cn",
    "ENV_TYPE": "prod",
    "SLACK_HOOK_REPORT": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    "SLACK_HOOK_DEBUG": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
    "S3_BUCKET_NAME": "application-users",
}

and these are secrets:

{
    "DB_HOST": "localhost",
    "DB_PORT": 5432,
    "DB_NAME": "app_db",
    "DB_USER": "app_user",
    "DB_PASSWORD": "123Qwerty",
    "REPORT_KEY": "ZntLAWQH73EGsJQz",
    "APP_AUTH_KEY": "Basic 4ctCTDK9PMd9fyQcfUPQqKtFedLVLvbbtLUvq3jTbGeH",
    "APP_CLIENT_KEY": "cf6eS52P7w5GCCspxNu9e6JFGD5NDCyG"
}

Except AWS SecretsManager, there are a lot of other third party tools and libraries designed for handling settings and secrets. If you could recommend some of them, please do it in comments and thank you for reading! :)