Written by Meysam Azad

Intro

Over the last decade or so, due to the technological advancement in operations and tools such as CI/CD, containers, IaaS, etc., more and more people (or software engineers we shall say) are familiar with the operation part of the business.

(Yay!🥳)

This has made it easier and easier to talk about things such as configurations to the developer team and the way they employ it in their code (Take a look at the 12-factor app if you don’t know what I’m talking about).

Though there is still a long journey ahead of us, we have come a long way.

In this article, I’m trying to propose a common problem, namely configuration, and provide a proper solution on how to address that.

So, what is it anyway? And who cares?

When it comes to configuration, each developer has its preference on how to read some values that might change based on different deployments/environments.

Some might use the .env file; I know for a fact that the JavaScript guys are more interested in having .env.ENV_NAME in their source code.

Other people would go for other names; I’ve seen .envlocal , .env.debug , .env-test to mention a few.

What’s the worst part about all of this? It’s that it’s wrong right to its bone. The configuration of an app should never be committed to the source code. To quote the 12-factor guys:

A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.

The dev guys should only have some examples of the required and optional configuration that their app needs; some files similar to this would be nice: .env-example.

Ideally, the content of such a file would be something like this:

ENVIRONMENT=CHOOSE_FROM_TEST_DEV_PROD
REDIS_URI=CHANGE_THIS
MONGO_URI=CHANGE_THIS

And so on.

You should NEVER, I repeat, NEVER commit the configuration to either the source code or the image of your app (I’m talking about Docker image but any similar concept applies the same).

As a DevOps Engineer, I’m responsible for providing those values (whether required or optional) to your application and when and where I forget to do so, your app should (or better, must) fail and complain about the missing value; something similar to this perhaps right before the runtime:

ValidationError: 1 validation error for Config
MONGO_URI
  field required (type=value_error.missing)

Apart from the biggest mistake I’ve seen people make when committing the env file to the source code, I’ve also witnessed people putting the env file to the image of the app; I’m saying this loud and clear so that everyone can hear: IT IS WRONG!

Never place your env file inside the image of your app because it makes the behavior of the app nondeterministic.

Any person (with the right amount of access privilege) should be able to run your application with as many instances and as many different configurations as he/she desires without the need to tweak some file in your image or do some other weird stuff to overwrite a, let’s say, MongoDB URI.

That means, with the identical source code, I, as an operation guy, should be able to run your application on either or all of the environments I see fit e.g. testing, staging, development, production, etc.

Again, let me quote the 12-factor guys:

A codebase is transformed into a (non-development) deploy through three stages:

1. The build stage is a transform which converts a code repo into an executable bundle known as a build. Using a version of the code at a commit specified by the deployment process, the build stage fetches vendors dependencies and compiles binaries and assets.

2. The release stage takes the build produced by the build stage and combines it with the deploy’s current config. The resulting release contains both the build and the config and is ready for immediate execution in the execution environment.

3. The run stage (also known as “runtime”) runs the app in the execution environment, by launching some set of the app’s processes against a selected release.

As a final touch, now that I have outlined the problem clearly, I plan to provide [an opinionated] solution.

As I am a Python engineer, I’m gonna talk about a library that I adore, admire and support (both spiritually and financially) on the Python ecosystem but you won’t get into trouble finding the equivalent in your language.

Lo and behold Pydantic 🥁

Pydantic is my personal preference when it comes to validation. But aside from all the cool features, it provides for a robust production application, it also comes with a Settings API which you can employ in your app to avoid having to read configurations from multiple places and therefore confusing both yourself and the operations team.

Before diving right into the code, let us review the exact requirement one more time, just to make sure that we realize what we are trying to solve here:

Any DevOps guy has to be able to run the same app as many times as he/she desires, on the same machine or many, with different sets of configurations (or environmental variables).

So, enough talking; “talk is cheap, show me the code” 😍

"""
priorities:
1. arg when instance is initialized
2. `export ENV=VAR` from shell
3. `.env` file or whatever else you specify
4. default value if provided
"""

import os
from typing import Optional

import pydantic
import pytest


class BaseSettings(pydantic.BaseSettings):
    class Config:
        env_file = ".env"
        env_file_encoding = "utf-8"
        case_sensitive = True


class Config(BaseSettings):
    REQUIRED: str
    OPTIONAL: Optional[str] = "default"
    FLOAT: float = 1
    INT: int = 1
    BOOL: bool = True


def test_required_complains_if_missing():
    with pytest.raises(ValueError):
        Config()


def test_optional_is_optional():
    value = "required"
    c = Config(REQUIRED=value)
    assert c.REQUIRED == value
    assert c.OPTIONAL == "default"


def test_optional_can_be_set_to_none():
    c = Config(REQUIRED="dummy", OPTIONAL=None)
    assert c.OPTIONAL is None


def test_float_is_float():
    c = Config(REQUIRED="dummy", FLOAT=1)
    assert isinstance(c.FLOAT, float)


def test_int_is_int():
    c = Config(REQUIRED="dummy", INT=1.9)
    assert isinstance(c.INT, int)


def test_bool_is_bool():
    c = Config(REQUIRED="dummy", BOOL="1")
    assert isinstance(c.BOOL, bool)


@pytest.fixture
def write_env_file():
    """
    Every test that has `write_env_file` in their arguments, will be able to
    see a file next to the current directory with the name `.env` and the
    content `REQUIRED=env`.
    This file will be removed once the test is done; the magic is the use of
    `yield` in the fixture.
    """
    with open(".env", "w") as f:
        f.write("REQUIRED=dummy")
    yield
    os.remove(".env")


@pytest.fixture
def export_env_var():
    os.environ["REQUIRED"] = "even-dummier"
    yield
    del os.environ["REQUIRED"]


def test_env_var_is_working(export_env_var):
    c = Config()
    assert c.REQUIRED == os.environ["REQUIRED"]


def test_env_file_is_working(write_env_file):
    with open(".env", "r") as f:
        value = f.readline().split("=")[1]
    c = Config()
    assert c.REQUIRED == value


def test_env_var_overrides_env_file(write_env_file, export_env_var):
    c = Config()
    assert c.REQUIRED == os.environ["REQUIRED"]


if __name__ == "__main__":
    pytest.main()

The language is Python and the syntax is pretty straightforward so you won’t have much trouble picking up what is going on; therefore, there is no point in me wasting words and your time around it!

You can easily run this file to make sure that the promise is held (python3 test_pydantic.py). Using this style for your settings, any kind of operations is possible with different sets of values provided as the config for your app.

If you are interested to know more, head out to the other article that summarized the 12-factor app shortly and sweetly below 😁.

https://medium.com/licenseware/stop-committing-configurations-to-your-source-code-fb37be351492

Written by our brilliant Meysam Azad! – Original Article can be found here