A lightweight auditable config system

Static config

The simplest and most predictable configs are the ones that are statically baked into the code. These are usually easy to unit test because they don’t change after the software is deployed.

Deployment config

Deployment configs work best for variables that are tightly coupled to the concept of a deployment. For example:

  • Database credentials which might be different from region to region
  • Sidecar information about how to use an decoupled adjacent service for example the metrics agent
  • etc.

Runtime config

The business logic may need to change on the fly without having to rebuild and redeploy the software. This can include:

  • Settings: required configs for the internal components of the software
  • Some others rely on a purpose built off the shelf software like Unleash or SaaS solutions like Configit or Hosted Unleash
  • Some companies build their own solution

A lightweight implementation

Although configurability can increase software complexity, the config system is not a very complex piece of software.

  • Availability: the config should be available across regions when needed. New instances may fetch it while the old ones may poll it for any changes.
  • Auditability: it should be easy to see who changed what config and when

Tech Stack

JSON is a common format that works across architectures. It is safe to assume that our microservices will be able to fetch and parse configs in this format.

Architecture

At a high level the architecture looks like this:

A rough sketch of the high level architecture
  • Change the necessary configs and run verification test suit
  • Preferably test an instance of the consumers (microservices) against the local config and verify the behavior
  • Make a PR and get it merged
  • Upon merge to master, the config is pushed to S3 where it is readily available for microservices to fetch
  • Microservices which have the config check the ETAG of their last stored config against what’s available on S3 and fetch if it the ETAGs don’t match

Practical tips

  • In our setup, only a tiny fraction of the configs needed to be adjusted so we created a GUI on top of the git repo which would allow non technical people to edit those configs. Apart from the cost of protecting and running that GUI, we were missing the audit feature of Git which by that time could easily be replaced by something like MongoDB.
  • Travis has built-in support for uploading to a S3 bucket.
  • As our config grew, we broke it into a directory structure. An open source project would combine them into one JSON ready for testing and deployment.

Conclusion

In general it is best to reduce the runtime configuration because it is hard (if not impossible) to guarantee correctness for all permutations of config.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store