A lightweight auditable config system

Static config

Deployment config

  • Upstream URLs which might be different depending on which stage the application is (eg. staging, production, etc.)
  • Database credentials which might be different from region to region
  • Sidecar information about how to use an decoupled adjacent service for example the metrics agent
  • etc.

Runtime config

  • Feature flags: to enable/disable or adjust a certain feature of the software
  • Settings: required configs for the internal components of the software
  • Some services rely on a specially formatted header that is injected into every request at the edge layer
  • Some others rely on a purpose built off the shelf software like Unleash or SaaS solutions like Configit or Hosted Unleash
  • Some companies build their own solution

A lightweight implementation

  • Reliability: since the config is external to the software, we need elaborate verification and constraints in place to make sure that a change does not break the services in production.
  • Availability: the config should be available across regions when needed. New instances may fetch it while the old ones may poll it for any changes.
  • Auditability: it should be easy to see who changed what config and when

Tech Stack

Architecture

A rough sketch of the high level architecture
  • Clone or update the repo on a local machine
  • Change the necessary configs and run verification test suit
  • Preferably test an instance of the consumers (microservices) against the local config and verify the behavior
  • Make a PR and get it merged
  • Upon merge to master, the config is pushed to S3 where it is readily available for microservices to fetch
  • Microservices which have the config check the ETAG of their last stored config against what’s available on S3 and fetch if it the ETAGs don’t match

Practical tips

  • In our setup, only a tiny fraction of the configs needed to be adjusted so we created a GUI on top of the git repo which would allow non technical people to edit those configs. Apart from the cost of protecting and running that GUI, we were missing the audit feature of Git which by that time could easily be replaced by something like MongoDB.
  • Travis has built-in support for uploading to a S3 bucket.
  • As our config grew, we broke it into a directory structure. An open source project would combine them into one JSON ready for testing and deployment.

Conclusion

--

--

--

Sr. Staff Engineer, Knowledge Worker, MSc Systems Engineering, Tech Lead, Web Developer

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Content Delivery Network On Azure:-

Embold to integrate with Codesphere to bring advanced code analysis to the cloud

Sending Slack Alerts on Airflow Task’s Failure

Resolving DNS for Hybrid Cloud using Route 53 Resolver via terraform

Public, Private, Protected, Default -Java Access Modifiers

An interesting way to improve the performance of any PC

[PDF] ???????, ????? ? ???? ? ????????????? ???????????. ?., 1996 ????????? ?.?.

Write Logs Directly to AWS S3 from Memory Without First Writing to stdout? (Python, boto3)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alex Ewerlöf

Alex Ewerlöf

Sr. Staff Engineer, Knowledge Worker, MSc Systems Engineering, Tech Lead, Web Developer

More from Medium

Do Google’s Engineering Practices Work for a Startup?

Rows of books arranged by color on a bookshelf.

Twitter’s Tough Architectural Decision

A day selling lechugas (as backend engineer)

System Design: Designing a Distributed Job Scheduler