Shared Repo

A tale of evolution

There’s a limit to how much complexity even a group of humans can handle
  • Separation of concerns: group relevant pieces together
  • Loose coupling: minimize the interaction surface between those parts to allow internal change with minimal external consequences
S=Size, C=Complexity. Breaking the software into smaller components allows each individual component grow in complexity while still manageable by a group of humans

Web as a runtime

“Any application that can be written in JavaScript, will eventually be written in JavaScript.” — Jeff Atwood

Our massive app

The domain

One problem, multiple solutions

The company way paying multiple times to solve the same problem
  • Duplication of work: if one brand had a feature that another one needed, the other brand had to reimplement it from scratch due to incompatible underlying technologies.
  • Resource sharing: due to technology fragmentation, it was hard to share the engineers and experience across brands.
  • High cost: every brand had their own way of fixing bugs, building, testing and deploying the product which meant the company ended up paying for multiple solutions to the same problem.

Take 1: The SDK

The SDK contained some common code that was used by the actual sites

Advantages of the SDK

  • Code sharing: SDK is a feasible way to share code to multiple consumers.
  • Abstraction: Building SDK is a good exercise in loose coupling because it enforces thinking about the abstraction layers and separation of concerns.

Disadvantages of the SDK

  • Ownership: The Core team was solely responsible for the SDK. This meant for the most part, every feature request or bug fix had to go through them. The brand developers had to negotiate every use case and fight for prioritization in an ever growing backlog. The Core became a bottleneck.
  • Mandate: So long the migration to adopt the SDK was optional, the brands didn’t see a need to prioritize it. The Core team didn’t have any mandate to enforce the adoption either.
  • Innovation: if the backend introduced feature A that was incompatible with SDK version 1.1, but supported in SDK version 2.0, they could not roll it out until every site had upgraded their SDK to version 2.0. This hindered the pace of innovation.
  • Incomplete SDK QA: creating an SDK is hard especially because the code is supposed to run in an unknown context. An SDK is inherently impossible to test without a context, so it’s often tested against a mock context which doesn’t really guarantee a quality of any kind in its final real habitat.
  • Time consuming site QA: whenever a new version of the SDK was released, the sites had to go through the daunting task of ensuring that everything works as expected even if the resulting PR would be just a version bump! Due to loose coupling nature of the SDK, the integration testing was the first chance of proper quality assurance.
  • Feasibility: Some sites were technologically incompatible with the React which was used by the SDK. As far as they were concerned the problems that this initiative set out to solve were untouched. The alternative was to do a massive refactoring which was not favoured by the developers who created that solution in the first place.
  • Cost: As the sites were allowed to keep their fragmented technologies, the company continued paying for all those fragmented implementations and the Core team on top of that! All while the company was facing fierce competition from Google and Facebook.
  • Complexity: There’s a whole lot of ceremony for packaging, publishing and consuming a package (or in this case tens of packages): more build & deploy pipelines, private NPM registry, tooling for automating the updates (think dependobot) and user documentation (cause the code is “hidden” or hard to see).
  • Heavier artifacts: packaged as independently consumable code, the individual deliverables cannot have any hard assumption about the context they are used. Therefore they bundled all of their dependencies. If two packages used lodash, it either had to be a peer dependency which meant extra steps or in case of TypeScript or WebPack it meant duplicated boilerplate that the transpilers inject into the result. This wasn’t a major issue, but inefficient at best. Size still matters in the front end world.

SDK was not the best abstraction for the job!

Take 2: The Platform

  • felt forced to a new tech stack
  • lost the autonomy they were used to.
  • src/core: the platform and default components
  • src/sites/BRAND_NAME: site configs and any site-specific component. The sites could also override the core components in their folder
A high level diagram of code structure

The birth of a community

The brand developers and the core team created a vibrant community

But isn’t it just a monorepo?

But isn’t it just a monolith?

Why not micro frontends?

  • Integration: the sum of all these micro frontends integrates in the browser. This is very similar to the issues that the SDK has: the code is not developed in its final context. In fact the micro frontends are even worse because they really integrate in the end user’s browser while the SDK users deal with that headache at development time.
  • Overhead: each of those micro frontends usually runs in their own browser context (eg. iframe or custom elements) which are isolated from each other. Therefore each comes bundled with their dependencies. There’s also a communication overhead between those parts that is more expensive than a simple function call. The end user’s browser downloads and executes more code. This hurts the data usage and battery life of the mobile users the most. It’s not the most efficient architecture but it has its place.
  • Fragmentation: one of the possibilities that micro frontends unlock is to allow each part to use their own tech stack. While this level of autonomy can potentially lead to using the right tool for the job, it can also lower the bar for diverging tech stack and ending up with a fragmented tech stack that hurts human resource sharing. Guaranteeing a consistent UX commands rigorous work to ensure that when those parts use different technologies, that implementation detail is hidden from the end user.

Advantables of the share repo

  • Quality: Testing frontend SDKs in isolation is much harder than testing everything together in a real setup as the end user interacts with it. You’re continuously integration mode. Also, there are simply more eyes on the same code base: “given enough eyeballs, all bugs are shallow” Linus’ law
  • Consistency: There’s a fair bit of boilerplate that is duplicated (and need to be kept in sync for consistency) across individual repositories for example: Linting rules, .editorconfig, documentation, debugging profiles, setting up the test framework, build & deploy pipeline.
  • High cohesion: many features or bug fixes touch code that would otherwise be scattered across different repos and suffer from the asynchronous deployments. By contrast, in a shared repo, these changes could come in one cohesive PR which made the code much easier to reason about.
  • Effectivity: when working on an SDK it is easy to lose sight of the big picture because by definition the focus is inside the abstraction. With a foggy holistic picture, the developers are not as effective because the consequences of their choices are not immediately tangible. With shared repo on the other hand, they are constantly in integration mode where the consequences surface just like the final product is experienced by the end users.

Disadvantages of the share repo

  • Time: since there is more code, everything takes longer: linting, building, testing, etc.
  • Review fatigue: The commit rate is too high, so a clear ownership model is needed along with tooling to help with that (see how Chromium team uses codeowner)
  • Gatekeeping: People become specialized in parts of the code which is not a bad thing on its own but for quality work they may need to collaborate tightly. A single architect or tech lead may have difficulty to be on top of the latest state of the repo, but that’s the model I recommend for avoiding anarchy
  • Code sharing: If some code has other external usage, developers may end up copying it because the SDK workflows and versioning are removed at that point. Another alternative is deploying it separately as a package which turns it into a monorepo (with multiple deployables).
  • Overrides: the structure allows any site to override the shared functionality as they see fit. Despite great flexibility, it created complexity: reading the code, one cannot easily deduce what code is going to execute because the config in production could change that, it was hard to reason about.

Take 2.1: Ownership

  • All brands used the common code that lived there. But it was named after the Core team. So they assumed that any change in that folder should be checked with the Core team where in reality, if they broke something, they would most probably break another brand.
  • The Core team acknowledged the importance of that folder but as the brand developers did most of the work for their respective site and evolved that code, the Core lost touch with the actual brand problems and it was hard to contribute in meaningful ways when they asked for it.
  • As this folder grew, it got harder for the Core team to keep tabs on its latest evolution. We felt that we cannot own this properly and at the same time spend time on our SRE and DevOps responsibilities.
  • People would change, the code may end up without an owner
  • The functionalities may be spread across different directories which may be owned by different people. This makes the PR review harder

Conclusion

--

--

I’ve moved to Substack: https://blog.alexewerlof.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store