Governance Through Code for Microservices at Scale

In his great book “Building Microservices”, Sam Newman discusses Governance Through Code. He shares two techniques that he has seen work well in making it easy for developers to follow development guidelines. These are:

• Exemplars — example code implementations of the standards, policies, and leading practices that you want developers to implement. Developers like working code. Sam Newman also suggests that these examples should ideally be real-world services that might have already been developed as opposed to isolated fragments of code that have been solely written for the purpose of serving as an exemplar.
• Tailored Service Template — a template that implements most of the guidelines making it easy for developers to adhere to the guidelines. The template is a scaffolded implementation that developers use as a starting point to build their microservice. Sam raised the point about having service templates for different languages. Otherwise, you encourage developers to write their microservices in the language that the service template is available for. This clearly has both benefits and drawbacks.

What I’d like to do in this article is share with you an approach (setup and developer experience) that I have found to work well based on having implemented variations during several large Microservices transformation efforts. I also want to share some lessons learned as we implemented this approach. The following figure illustrates the approach I will cover in this article.

Figure 1 — Governing while making it easy for the developers

Overview of the key components

Product Build Squad — the Microservices Framework and the companion Developer Onboarding Tool are products that are developed and maintained by a Product Build Squad. These products leverage an Inner Source approach i.e., they take advantage of open source software development practices and culture within the organization. You will see in the lessons learned section why an Inner Source approach was beneficial and in some of my experiences absolutely necessary.

Microservices Framework — the Microservices Framework is a scaffolded Microservice that has implemented most of the organization’s standards, guidelines, and leading practices. With the SW plumbing taken care of, developers can focus on implementing the business capability of their Microservice.

I always get asked about polyglot microservices. And yes, you ideally want to use the right language for the job you are trying to do. No two programming languages are the same. You also want to give developers some freedom to make the language choice. Having said that, large enterprises typically pick two or three languages for a particular technical area such as Microservices. I find Java to be common for business microservices that might be CPU intensive and NodeJS to be used for Microservices such as the BFF (Backend For Frontend) that might be IO intensive. You will want Microservices Frameworks for the languages you choose to support. You will also want a level of parity in capabilities for all languages as well. The framework becomes a good vehicle to steer the development teams towards the selected languages assuming developers find the framework helpful and easy to use.

Developer Onboarding Tool — the Developer Onboarding Tool is used by developers to create a new Microservice using the Microservices Framework. The tool also performs other actions such as create a repository in the source code management system, checks in the code of the newly created Microservice, creates a CI/CD pipeline, builds and deploys it to the development environment among other things. In my projects we have implemented this Developer Onboarding Tool as a Web UI in some cases and as a Command Line Interface (CLI) in other cases. Both approaches seem to work equally well. We have used the Spring Initializer project as a starting point to create the Web UI. For the CLI we have used frameworks such as picocli.

Both the Microservices Framework and the Developer Onboarding Tool are stored in a source code management repository such as Git. There may be libraries that the Microservices Framework depends on stored in a binary repository such as Artifactory.

Let’s take a walkthrough of the developer experience to understand how the Microservices Framework and the Developer Onboarding Tool work together.

Developer Experience

Following the numbers in Figure 1:

1. Pull developer onboarding tool — figure 1 illustrates the case when the Developer Onboarding Tool is a CLI. The first step would be for the developer to pull the CLI from the source code repository onto their laptop. In the case of a Web UI based Developer Onboarding Tool, the developer would simply go to the URL for the tool in a browser.
2. Use to create new Microservice — the developer would use the Developer Onboarding Tool to create a new Microservice/API. The developer would need to provide necessary information that could include: (a.) Name of the Microservice — this would be used to name the project that gets generated and also the Git repository and may be other things as well.; (b.) What kind of Microservice it is — is it a REST API? What does it use to communicate with other services or systems? Etc. This will ensure the appropriate library dependencies are included.; (c.) Swagger file for a REST based API — this could result in some code generation to make it easier for the developer to complete the implementation.; (d.) Description of what the Microservice does — any documentation that could be included in the source code repository and other tooling such as an API Registry.; (e.) Etc.
3. Pull microservices framework — the Developer Onboarding Tool would pull the latest Microservices Framework from the source code repository and use it to create a Microservices project using the information provided by the user.
4. Load cross-cutting JARs — any required cross-cutting JARs would need to be loaded from the binary repository. Depending on the programming language and how you have built the framework, this would be a package manager of sorts. For example, npm for Node. The concept is similar.
5. Create repo for new microservice and push scaffolded code — once the project for the new Microservice has been created, the Developer Onboarding Tool creates a repository in the source code management system and pushes the newly generated scaffolded Microservices project to the repo.
6. Create Jenkins CI/CD pipeline and trigger build/deploy to dev environment — once the newly created Microservice is in the source code repository, the Developer Onboarding Tool creates a CI/CD pipeline for the Microservice. It triggers a build and a deploy to the development environment.
7. Deploys — the newly created Microservice is deployed to the development environment.

What the developer ends up with is a Git repository and a running Microservice in the development environment. The developer can now focus on implementing the business capability. As you can tell, the developer has a lot less to worry about with an approach like this. The Microservices Framework has all of the cross-cutting concerns implemented in a leading practices way that conforms to the organization’s standards and guidelines. They just need to focus on implementing the business capability.

This is the basic starting point that I have since evolved by bolting on additional capabilities. For example, in one case while implementing the Build to Manage approach, as part of the deployment of the Microservice to the development environment we created a basic synthetic monitoring transaction in the Application Performance Monitoring (APM) tool. In this case it was Dynatrace. The developer could than go and tweak the synthetic monitor, but it gave them a great start and also made the point that they need synthetic monitoring. I leave it to your imagination of all the other things you could do to make life easier for the developer but make sure you read the lessons learned that follow and don’t go too far.

Lessons Learned

Focus on making things easier for developers

Sometimes it is so much fun building tools that we lose sight of why we are doing this. It is to make life easier for the developers and that should be the focus. If we make it easier for the developers, we end up with Microservices that conform to the standards, guidelines, and leading practices and that are consistent in how things have been implemented.

Get developers engaged to help build the framework

In one of my projects we built the tooling and pushed it onto the development squads. I’m sure you can guess how that went. We faced significant resistance from the developers. They weren’t involved in the design or build. They didn’t know how it all worked. The structure of the scaffolded Microservice code that was generated was foreign to them. They weren’t happy to put it mildly and they didn’t want to use it. Don’t use a “build it and they will come” approach. An Inner Source approach works a lot better. The development teams get very engaged in the design and the build of the tooling. They help prioritize the features they need the most. They feel being a part of it, they feel a sense of ownership, and become advocates for it.

Don’t go too far

Don’t go too far and take all decision making away from the development squads. It’s natural to think that the more we implement and the fewer the choices developers have to make the better. However, I’ve found that there is such a thing as going too far. On one of my projects the architecture team was making all of the decisions and creating frameworks/tools to govern through code. I still recall vividly one of the developers coming to me and saying

“If you architects want to make all of the decisions and tell us how to implement things, then put your cell phone number in Pager Duty! If you want me to be accountable and be woken up at 3 AM when my code breaks in production, then I am going to make the decisions.”

A decentralized governance approach was necessary and the role of the architect needed to be as a boundary setter.

Drift can be a real problem at scale

While the Product Build Squad is designing and building the Microservices Framework and the Developer Onboarding Tool (using an Inner Source approach with contributions from other developers), development squads are already using the framework and tool. Depending on how many development squads your project/program has, you could have many Microservices with say Version 1.0 of the Microservices Framework. What happens when you have Version 1.1, 1.2, 2.0, etc. of the Microservices Framework? There is version drift. What bugs and may be security exposures exist in the older versions? How do you know what versions and how many of them are out there? How do you detect, visualize, quantify the risk, and automatically correct the drift? Keep in mind that drift is inevitable, and it is not a bad thing. It is a sign of progress. You just need to be able to keep a handle on it.

On a recent project we had 100+ developers that built 300+ Microservices. Drift in framework version became very apparent very quickly. In a future blog, I will share some drift management strategies and approaches.

Summary

Governance Through Code is an approach that plays an important role in helping developers adhere to standards, guidelines, and leading practices. There is still a need for enforcement. Enforcement should be implemented as far left as possible i.e. as early as possible in the development lifecycle. There are many approaches that can be leveraged. Hooks in the source code repository could perform checks when code is committed, stages in a CI/CD pipeline could perform checks, architecture unit tests using a framework such as ArchUnit could be performed in the IDE to ensure architectural standards, etc. A recent paper on the Seven Persistent Patterns driving cloud adoption today talks about the “governance as code” trend. Implementing governance as code and leveraging a policy engine such as Open Policy Agent (OPA) is an approach we are starting to see more of on the governance enforcement side.

I hope you have found this to be a useful read. What I have shared is based on my project delivery experiences; how we did it and lessons we learned. Your mileage will be different. I am always learning, adjusting my approaches, and experimenting with the variations. I’d love to hear your experiences with Governance Through Code approaches. Please share them in the comments section.

References

1. S. Newman, “Building Microservices”, O’Reilly Online Learning, 2020. [Online]. Available: https://www.oreilly.com/library/view/building-microservices/9781491950340/. [Accessed: 25- Sept- 2020]
2. “InnerSource Commons”, InnerSource Commons, 2020. [Online]. Available: https://innersourcecommons.org. [Accessed: 26- Nov- 2020]
3. “picocli — a mighty tiny command line interface”, Picocli.info, 2020. [Online]. Available: https://picocli.info. [Accessed: 27- Nov- 2020]
4. “Spring Initializr”, Spring Initializr, 2020. [Online]. Available: https://start.spring.io. [Accessed: 27- Nov- 2020]
5. I. Averdunk, “IBM Garage Methodology — Build To Manage”, ibm.com, 2020. [Online]. Available: https://www.ibm.com/garage/method/practices/code/build-to-manage/. [Accessed: 26- Nov- 2020]
6. S. Daya and D. Rajput, “Microservices Technical Governance”, Medium, 2020. [Online]. Available: https://medium.com/ibm-garage/microservices-technical-governance-f5aed10189d1. [Accessed: 25- Oct- 2020]
7. K. Brown, D. Lalor and S. Daya, “The Role of an Agile Architect”, Medium, 2020. [Online]. Available: https://kylegenebrown.medium.com/the-role-of-an-agile-architect-ead1e114c8f2. [Accessed: 27- Nov- 2020]
8. I. Averdunk, K. Brown and N. Emuchay, “Seven persistent patterns driving cloud adoption today”, Ibm.com, 2020. [Online]. Available: https://www.ibm.com/downloads/cas/BD6WD9EY. [Accessed: 27- Nov- 2020]
9. P. Gafert, “Unit test your Java architecture”, ArchUnit, 2020. [Online]. Available: https://www.archunit.org. [Accessed: 27- Nov- 2020
10. “Open Policy Agent”, Openpolicyagent.org, 2020. [Online]. Available: https://www.openpolicyagent.org. [Accessed: 27- Nov- 2020]