Governing API Resource Modelling

By Muneeb S. Ahmad, Microservices Architect, Global Business Services and Shahir A. Daya, IBM Distinguished Engineer and CTO Financial Services

Muneeb and I have worked together on some large banking client transformation programs that have targeted a Cloud Native/Microservices architecture. We were conforming to industry-standard data models such as the Banking Industry Architecture Network (BIAN) model in our client work. Ensuring that our APIs, developed by many full-stack squads, were consistently applying the industry-standard model was challenging. On a recent client engagement, we have devised an approach to ensure consistency in applying the BIAN object model (BOM). That is the subject of this article. We want to share with you this approach that we are starting to see some success with in governing the API resource modelling aspects of API development.

There are generally two main approaches to building APIs in terms of how requirements get captured; “Design-First” and “Code-First.” We are not going to debate which one is right or wrong as both have pros and cons. Still, in our experience for the business domain, which falls under the high level of regulatory compliance like banking, insurance, etc., the “Design-First” approach is relatively a better fit for the following reasons:

1. Because the APIs get documented, shared, and even virtualized (or mocked) before writing any code, you can get early feedback from API consumers regarding domain model consistency and regulatory compliance requirements.
2. Before writing any API code, the domain entities get modelled using industry-standard architecture frameworks (like BIAN, ISO 20022, etc. for banking applications)

When using a “Design-First” approach, during the designing of the API specification (i.e. the Swagger file or Open API spec), is the perfect time to govern by providing an easy means for the designer to simply pick from an object model catalog that contains the industry-standard aligned object model.

End-to-End Modeling to API Approach

The following figure illustrates the approach we have had success with and that we will cover in this article.

Figure 1: API Resource Modeling Approach

In a previous article, we proposed how the Microservices Technical Governance model can bring a healthy balance between autonomy and standard enforcement in developing Microservices. We also discussed Governance Through Code for Microservices at Scale and discussed how important it was to make it easier for developers to do the right thing. Figure 1 (above) shows how a similar approach can be applied in the domain/resource modelling aspect of API development. Even though we show three separate teams for API Development, API Modeling, and API Governance, all contributing to the end-to-end modelling to API approach, a single team would work fine. Still, we only recommend that for smaller projects and organizations.

Key steps and components

The following are the key steps and components of the approach:

Domain Modeling Tool: A data modelling tool like Erwin is used to create domain models. This step is done by individuals who are considered domain experts and are good at establishing Entity-Relationship (ER) diagrams. The modelling tool should satisfy the following criteria:

• Support re-use of entities and attributes across domains
• Provide export functionality of model metadata as CSV or similar data formats preserving entity relationships

Model Metadata Files: The exported metadata model files are pushed to a dedicated git repository.

Model Transformer and Upload Service: The model (metadata) files are fed to a Transformer and Upload service. This service can be a REST API or CLI, which performs the following functions:

• It converts the data format from CSV to a format of how the data is stored in the Model Catalog database.
• It also invokes an endpoint of Model Catalog APIs to insert or update a record in the Model Catalog database.
• The “Trigger” shown in Figure 1 can be a WebHook or a Pipeline step and is responsible for detecting model metadata files being pushed to the Model Catalog repository.

Model Catalog APIs: These APIs play the role of System of Record for the API Resource Modeling solution. These APIs are consumed by various components, including Upload, Validation, and UI toolsets. The database shown as a Model Catalog database can be relational or non-relational. In a recent project, our team decided to use MongoDB for storing model metadata.

API Specification Design Tool: One of the critical steps in API development, especially in the Design-First approach, is to design API specifications (or swaggers). The API Specification Design Tool is a custom-built UI very similar to SwaggerHub. We’ve had to custom-build the UI so that we could integrate it with the Model Catalog.

API Specification Validation Service: This step is important to ensure that teams follow standards around API domain modelling. The basic theme is to fetch the contents of API specifications (i.e. the swagger file) which is pushed to the application git repository, and compare its model directives with the models stored in the Model Catalog. The validation service may also interact with the API Governance database or system to determine if the application/repository qualifies for model validation. Below are some key points of this step.

• After composing the specification (swagger) file, the API development team pushes it to the application repository.
• A step in the CI pipeline invokes the specification validation service.
• The validation service checks if the application qualifies for enforcing the model validation standards by evaluating enterprise governance rules, including exceptions. These rules are generally fetched from the API Compliance database.
• The validation service fetches the target models from the Model Catalog by invoking the Model Catalog APIs.
• The validation service then compares the model defined in the input specification (swagger) file with the one stored in the Model Catalog. The comparison is quantified as a “matching score” by comparing domain names and their attributes. The CI step is marked as a success if the score satisfies a pre-defined threshold. This validation service is configurable in terms of how strict or relaxed the rules are in comparing domain and attribute names and computing the “matching score.”

Model Metadata Artifacts

The API resource modelling process starts with a tool used to draw model (entity relationship) diagrams. As mentioned earlier, this tool must support exporting the model artifacts in some program-readable data formats (CSV, JSON, YAML, XML, etc.). Depending upon the exported format, the model metadata can be a single file or a set of files with information on model entities, attributes, and relationships. Below is an example where model metadata artifacts are exported as CSV files (shown as tables).

Figure 2: Model metadata artifacts in CSV format

API Specification Design Tool

One of the objectives of automating API resource modelling is to feed the API models to the API Specification design tool. As mentioned earlier, this integration becomes easier if the specification (swagger) design tool is custom built where the UI layer can consume the Model Catalog APIs to fetch the resource models and their attributes. Figure 3 (below) illustrates how UI elements can be implemented to search domain names and their associated models (entities) and to add them to the API specification (swagger) document.

Figure 3: A sample UI design showing how the Model Catalog can be integrated with API Specification Design tool

Model Catalog Database

The model metadata can be stored in a relational or non-relational database. Here is some crucial information that needs to be held:

• Entity name, description, version, etc.
• The domain name associated with the entity
• List of entity attributes including name, type, description, etc.
• List of other entities associated with the subject entity.

Summary

A healthy amount of governance is necessary when you are building at scale and have a large team of developers. The key is to make it easy for the developers to make the right choices, and that’s the goal of the custom built API Specification Design Tool integrated with a Model Catalog.

I hope you have found this article to be a worthwhile read. What Muneeb and I have shared is based on our project experiences to date. We continue to learn, so please share with us in the comments section your experiences in governing adherence of your APIs to an industry data model.

References

1. S. Daya and D. Rajput, “Microservices Technical Governance”, Medium, 2021. [Online]. Available: https://medium.com/ibm-garage/microservices-technical-governance-f5aed10189d1. [Accessed: 04- May- 2021]
2. S. Daya, “Governance Through Code for Microservices at Scale”, Medium, 2021. [Online]. Available: https://shahirdaya.medium.com/governance-through-code-for-microservices-at-scale-aea4b659f44a. [Accessed: 04- May- 2021]
3. “BIAN — Banking Industry Architecture Network”, BIAN, 2021. [Online]. Available: https://www.bian.org/. [Accessed: 04- May- 2021]
4. P. Ritchie, “Mapping a Domain Model to RESTful”, Peter Ritchie, 2021. [Online]. Available: https://blog.peterritchie.com/Mapping+a+Domain+Model+To+RESTful/. [Accessed: 04- May- 2021]
5. P. Sturgeon, “API Design-First vs Code First”, Apisyouwonthate.com, 2021. [Online]. Available: https://apisyouwonthate.com/blog/api-design-first-vs-code-first. [Accessed: 04- May- 2021]
6. “SwaggerHub”, 2021. [Online]. Available: https://swagger.io/tools/swaggerhub/. [Accessed: 04- May- 2021]