Overview
The financial industry has complex and evolving regulatory standards that require thorough yet flexible governance solutions to ensure consistent compliance without lapses in reliability. Because of these strict requirements, financial institutions often struggle to maintain continuous improvement in a fast-paced ever evolving technological landscape.
Organizations frequently approach governance with ad hoc methods, leading to the introduction of human error as pipeline builders must individually implement governance gates. These gates act as checkpoints to automate compliance checks on a pipeline. With each pipeline, there is an additional risk of a gate not being properly implemented. This risk multiplies by the number of compliance standards and subsequent updates to those ever-evolving standards. Even if an organization allocates additional resources to mitigate these inconsistencies, these ad-hoc governance practices still demand time and effort for the implementation and maintance of each checkpoint. This costs further development time that can be used elsewhere.
In Vertical Relevance’s Governance Gating Solution, we provide a flexible approach to meet the constantly evolving regulatory standards of the fiancial industry, while also ensuring strict uniformity across all of an organization’s pipelines. With this solution, a financial institution no longer needs to choose rigid compliance at the expense of deployment speed. Instead, this solution increases deployment speed by ensuring consistent compliance across an organization.
This governance solution creates a single source of truth for all required governance gates within an organization that eliminates inconsistencies, fragmentation, and deprecated gates amongst development teams. There is also no longer time wasted on updating each individual pipeline as governance gates are only updated a single time in a single place. A single source of truth also allows for company-wide transparency that makes auditing simpler.
Instead of ad-hoc gates added to each pipeline independently, this solution only requires each pipeline to make a single call to the governance service. The service will return a PASS, FAIL, or WARN response based on how it fares against the criteria for all of an organization’s gates. The gates are run by the service simultaneously as independent services. The modularity of this design makes it extensible and configurable for large organizations that can simply choose to add, select, or unselect a gate to use. The framework can be exposed as a service, creating a Governance-as-Service solution or it can be triggered directly as shown in the current blueprint.
Prescriptive Guidance
Before diving into the Governance Gating solution, it is important to acknowledge that while this solution can guarantee flexibility and uniformity of standards, the quality of standards is still beholden to the quality of the gates themselves. This solution is not intended to supplement quality standards but to provide a solution for implementing them consistently while providing a simple way to update and maintain them.
Another consideration is that depending on use case, different storage solutions might be more practical or cost-effective. This blueprint uses AWS S3, which is a simple and relatively cheap storage solution but in some more complex use cases something like an AWS DynamoDB database might be more practical.
Definitions
- Infrastructure as Code (IaC) – infrastructure managed and provisioned through code such as AWS CloudFormation templates or AWS Cloud Development Kit
- Governance as a Service (GaaS) – governance of security and compliance of an organization delivered through a call to a single service
- Continuous Integration and Deployment (CI/CD) Pipeline – a pipeline that automates building, testing, and deployment of code changes
- Deployment Gate – an automated evaluation or control to ensure compliance standards are met by infrastructure and application code
Best Practices / Design Principles
Logging
Gating results are used to automatically stop or allow a pipeline, but the detailed results of each gate should be available to review. Vertical Relevance recommends that the logs provide not only insights to causes of pipeline failures to allow for quick corrections, but also provide information on pass and warn cases for transparency on gating quality.
Pipeline Agnosticism
This solution is presented with an example Jenkins pipeline running on an EC2 instance. The scripts included can be used to call the Governance Gating solution from any pipeline that can use AWS command line interface. In rare use cases, there may be an additional entry point required. We recommend that the solution be independent of any pipeline and workflow orchestration tool. A key objective behind this solution is to ensure that gating changes are implemented consistently without any manual changes to individual pipelines.
Gating Strategy
We recommend a gating strategy that covers 3 major categories of gates for every code change that is released. In more complex use cases, it might make sense to build out multiple versions of this solution for different stages in the deployment pipeline. Typically, no pipeline stage should require more than one implementation of this solution. Each of the following categories is typically introduced at different stages in a pipeline.
- Static Checks and Scans – These scans will include vulnerability checks, code coverage, source code version control, and linting. Both of the gates included in this example solution belong in this category.
- Dynamic Tests – The second category of gates involve dynamic tests that include integration and performance testing. While the first category of gates make check for known risks, these sorts of gates check for the unforseen problems that are only discovered upon running the code in production-like environments.
- Post-Deployment Checks – The final category of gates are post-deployment checks to determine when rollbacks are necessary and trigger those rollbacks.
Enterprise Gating Types
Along with a strategy that targets different code testing phases, Vertical Relevance also recommends ensuring that checks cover 3 major compliance areas. The following compliance areas can be checked through each of the gating strategy phases recommended.
- Security – Security checks will ensure proper access restrictions are set and check for unintended access.
- Code Quality – Checks for code quality can ensure proper formatting and that industry-specific regulatory standards are met.
- Reliability – Gates checking for reliability can ensure that changes do not negatively impact reliability and make rollback decisions to make sure reliability is maintained.
Gating Service
There are often many deployment pipelines within a company that all require gates to ensure quality standards are met. These gates are subject to drift as new pipelines are added and these tests are copied and rewritten over and over for each subsequent pipeline. It is also difficult to make updates to all of them quickly. This solution will address these problems by having a single source of truth for all gate tests.
Components
- AWS Step Function Workflow – visual workflow service to build distributed applications, orchestrate microservices, and automate processes with AWS services
- AWS Lambda – a serverless, event-driven compute service
- AWS Simple Storage Service – object storage service offering industry-leading scalability, data availability, security, and performance
- AWS Simple Notification Service – a fully managed Pub/Sub service for A2A and A2P messaging
How it works
Before a pipeline continues to the next stage it makes a call using AWS CLI to trigger a Step Functions workflow that may return a ‘PASS’, ‘FAIL’, or ‘WARN’. If the final result is ‘FAIL’, the pipeline stops. If the result is ‘PASS’, the pipeline continues to the next pipeline phase. If the result is ‘WARN’, the pipeline will require manual approval to move forward. In the case of a ‘FAIL’ or ‘WARN’ result, the Step Functions workflow will also publish a notification to an SNS topic subscribable by interested parties.
Figure-01
Looking at the diagram above in Figure 1, the interior architecture of the Step Functions workflow becomes visible. In this diagram, there are two AWS Lambdas being used as entry points to two individual gates. All gate entry point Lambdas are run simultaneously using the information from the initial call from the pipeline. These lambdas run gate checks and pass along information on how to retrieve the results to a final lambda which will aggregate the results and determine the final response that is returned to the pipeline.
The two gate entry point Lambdas in Figure 1 represent two options for for how gate results might be handled. The results of one gate are saved directly in an AWS S3 bucket by the gate entry point Lambda which then gets a presigned url that can be used to retrieve the results. This presigned url is what is passed to the final lambda to be combined with the other gating results.
The second gate entry point Lambda in Figure 1 calls a gate that requires its own Step Functions workflow, VR’s Control Broker. After the final lambda retrieves the results using the presigned url, these gate results will require extra cleaning to standardize them to the Governance Gating format. The final lambda will call a parsing Lambda to clean and format the results before the final Lambda aggregates them.
Once the final Lambda has received all of the results, it will determine a final response for the pipeline and publish those results to the SNS topic if necessary.
Blueprint
This blueprint comprises a Governance Gating solution with two gates, Pylint and Vertical Relevance’s own Control Broker, along with a sample Jenkins pipeline run on a single AWS EC2 instance. The Jenkins pipeline also contains script files that can be repurposed to call the Governance Gating solution from other pipelines.
- Governance Gating CDK Stack – a CDK stack containing the required lambdas, state machine, and SNS topic to run two gates, Pylint and Control Broker
- Jenkins Pipeline EC2 Stack – a CDK stack containing the required infrastructure to run the sample Jenkins pipeline on a single EC2 instance
- Sample Stack – a CDK stack with a Jenkinsfile and scripts to deploy via the Jenkinsfile. This is the stack that can be used to test the gates
Benefits
- Organization-Wide Uniformity – Deployment gates across the organization will be standardized through a single source of truth approach. There is no longer the risk of overlooking a single gate when introducing a pipeline or missing the deprecation of another pipeline’s compliance.
- Fast Compliance Updates – A single change to the gating solution immediately leads to the standard being implemented organization-wide. No longer will resources be spent making updates to each pipeline individually.
- Flexible Gating Strategy – This solution has a modular architecture that allows the easy addition, removal, or update of a single gate to keep up with ever-evolving regulations.
- Governance Transparency – The gates implemented in all pipelines are viewable in a singular location making it easy to audit pipeline governance.
End Result
Governance of deployment pipelines to meet the financial industry’s high standards can be a daunting task without a systemic approach like the one taken in this solution. Vertical Relevance’s Governance Gating solution will streamline the application, maintenance, and enforcement of an organization’s governance strategy. Resources could be spent raising those high compliance standards even higher with the development of new policies and gates rather than enforcement of old standards.
