Overview
Open Policy Agent is a tool for validating IAC and application manifests, but its syntax can be challenging to understand and write. The OPA Policy Generator aims to provide a user-friendly interface to OPA policy generation for Terraform AWS-based IAC code.
Problem Statement
Open Policy Agent is quickly becoming the defacto tool for validating IAC and application manifests to ensure they comply with security and best practice standards. OPA policy documents are written in the Rego language, which lacks some of the basic logical operator functions found in most modern programming languages. These omissions make the language unintuitive for even seasoned developers and security professionals. OPA Policy Generator tries to alleviate this by providing a means of developing simple, straightforward OPA policies using natural language as an input.
| Problem | Details |
| OPA Syntax is not Intuitive | Rego lacks built in ‘and’ and ‘or’ logical functions, requiring the developer to implement this functionality using things like list comparisons and overloading function which |
| IAC Resource Attributes not Standardized | To check an attributes value for a resource you need to know the exact name of the attribute key for the resource. This requires referencing the IAC documentation to ensure the spelling using in the document is correct. |
| AI Policy Generation is Inaccurate and Training is Prohibitively Expensive | Using a general-purpose model to generate OPA policies has proven to be inaccurate. Further training to improve reliability quickly becomes cost prohibitive. |
Solution
Background
OPA Policy Generator is a tool to help create OPA policy documents. The created policy compares the number of resources of the specified type with the number of resources that comply with all specified values as defined in the manifest file. If the number of compliant resources doesn’t equal the total number of resources of the specified type the “deny” variable is set to true. Otherwise “deny” is set to false. Pipelines can examine the value of “deny” determining whether they should continue with the deployment.
The command line version takes a manifest filename as input. This manifest is a JSON document that describes a single policy for a single type of AWS resource. (IE Cloudfront Distribution, EC2 Security Group, etc) The policy generated will validate attribute keys and values from the resource_changes section of the Terraform plan. This static location is defined in the Terraform/AWS processor module and can be changed or extended without affecting any of the core functionality of the policy generator.
The natural language interface to OPG is referred to as the generator. It takes the input type, category, resource and a description of the types of attributes and their values to validate. The generator uses openai to parse the natural language and generate an OPG compatible manifest file. This further reduces the learning curve needed to generate a valid OPA policy document by eliminating the need to know the exact attribute names for a resource. Care still needs to be given when selecting attributes and their values to make sure mutually exclusive attributes are not being requested.
Components
- Open Policy Agent – Open-source policy engine used for enforcing policy through analysis of json input
- Terraform – IAC tool for managing infrastructure through IAC
- Python – Language used to implement OPG logic and workflow
Implementation Details
OPG consists of two main components. The OPG tool and the generator module. Both are invoked through the command prompt. The generator is the recommended way to interact with OPG as it doesn’t require the user to have detailed knowledge of terraform resource parameters.
The OPG tool is responsible for the generation of the OPA template based on the provided manifest. Each manifest file produces a single OPA policy which defines the required checks and criteria for a single resource type. The policy generated will validate attribute keys and values from the resource_changes section of the Terraform plan. This is a product of how the processor module has been implemented for terraform/aws. The manifest file can only reference a resource attribute once and this reference can only use one of the match or exist types.
Example manifest file:
Figure-01
The manifest file consists of the following keys:
- name str (Required) – Name to use for the generated policy document. The name should only contain alphanumeric characters and _. This is used for the OPA package name and the policy subdirectory where the policy document is saved.
- description str (Required) – General description to use for the generated policy.
- input_type str (Required) – Type of input the policy will examine. “terraform” is the only supported value.
- category str (Required) – Name of the “provider” used to generate the input. “aws” is the only supported category.
- service str (Required) – Name of the service which contains the resources to examine.
- resource str (Required) – Resource type to validate.
- attributes dict (Required) – Dictionary containing the key/value pairs which define the content of the generated policy.
The type of comparison done on an attribute is indicated by its string value or, in the case of a dictionary value, the value dictionary key.
Example:
"attributes": {
"enabled": {"in": true},
"ssl_only": "present”
} Currently OPG supports several match, exist and function type comparisons.
Match Types:
- in – The value of the specified key is one of values in the list provided in the manifest.
- not_in – The value of the specified key is NOT one of values in the list provided in the manifest.
- matches_pattern – The value of the specified key matches the regular expression provided in the manifest.
- equal – The value of the specified key must equal the value indicated in the manifest.
- in_list – The value of the specified key will be a list and the values in the manifest must appear within it.
- greater_than – The value of the specified key will be a value greater than the number specified in the manifest.
- less_than – The value of the specified key will be a value less than the number specified in the manifest.
Exist Types:
- present – The key provided in the manifest exists.
- not_present – The key provided in the manifest DOES NOT exist.
Function Types:
- opg_ref – The resource type being validated by the manifest file is referenced by at least one resource type defined by the attribute map key value ref_resource_type.
The user is responsible for understanding the attribute keys available and their potential value(s)/type(s). The user is also responsible for ensuring that the checks specified in the manifest file are not in conflict with each other.
At the core of the OPG tool implementation are the policy and attribute container classes. These classes are used for any input type. Each input type has its own processor class. The processor class contains the logic necessary for parsing a particular type of input. These three classes combined provide an OPA policy generation solution for a specific OPA input type. As mentioned above, the initial OPG implementation supports the Terraform input type with an AWS category.
Figure-02
The policy class is a container for all attributes associated with a single policy. An instance of the policy class contains an instance of the processor class for the input type indicated and an attribute class instance for every attribute included in the manifest file.
The attribute class is responsible for rendering the strings that make up the comparison of a single attribute. For example, ami_id = “ami-8873788432”. Each attribute class is provided the path to the attribute to compare as well as the type of comparison which is used to determine the method to create the OPA policy string(s) to validate the attribute for some input.
The OPG tool can be invoked directly from the command line using the syntax below.
python3 opg.py <manifest filename> The generator module is written in python and presents a natural language interface to the OPG tool. The user input is interpreted using OpenAI and produces an OPG compatible manifest.
The generator module creates a system message that includes the JSON schema describing the manifest document. Additionally, the valid attributes for the resource indicated are included to ensure openai selects valid properties for the resource.
The generator makes the call to openai using the built system and user messages requesting a new JSON document that meets the criteria the user has requested. The new JSON manifest is validated for required keys and attribute values. If it is found valid, the manifest is saved out as a file and the OPG tool python module is called to generate the OPA policy file.
The generator can be invoked directly from the command line using the syntax below.
python3 generate.py --itype terraform --category aws --resource <terraform resource type> ** User will be prompted to provide the natural language string defining what attributes and values the policy should inspect for all instances of the provided terraform resource type.
The output produced by OPG is the same regardless of whether the OPG tool or generator are the invocation source.
Summary
OPA adoption is becoming more frequent as the need for ensuring sound and secure IAC is moved to the forefront of many organizations, but writing OPA policies can be challenging and time consuming due to its unintuitive syntax. OPG takes the pain out of writing OPA policies for a more secure infrastructure.
