Secure Terraform Delivery Pipeline – Best Practices. Part 2/2

With the ease and speed of introducing changes in resource configuration of cloud resources, comes a great risk of introducing issues, as well as breaking compliance rules or company standards.

The goal of infrastructure testing and continuous compliance is to ensure automated verification of infrastructure rules. For example: 

• Limit allowed resource types and locations,
• Verify machine types and sizes,
• Verify resource configuration (e.g. parameters, naming, tags, tiers, encryption configuration),
• Check software versions and extensions installed on VMs,
• Check audit configuration applied to resources,
• Verify IAM roles and assignments (e.g. min/max count of administrators),
• Verify the configuration of Kubernetes deployments like allowed images, ports, limits, naming conventions.

Terraform can introduce changes very quickly and have a huge impact on the infrastructure. This is why automated testing and policy verification is just as important as in any other programming platform. 

It can mean:

• Requirements verification
  Automated verification of some non-functional requirements and assumptions for the project that need to be verified continuously before the sign-off. This is similar to unit/integration testing in software development. Usually, the team that creates Terraform scripts provides the tests as well.
• Compliance as Code
  Automated verification of company, organization or regulatory compliance policies using a set of rules. Rules can be related to resource type (e.g. forbidden services), resource location, machine types, OS type and version, replication options, pricing tier/SLA, tagging or naming conventions etc. required across a whole organization. A separate team might provide company-wide policies and compliance standards.
• Security as Code 
  Automated verification of security policies of introduced infrastructure. Rules can be related to RBAC control, network, firewalls, cloud access control, key vaults, keys, secrets and certificates, encryption etc. required across the entire organization. Security rules may be built with the IT Security team as well as with third-party tooling.

In general, there are two layers of Infrastructure as Code verification:

• Pre-deployment – Verification of the Terraform code or Terraform plan, more akin to Wstatic code analysis,
• Post-deployment – Verification of the resources created in the cloud environment after Terraform configuration is applied.

These are:

• AWS Config
• Azure Policy + Azure Security Center
• Forseti Config Validator for GCP

Cloud compliance services are sometimes provided with a set of rules mapped to industry standards such as HIPAA, ISO 27001 or CSA Benchmarks. Creating custom rules is not always easy. These tools can scope policy verification over a set of company projects/accounts/subscriptions not always „on-demand“ during the Terraform pipeline run.

One of the approaches observed in large organisations is that there are separate teams maintaining company-wide compliance rules and infrastructure as code. This means that the infrastructure team needs to adhere to standards and policies but is not always the author of new rules. The continuous policy tools should be used in addition to infrastructure testing.

AWS Config is a service that continuously monitors and records AWS resource configurations and allows verifying overall compliance against the configurations specified in the internal company guidelines. It comes with a set of around 150 pre-built managed rules as well with the SDK for creating and testing custom AWS Config rules. 

Since AWS Config rules are in fact AWS Lambda functions defined in NodeJS or Python, there is a large library of rules available in Github.

A sample fragment of code in Python:

In general, verification can be triggered by schedule or when resources are created or modified. The results can be asynchronously verified using the AWS API or Console. 

For a large scale project, a complete framework Compliance Engine is available to manage RuleSet in separate compliance account.

AWS Config verification can be woven into the Terraform Deployment pipeline as a post-release check. The results are however not immediate and some coding will be required to gather an end-to-end compliance report. Therefore, this is to ensure that the implemented change still adheres to company standards rather than to use it as a testing step.

AWS Config allows grouping the rules together with remediation actions into Conformance Packs (also „as a code“ using YAML templates) to be easily deployable into multiple accounts and regions.
Sample conformance packs:

• Operational Best Practices For Amazon S3
• Operational Best Practices For Amazon DynamoDB
• Operational Best Practices For PCI-DSS
• Operational Best Practices For AWS Identity And Access Management

With the use of AWS Control Tower, it is possible to:

• integrate AWS Config rules into an end-to-end compliance and conformance overview dashboard over a multi-account organization,
• have an Account Factory for creating new AWS Accounts with predefined rules and settings.

In general, AWS Config is a versatile solution to handle company-wide standard compliance as a code and security as a code. It might not be the fastest way to implement individual solution requirements verification, where simple tests may be easier to maintain. Its use in Terraform pipeline is possible as an addition to built-in AWS continuous compliance solution, but not necessary.

Pros:

• Flexibility, and extensibility since the rules are actual code in Python or JavaScript,
• SDK for rules development and a wide set of open-source rules in addition to built-in ones,
• Open-source tools for the whole multi-account Compliance Engine available as well as integration with Control Tower.

Cons:

• The rules code can become complex and constitute a whole programming project,
• The rule cannot prevent creating a non-compliant resource (only detective mode),
• Including asynchronous rules verification into Terraform CD pipeline requires a complex solution.

Azure Policy is a system built with declaratively defined rules applied to all resources in the scope of the assigned policy. Azure Policies can be assigned on Azure Subscription level as well as on Management Group level (a group of Subscriptions, e.g. whole organisation, all production subscriptions etc.).

Azure provides over 1000 predefined, parameterised policies. Custom policies are defined in JSON code and each policy consists of 3 parts:

• Parameters – they are defined during the assignment
• Policy Rule – the „if“ part of the rule 
• Effect – policy can either raise an alert (audit)  or prevent creating a resource (deny)

A sample fragment of code of a policy:

Policies in „deny“ mode work like additional validation rules, which means that a resource that is not passing the verification will not be created.

In addition to that, policies can remediate some threats – e.g. automatically install required VM extensions or modify the configuration.

Azure Policy check can be included as a post-deployment correctness check in the Azure DevOps release pipeline.

Besides individual policies, there is a number of predefined Policy Initiatives in Azure, for example:

• Audit ISO 27001:2013 controls and deploy specific VM Extensions to support audit requirements (56 policy checks)
• Audit PCI v3.2.1:2018 controls and deploy specific VM Extensions to support audit requirements (37 policy checks)
• Audit CIS Microsoft Azure Foundations Benchmark 1.1.0 recommendations and deploy specific supporting VM Extensions (83 policy checks)

Policy Initiatives are parameterised groups of policies to be assigned on Subscription or Management Group level and can be custom-created for company standards. There is a default Security Center initiative, containing over 90 configurable policies and is assigned by default to every Azure subscription. 

Azure Security Center is a single place to govern results of all policy checks across the organisation as well as group results of different threat detection systems (Network, Active Directory, VMs etc.). Since policies are verified periodically, the Security Center can address continuous compliance in Azure providing the alerting mechanism and verification history. To simplify the process of managing corporate-wide compliance, companies can also maintain Azure Blueprints. A blueprint is a combination of policies and initiatives together with default resource groups and IAM access configuration.

Azure Policies are very powerful but the tool does not provide a developer-friendly interface for creating custom rules, especially, when JSON is used as a language. Even without custom policies, the set of predefined policies is impressive and can address a wide range of compliance requirements. A complete Compliance as Code solution may combine infrastructure tests and Azure policies.

Pros:

• A large set predefined policies and initiatives for industry-standard compliance requirements,
• Built-in integration with Azure DevOps and Azure Security Center,
• The policy can work in „deny“ mode,
• Policy management with initiatives and blueprints.

Cons:

• Developing custom policies in JSON is hard,
• Executing policies „on-demand“ is not possible.

Google has open-sourced the Forseti Security project to address security rules validation and policy enforcement in Google Cloud. This is a policy-as-code system that consists of multiple modules and works together with:

• Forseti Security service that runs in Google Cloud and takes configuration snapshots for policy monitoring.
• Forseti Config Validator that evaluates GCP resources against Forseti rules.
• Forseti Terraform Validator that verifies terraform plan against Forseti rules/

The Forseti Rule Library is Open Source and uses Rego (by Open Policy Agent framework) files to define policy rule templates.

Here is a sample policy that forbids public IPs for Cloud SQL databases:

Using policy templates only requires defining constraints in YAML code. If a policy template for the actual use case is missing, Forseti provides tools and guidelines for authoring and testing custom policies. 

Setting up Forseti Security requires running dedicated infrastructure in GCP that consists of Cloud SQL, compute and Cloud Storage. Google recommends creating a separate project to serve as a policy monitoring environment. Forseti provides a Terraform configuration for installation. The server picks policy configuration deployed to Google Cloud Storage and then:

• constantly monitors policies,
• can enforce security rules,
• stores Cloud Configuration snapshots in Cloud SQL.

To increase the overall policy and security status visibility, Google offers the Security Command Center dashboard. Besides Forseti, GCSCC can use other threat detection systems as a source of vulnerabilities alerts like:

• Cloud Data Loss Prevention Discovery,
• Anomaly Detection,
• Event Threat Detection,
• 3rd party cloud security tools from Acalvio, Capsule8, Cavirin, Chef, Check Point CloudGuard Dome9, Cloudflare, CloudQuest, McAfee, Qualys, Redblaze, Redlock, StackRox, Tenable.io, and Twistlock.

Forseti Security offers complete compliance-as-code tooling that can be used as part of Terraform CD pipeline in the pre-deployment step (with Forseti Terraform Validator) as well as post-deployment (with Forseti Config Validator). Continuous compliance support with Cloud Command Center and other plug-in systems adds up to a complete solution. 

Being a security-oriented solution, this may require significant effort to implement additional custom non-functional requirements checks, thus, it might be a good idea to combine it with infrastructure testing approach.

Pros:

• Support for GCP config validation as well as for Terraform validation,
• Declarative rules language with tooling support,
• Integration with Security Command Center.

Cons:

• Requires installation and maintenance of infrastructure and setup of open-source components,
• A small library of predefined rules (around 70 templates)  and lack of industry-standard policy sets (e.g. CIS Benchmarks, HIPAA etc.),
• Lack of preventive-mode, only reactive/detection mode.

For a complete guide for setting up Forseti with Config Validator and Terraform Validator, check out the blog article Protecting your GCP infrastructure at scale with Forseti Config Validator (plus part 2, part 3 and part 4).