How to integrate CI/CD workflows with Terraform in GitHub Actions?

- name: run-digger
  uses: diggerhq/[email protected]

digger

Digger is an open source IaC orchestration tool. Digger allows you to run IaC in your existing CI pipeline ⚡️

We've been diving into integrating CI/CD workflows with Terraform and stumbled upon a tool named Digger. It's pretty neat for streamlining Terraform operations within our existing CI/CD setup.

Digger eliminates the need for a separate CI system just for Terraform by leveraging our existing CI infrastructure. This approach is not only secure—since it doesn't require sharing cloud access secrets with third-party systems—but it's also cost-effective because it utilizes the compute resources we already pay for.

Key Features:

• Direct execution of Terraform plans and apply operations within pull request comments.
• Utilizes the same CI runners, hence no need for separate ones.
• Supports Open Policy Agent (OPA) for role-based access control.
• Implements PR-level locks to prevent conflicts across multiple pull requests, which is a neat addition to the native state locks provided by Terraform.
• Compatibility with multiple tools and services like Terragrunt, various Terraform versions, and static analysis tools like Checkov.
• It also offers drift detection to keep track of unwanted changes.

From my understanding, Digger seems particularly useful for teams looking to maintain a lean CI/CD stack while ensuring secure, scalable, and efficient Terraform management. It reduces overhead and simplifies management by integrating directly with existing CI tools, potentially speeding up deployment cycles and reducing security risks.

How to Apply on Merge in Terraform?​

By default, Digger avoids running the apply command automatically when a merge occurs. Instead, it promotes running apply manually while the PR is open and only proceeding with the merge once it has successfully executed across all target environments. This approach prioritizes safety by ensuring that changes are thoroughly vetted before integration into the main branch. However, this method can cause the main branch to lag behind the actual state of our environments. F rom a strict CI/CD standpoint, this isn't ideal because the theory holds that the plan should be treated as an immutable artifact, and apply should merely deploy that artifact without discrepancies.

For teams preferring automation, Digger allows configuration adjustments to automate the apply on merge. Here’s how you can set it up in the digger.yml:

projects:
  ...
workflows:
  dev:
    workflow_configuration:
      on_pull_request_pushed: [digger plan]
      on_pull_request_closed: [digger unlock]
      on_commit_to_default: [digger apply]

With this setup, digger plan is triggered when a pull request is pushed, ensuring that plans are generated and reviewed. Once the pull request is closed, Digger releases any locks with digger unlock. Most critically, when commits are made to the default branch (typically after a merge), digger apply is executed automatically, aligning the main branch with the current state of the environments swiftly.

This configuration could streamline our workflow by automating parts of the process, but it’s important to balance the benefits of automation against the risks of reduced oversight on each change. It’s definitely a setup worth considering if we're confident in our testing and review processes.

How to Auto-merge with Digger Terraform?​

Digger allows for an auto_merge option, which can be extremely handy. When set to true, this option will automatically merge the pull request as soon as all checks pass, including the successful application of the Terraform changes. This is particularly useful if we want to streamline our workflow and reduce the manual steps required during the integration process.

Here's how you can configure it in the digger.yml file:

projects:
  - name: development
    dir: dev
  - name: production
    dir: prod
auto_merge: true

It's important to note that the auto_merge setting is applied at the top level of the configuration, not at the individual project level. This means it affects all projects defined in the YAML file.

Implementing auto_merge can significantly speed up our development cycle by ensuring that merges are only delayed by necessary checks and not by manual processes. However, it's vital to ensure our testing and review systems are robust to prevent any unanticipated issues from slipping through into production.

How to use Mutliple Aws accounts with Digger Terraform?​

We've been getting up to speed with setting up Digger for managing Terraform in multiple AWS accounts. Here's a simplified guide on how to configure Digger to operate separately across development and production environments, each with their dedicated AWS accounts.

• Configuration File:

  • Create a digger.yml file and place it at the root of your repository. This file will specify paths to the Terraform directories for development and production:

    projects:
      - name: develop
        dir: dev
        workflow_file: digger-run-dev.yml
    
      - name: production
        dir: prod
        workflow_file: digger-run-prod.yml
    

• GitHub Environments Setup:

  • Navigate to your repository settings in GitHub, then to Environments.
  • Create two environments: development and production.
  • For each environment, configure two secrets: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

• GitHub Actions Workflows:

  • Create two GitHub Actions workflow files:
    • .github/workflows/digger-run-dev.yml for the development environment.
    • .github/workflows/digger-run-prod.yml for the production environment.
  • Adjust each workflow to point to the correct environment and modify steps to rename and use the appropriate digger.yml configuration.

  name: Digger Dev
  on:
    pull_request:
      branches: [ "main" ]
      types: [ opened, synchronize ]
    issue_comment:
      types: [created]
    workflow_dispatch:
  
  jobs:
    plan:
      runs-on: ubuntu-latest
      environment: development
      permissions:
        contents: write
        id-token: write
        pull-requests: write
        statuses: write
  
      steps:
        - uses: actions/checkout@v4
        - name: Rename
          run: |
            mv digger.dev.yml digger.yml
        - name: digger run
          uses: diggerhq/[email protected]
          with:
            setup-aws: true
            aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
            aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
            aws-region: us-east-1
          env:
            GITHUB_CONTEXT: ${{ toJson(github) }}
            GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}