Optimize Your CI/CD Pipeline

Get instant insights into your CI/CD performance and costs. Reduce build times by up to 45% and save on infrastructure costs.

45% Faster Builds

60% Cost Reduction

View Documentation →

This article was last updated on December 20, 2024, to include advanced caching strategies such as Matrix-Based Caching, Selective Cache Invalidation, and Cross-Job Caching, along with simplified explanations for better understanding.

Introduction

Quick Introduction: GitHub Actions Cache saves data like dependencies, build files, and test results so your pipelines run faster. Instead of downloading the same stuff again and again, it reuses what’s already there, cutting build times by up to 80% and saving bandwidth.

After more than a decade of tuning CI/CD pipelines, I came to realize that one of the most powerful and yet misunderstood features in CICD is caching. In this tutorial, I am going to share my real-world experience with GitHub Actions caching and show you how you can dramatically reduce your build times.

Steps we'll cover:

• What is Caching in CI/CD?
  • Why Cache Matters
• Types of Caches
• Interactive Cache Strategy Helper
• Package Manager Caching Examples
  • NPM Cache Example
  • Python Pip Cache
• Docker Layer Caching
• Advanced Caching Strategies In GitHub Actions
  • Matrix-Based Caching
  • Selective Cache Invalidation
  • Cross-Job Caching
• Best Practices When Using Github Actions Cache

What is Caching in CI/CD?

Click to zoom

Think of the cache as the memory of your CI pipeline. If not used, every single time it starts fresh, having to download the same dependencies over and over. I have seen builds that were taking 15 minutes reduced to 3 minutes just by implementing proper caching.

Why Cache Matters

Based on my experience with managing large-scale CI systems, here is what proper caching can achieve: Reduce build times by 40-80% Lower bandwidth costs Decrease load on package servers Improve developer productivity

Types of Caches

Through my years of working with the optimization of CI/CD, I used to work with several types of caching:

Package Manager Cache

• npm/yarn for JavaScript
• pip for Python
• maven for Java
• go mod for Golang

Docker Layer Cache

• Image layers
• Build cache
• Multi-stage build cache

Build Output Cache

• Compiled assets
• Generated files
• Test results

Interactive Cache Strategy Helper

Below is an interactive tool that should help you understand which caching strategy will work for your project:

Cache Strategy Finder

What type of project are you working with?Select project typeNode.jsPythonJavaOther

How long are your builds taking?Select build timeLess than 5 minutes5-15 minutesMore than 15 minutes

How many dependencies does your project have?Select dependency countFew (< 20)Many (20+)

Are you using Docker?Select Docker usageYesNo

Package Manager Caching Examples

NPM Cache Example

Here is a basic configuration for npm caching:

steps:
- uses: actions/cache@v4
  with:
    path: |
      ~/.npm
      node_modules
    key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      npm-${{ runner.os }}-

- name: Install dependencies
  run: npm ci

I have found this pattern to work particularly well because it: caches both global (~/.npm) and local (node_modules) dependencies; uses OS-specific caching to avoid cross-platform issues; includes fallback restore-keys for partial cache hits.

Python Pip Cache

Following is the configuration I use for Python projects:

steps:
- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ runner.os }}-${{ hashFiles('**/requirements.txt') }}
    restore-keys: |
      pip-${{ runner.os }}-

- name: Install dependencies
  run: pip install -r requirements.txt

Docker Layer Caching

Docker caching is where I've seen the most dramatic improvements. Here's my optimized approach:

name: Build and Cache the Docker Image

on: [push]

jobs:
  Build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Create Docker Buildx
      uses: docker/setup-buildx-action@v3

    - name: Cache Docker layers
      uses: actions/cache@v4
      with:
        path: /tmp/.buildx-cache
        key: ${{ runner.os }}-buildx-${{ github.sha }}
        restore-keys: |
          ${{ runner.os }}-buildx-

    - name: Build and cache
      uses: docker/build-push-action@v5
      with:
        context: .
        push: false
        cache-from: type=local,src=/tmp/.buildx-cache
        cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

    # Temp fix for https://github.com/docker/build-push-action/issues/252

    - name: Move cache
      run: |
        rm -rf /tmp/.buildx-cache
        mv /tmp/.buildx-cache-new /tmp/.buildx-cache

This has saved my teams hours upon hours by: a) caching individual docker layers; b) using Buildx to have said cache managed much better by default; c) using our workaround for the cache size, which keeps growing.

Advanced Caching Strategies In GitHub Actions

Over the years, I've developed some advanced caching patterns:

Matrix-Based Caching

This configuration caches the Node.js dependencies for the various versions such as 14, 16, and 18.

strategy:
  matrix:
    node-version: [14, 16, 18]

steps:
- uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ matrix.node-version }}-${{ hashFiles('**/package-lock.json') }}

Using matrix.node-version in the cache key means each version has its own cache. The hashFiles function updates the cache only when package-lock.json changes to save time from re-downloading dependencies for unchanged configurations.

Selective Cache Invalidation

This caching strategy targets a specific folder or file type, such as specific/path/**/*.ext.

steps:
- uses: actions/cache@v4
  with:
    path: ~/.cache/custom
    key: cache-${{ hashFiles('specific/path/**/*.ext') }}-${{ github.ref }}

The cache updates only when these files change because of hashFiles. Adding github.ref to the key keeps caches separated by their respective branches. This is great for caching custom outputs, so you’re not stuck building stuff unnecessarily.

Cross-Job Caching

This setup shares a cache between jobs.

jobs:
  build:
    outputs:
      cache-key: ${{ steps.cache-key.outputs.value }}
    steps:
    - id: cache-key
      run: echo "value=${{ hashFiles('**/package-lock.json') }}" >> $GITHUB_OUTPUT

  test:
    needs: build
    steps:
    - uses: actions/cache@v4
      with:
        path: ~/.npm
        key: npm-${{ needs.build.outputs.cache-key }}

The build job generates a cache key from package-lock.json and passes it to the test job. This way, the test job can use the same dependencies downloaded in build. It avoids downloading the same things twice, saving time and keeping things consistent.

Best Practices When Using Github Actions Cache

From experience, here are some key practices on effective caching:

Strategy for Cache Key

• Add OS/Platform Information
• Use Hash of the lock files.
• Fallback Keys.

Cache Size Management

• Limit cached paths to necessary files only.
• Clean up old caches regularly.
• Regularly monitor cache hit rates and adjust configurations.

Security Considerations

• Do not cache sensitive data, such as secrets or API keys.
• Use cache scoping.
• Encrypt the cache when necessary.

Conclusion

Proper caching in GitHub Actions can turn your continuous integration/continuous deployment from what might previously have been a sluggish resource-intensive process into a lean, efficient running machine. I've seen teams reduce build times up to 80 percent just by implementing the strategies outlined here.

Caching isn't a set-it-and-forget-it feature, which means it does need constant monitoring and tuning, but this time pays for itself many times over in speed improvements on your builds and with developers who don't have to waste hours.

Need to monitor your cache performance? Check out the detailed information on GitHub Actions cache usage and optimization opportunities in CICube.