Kubernetes に Github Actions の self-hosted runner(Actions Runner Controller ) を構築して Terraform を自動化してみた

Kubernetes Cluster で Actions Runner Controller を構築し Github Actions の workflow を self-hosted runner で動作させてみたので紹介します。

モチベーション

• Terraform でデプロイしている ArgoCD を GitOps でバージョンアップしたかった
• private リポジトリでもコストを気にせず実行したかった
• ARM64 環境で実行したかった

公式情報

ドキュメント: https://docs.github.com/ja/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller
リポジトリ: https://github.com/actions/actions-runner-controller
Helm Chart repo: https://actions-runner-controller.github.io/actions-runner-controller
Helm Chart name: actions-runner-controller
Helm Chart version: 0.23.7

Kustomize helmCharts

Kustomize に内包されている helmCharts に actions-runner-controller の Helm を定義
(Kustomize helmCharts に関するブログについてはこちら)

helmCharts:
  - name: actions-runner-controller
    repo: https://actions-runner-controller.github.io/actions-runner-controller
    version: "0.23.7"
    namespace: arc-system
    releaseName: actions-runner-controller
    valuesFile: ../values.yaml

github token を使用する場合の values.yaml

authSecret:
  enabled: true
  create: true
  name: "controller-manager"
  github_token: <path:secret/data/github/actions#TOKEN>

RunnerDeployment

RunnerDeployment のカスタムリソースに workflow を実行するリポジトリの情報などを定義します。
この POD で workflow が実行されます。

もともと POD で実行する想定ですが、ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER: true で念のためコンテナ環境で workflow の実行を強制しています。

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: arc-runner-terraform-helm-argocd
  namespace: arc-system
spec:
  replicas: 2
  template:
    spec:
      repository: AbeYuki/terraform-helm-argocd
      labels:
        - self-hosted
        - kubernetes
        - arm64
      containerMode: kubernetes
      serviceAccountName: arc-runner-sa
      workVolumeClaimTemplate:
        storageClassName: "longhorn"
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
      env:
        - name: ACTIONS_RUNNER_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "true"

ArgoCD から POD を見ると Progressing の状態ですが workflow が実行できます。

workflows

name: '[staging]Terraform Plan to Apply'

on:
  push:
    branches:
      - staging/**
      - staging*

jobs:
  terraform-plan-apply-staging:
    runs-on: [self-hosted, kubernetes, arm64]
    container:
      image: hashicorp/terraform:latest
    steps:
      - name: Install git and setup kube config
        run: |
          apk add --no-cache git
          mkdir -p ~/.kube/
          printf "%s" "${{ secrets.KUBE_CONFIG_STAGING }}" > ~/.kube/config_rke_oci

      - name: Checkout repository
        run: git clone -b staging https://github.com/${{ github.repository }} ${{ github.repository }}

      - name: Initialize Terraform
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform init
        working-directory: ${{ github.repository }}/staging

      - name: Terraform Plan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform plan -out=tfplan
        working-directory: ${{ github.repository }}/staging

      - name: Terraform Apply
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform apply -auto-approve
        working-directory: ${{ github.repository }}/staging

name: '[Production]Terraform Plan on PR'

on:
  pull_request:

jobs:
  terraform-plan-production:
    runs-on: [self-hosted, kubernetes, arm64]
    container:
      image: hashicorp/terraform:latest
    steps:
      - name: Install git
        run: |
          apk add --no-cache git
          mkdir -p ~/.kube/
          printf "%s" "${{ secrets.KUBE_CONFIG_PRODUCTION }}" > ~/.kube/config_k3s

      - name: Checkout repository
        run: git clone -b ${{ github.head_ref }} https://github.com/${{ github.repository }} ${{ github.repository }}

      - name: Initialize Terraform
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform init
        working-directory: ${{ github.repository }}/production

      - name: Terraform Plan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform plan -out=tfplan
        working-directory: ${{ github.repository }}/production

name: '[Production]Terraform Apply on Merge'

on:
  pull_request:
    branches:
      - main
    types:
      - closed

jobs:
  terraform-apply-production:
    runs-on: [self-hosted, kubernetes, arm64]
    container:
      image: hashicorp/terraform:latest
    steps:
      - name: Install git
        run: |
          apk add --no-cache git
          mkdir -p ~/.kube/
          printf "%s" "${{ secrets.KUBE_CONFIG_PRODUCTION }}" > ~/.kube/config_k3s

      - name: Checkout repository
        run: git clone https://github.com/${{ github.repository }} ${{ github.repository }}

      - name: Initialize Terraform
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform init
        working-directory: ${{ github.repository }}/production

      - name: Terraform Apply
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        run: terraform apply -auto-approve
        working-directory: ${{ github.repository }}/production

利用してみた感想としては ARM64 環境で self-hosted することで同じ CPU アーキテクチャのコンテナイメージのビルドは行いやすい等の利点を感じましたが、ARM64 環境だと組み込みの actions/ が使えず自前でコードを作成しないといけない点は不便に感じました。 CircleCI の self-hosted 場合は ARM64 環境でも組み込みの物が使えたので CircleCI の方が ARM64 環境では便利かもしれません。