Skip to main content

Philips Labs GitHub Action Runners

If we are not deploying EKS, it's not worth the additional effort to set up Self-Hosted runners on EKS. Instead, we deploy Self-Hosted runners on EC2 instances. These are managed by an API Gateway and Lambda function that will automatically scale the number of runners based on the number of pending jobs in the queue. The queue is written to by the API Gateway from GitHub Events.

Quick Start​

StepsActions
1. Create GitHub AppClickOps
2. Upload GitHub App ID and Private Key to AWS SSMSet SSM Param "/pl-github-runners/id" and "/pl-github-runners/key" (base64 encoded)
3. Deploy GitHub OIDC ProviderDeploy GitHub OIDC to every needed account
4. Deploy GitHub Runnerstask terraform deploy philips-labs-github-runners -s core-use1-auto
5. Update Webhook (if changed or redeployed)ClickOps

Deploy​

The setup for the Philips Labs GitHub Action Runners requires first creating the GitHub App, then deploying the philips-labs-github-runner component, and then finalizing the GitHub App webhook. DevOps Accelerator typically does not have access to the customer's GitHub Organization settings, so the customer will need to create the initial GitHub App, then hand the setup back to DevOps Accelerator. Then DevOps Accelerator can deploy the component and generate the webhook. Finally, the customer will then need to add the webhook to the GitHub App and ensure the App is installed to all relevant GitHub repositories.

Follow the guide with the upstream module, philips-labs/terraform-aws-github-runner, or follow the steps below.

1 Vendor Components​

Vendor in the necessary components with the following workflow:

These are the commands included in the vendor/philips-labs workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Semaphore workflow! 🚀

2 Create the GitHub App​

Customer Requirement

This step requires access to the GitHub Organization. Customers will need to create this GitHub App in Jumpstart engagements.

  1. Create a new GitHub App

  2. Choose a name

  3. Choose a website (mandatory, not required for the module).

  4. Disable the webhook for now (we will configure this later or create an alternative webhook).

  5. Add the following permission for your chosen runner scope:

    Repository Permissions​

    • Actions: Read-only (check for queued jobs)
    • Checks: Read-only (receive events for new builds)
    • Metadata: Read-only (default/required)
    • Administration: Read & write (to register runner)

  6. Generate a Private Key

  7. If you are working with DevOps Accelerator, upload this Private Key and GitHub App ID to 1Password and inform DevOps Accelerator. Otherwise, continue to the next step.

3 Upload AWS SSM Parameters​

tip

This step does not require access to the GitHub Organization. DevOps Accelerator will run this deployment for Jumpstart engagements.

Now that the GitHub App has been created, upload the Private Key and GitHub App ID to AWS SSM Parameter Store in core-use1-auto (or your chosen region).

  1. Upload the PEM file key to the specified ssm path, /pl-github-runners/key, in core-use1-auto as a base64 encoded string.
  2. Upload the GitHub App ID to the specified ssm path, /pl-github-runners/id, in core-use1-auto.

Or run the upload/pl-secrets workflow with Taskfile to write the GitHub App information to the core-use1-auto SSM account and deploy the component.

These are the commands included in the upload/pl-secrets workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Semaphore workflow! 🚀

4 Deploy GitHub OIDC Providers​

First deploy the GitHub OIDC provider to all accounts where we want to grant GitHub access. The typical list of accounts is included with the deploy/github-oidc-provider workflow; run the following with super-admin-user:

These are the commands included in the deploy/github-oidc-provider workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Semaphore workflow! 🚀

5 Deploy the Philips Labs GitHub Runners​

Now that the GitHub App has been created and the SSM parameters have been uploaded, deploy the philips-labs-github-runners component.

These are the commands included in the deploy/pl-github-runners workflow in the examples/snippets/stacks/workflows/github.yaml file:
    No commands found
Too many commands? Consider using the Semaphore workflow! 🚀

6 Add the Webhook to the GitHub App​

Customer Requirement

This step requires access to the GitHub Organization. Customers will need to finalize the GitHub App in Jumpstart engagements.

Now that the component is deployed and the webhook has been created, add that webhook to the GitHub App. Both the webhook URL and secret should now be stored in 1Password. If not, you can retrieve these values from the output of the philips-labs-github-runners component in core-use1-auto as described in the previous step.

  1. Open the GitHub App created in Create the GitHub App above
  2. Enable the webhook.
  3. Provide the webhook url, should be part of the output of terraform.
  4. Provide the webhook secret (terraform output -raw <NAME_OUTPUT_VAR>).
  5. In the "Permissions & Events" section and then "Subscribe to Events" subsection, check "Workflow Job".
  6. Ensure the webhook for the GitHub app is enabled and pointing to the output of the module. The endpoint can be found by running task terraform output philips-labs-github-runners -s core-use1-auto 'webhook'

Usage​

Once you've deployed Self Hosted runners select the appropriate runner set with the runs-on configuration in any GitHub Actions workflow. For example, we can use the default runner set as such:

runs-on: ["self-hosted", "default"]

However, it's very likely you will have resource-intensive jobs that the default runner size may not satisfy. We recommend deploying additional runner sets for each tier of workflow resource requirements. For example in our internal GitHub Organization, we have default, medium, and large runners.

Using the terraform Label​

By default, we configure the Semaphore Terraform GitHub Actions to use the terraform labeled Self-Hosted runners.

runs-on: ["self-hosted", "terraform"]

However also by default we only have the single runner set. We recommend deploying a second runner set with a larger resource allocation for these specific jobs.

Remove the terraform label from the default runner set and add the terraform label to your new, larger runner set. Since the workflows are all labeled with terraform already, they will automatically select the new runner set on their next run.

FAQ​

I cannot assume the role from GitHub Actions after deploying​

The following error is very common if the GitHub workflow is missing proper permission.

Error: User: arn:aws:sts::***:assumed-role/acme-core-use1-auto-actions-runner@actions-runner-system/token-file-web-identity is not authorized to perform: sts:TagSession on resource: arn:aws:iam::999999999999:role/acme-plat-use1-dev-gha

In order to use a web identity, GitHub Action pipelines must have the following permission. See GitHub Action documentation for more.

permissions:
  id-token: write # This is required for requesting the JWT
  contents: read # This is required for actions/checkout