As 2023 flies by and the Cybersecurity incidents making headlines continue to roll in, I would like to discuss self-hosting your own GitHub Actions infrastructure.
First, I always encourage teams to use hosted solutions when possible. The keyword here, and in most good systems engineering, is “where possible” - which can be the hardest part for teams to isolate and identify given their backstory, knowledge gaps, and regulatory environment(s). For instance, I’ve seen powerful DevOps, Infrastructure, or Platform engineering teams decide to self-host Actions Runner environments because they have the resources and buy-in from management. However, there’s no strong reason beyond the team wanting to do it and thinking they have the free cycles. This kind of thinking is a classic trap for engineering teams because the objective is a sexy one, and I can’t deny that. GitHub Actions are getting more popular by the day, and it’s certainly a hot topic among teams already using GitHub for source control or considering the move from competing well-established ecosystems like GitLab, Gerrit, Jenkins, and similar alternatives.
The problem with taking this objective of self-hosted Actions runners on is the hidden complexity that comes with it. Those in the software engineering field who have stuck around long enough know this is a common theme among teams with less experienced management. I’ve seen this at Silicon Valley companies in business for a decade, startups that are just starting to get traction and finally have room to breathe, along with larger Fortune 500-esque businesses with free cycles to take on such projects.
Unless self-hosted GitHub Actions add value to your business, expensive and hard-to-find engineers will spend time on what boils down to a learning exercise for the team rather than time spent innovating and out-maneuvering the competition in new areas that add value to your product or business.
So when should you use self-hosted Actions?
Teams should only go down this self-hosted path when major blockers or regulatory concerns exist when using the GitHub.com public runner infrastructure hosted in Microsoft’s Azure infrastructure.
For instance, the GitHub.com Azure-hosted Actions:
- Are not (as of mid-2023) FedRamp certified.
- Are hosted in the United States.
- Are not (as of mid-2023) HIPAA compliant.
Depending on how you use Actions, these limitations might not impact you even if you have specific requirements regarding the three bullet points above. One important thing to remember is that at the initial implementation of Actions within your company - you might not be exfiltrating any PHI or US Federal systems data into GitHub.com’s public Actions infrastructure.
However, as time passes and tribal knowledge changes hands as you experience turnover and handoffs across teams, there might be inherent assumptions about these systems being acceptable to handle such highly regulated data. That’s a future audit demon waiting to rear its head. One day folks are using Actions simply for Continuous Integration checks on their PRs to ensure code coverage is acceptable - and due to how easy it is to use Actions, the following year, you’ve got teams using it to manage regulated data without realizing they’ve crossed an invisible regulatory line that could put your business in jeopardy.
That’s why it’s my professional opinion that any team with any whiff of running afoul of regulatory concerns like FedRamp, HIPAA, GDPR, or similar frameworks should go the self-hosted Actions route right out of the gate. Use GitHub.com’s public runners for an initial PoC to build internal confidence that this is a good choice for your teams - I am all for that. But once you’re past that stage, it’s time to get serious about building your dedicated and secure platform to host GitHub Actions and GitHub Enterprise Server (GHES.)
That’s all for now. I will follow up with additional articles shortly with my thoughts on some existing Open Source projects that help implement self-hosted Actions runners in VMs or Kubernetes using containers, along with DIY approaches and why one might want to consider that path given some issues I’ve experienced first-hand in the past using Open Source K8s solutions such as Actions Runner Controller.
- Kevin