GitOps security for deployment on Azure with NubesGen

[ML72]

Currently, my team uses GitOps to deploy all GitHub branches with the name pattern env-* to Azure via Terraform. However, we recently realized that there was a security vulnerability where members can create a branch name that matches that pattern, and it would automatically deploy to that resource on Azure.

Furthermore, since GitHub actions runs on the current branch & development branches (i.e. not main or env-* branches) aren't protected, it is possible for members to directly edit the GitOps and Terraform config to modify prod deployment (hard code env as prod in the Terraform config, and then create any random env branch such as env-randomrandom).

I considered the following solutions, but all of them either can't be implemented or are unsatisfactory:

1. Restrict GH branch naming (GH doesn't allow this)
2. Create a separate fork and do prod deployment there (I'd still need Azure credentials)
3. Disallow creation of certain resource groups on Azure (prevents creation of unintended envs, but doesn't protect prod deployment)

Hence, I just wanted to ask if anyone has an elegant way of addressing this security issue?

[cmaneu]

Hello @ML72,
Thank you for your message, and for using NubesGen :).

First, I want to acknowledge that this is indeed an issue. Before giving you some paths for an elegant way to solve this, I would like to discuss the "security" aspect of it. To you, how is being able to create several environments present a security issue? My understanding is that you want to prevent some developers to be able to change infrastructure on env-prod branches ?

Proposal

The way we currently manage environments in NubesGen (env-* naming convention) is basic and does not cover such scenarios. We have this in our roadmap, yet in the meantime, here are some ways to address that need:

• CODEOWNERS: With GitHub, you can define CodeOwners, and associate branch protections. Thus, you can add the GitHub Actions and terraform code paths to some codeowners. It will not prevent devs from creating new branches, but it will prevent devs from modifying the infrastructure code on these branches.

• Multiple Service Principals: One of the issues with the current setup in NubesGen is that we create (with the NubesGen CLI) a Service principal with Contributor rights on a Subscription Scope. This is the easiest way to start doing something, yet in bigger project / Enterprise scenarios it's wrong for a lot of reasons...including when you segregate dev and production subscriptions.
  One thing you can do without us updating NubesGen is to

  1. Create service principals limited to resource groups already corresponding to environments
  2. Update the GH Action to execute only on approved branches (we can do this with a if instruction, see below)
  3. Eventually, update the GH Action to use different service principals depending on which is the current branch

    steps:
      - name: Apply Terraform configuration prod
        if: $GITHUB_REF == "refs/heads/env-prod
        id: infrastructure-deployment
        uses: microsoft/nubesgen-actions/[email protected]
        with:
          azure_credentials: ${{ secrets.AZURE_CREDENTIALS_PROD }}
          tf_storage_account: ${{ secrets.TF_STORAGE_ACCOUNT_PROD }}

Let me know what you think. Happy to continue the discussion to find a workaround that works now, and to inform our future updates :).

[ML72]

Hello @cmaneu ,
Thanks for the quick & detailed reply!

To address your first point, my team considers the "security" on two levels:

1. Most importantly, developers should not be able to modify the prod deployment; this will be the "issue" I refer to in the rest of this reply
2. As an issue of much lower magnitude, developers should not be able create new environments, because the creation of these environments directly affects resource costs on Azure; however, this issue isn't very important

Regarding your other points, it seems that using a CODEOWNERS file or multiple service principals on one repository would not resolve this issue because as long as some Azure secret with credentials linked to prod is stored in the repo, developers are able to edit the terraform configuration on their own branches to use that.

For example, let's say I have a protected prod environment in the branch env-prod. A malicious developer could:

1. Create a new branch called abc-xxx (this branch name doesn't even need to follow the env-* naming convention, since the developer can modify /.github files directly on their branch)
2. Modify gitops.yml to run with ${{ secrets.AZURE_CREDENTIALS_PROD }} and ${{ secrets.TF_STORAGE_ACCOUNT_PROD }} on push to abc-xxx
3. Go to /terraform/main.tf and change all occurrences of local.environment to "prod", so that Azure thinks we are deploying to prod

And thus, our prod deployment is compromised.

Upon further exploration, I think that the most viable solution is to just not store Azure credentials with prod resource permissions inside the development repository. We could create two service principals - a dev credential with limited access to resources, and a prod credential with Contributor rights. The dev credential can be used in the development repository, while the prod credential can be used in a private fork of the development repository.

This workaround should work, but it does feel a bit unwieldy... I'd love to see some future Nubesgen update which simplifies the complexity of this! 😄

[cmaneu]

Thank you @ML72 for your details. I see your point!
Your "security threat model" considers that developers that have right access to the repository (and thus can create a branch+commit to it) should not be trusted.

In such case, my recommendation would be to use GitHub Environments. When deploying to an environment, you can:

• limit the source branches that are targeting that environment (like only commits made to main can go to production environment)
• Enable branch protection rules, including limited reviewers approval (few people are listed as reviewers of that environment)
• Use secrets that are scoped to that environment: GitHub actions are only run after a reviewer has accepted the change.

Enabling environments - outside creating them within the repository - requires only minimum changes within the code generated by NubesGen (see docs).

Note that Environments are only available to Organizations with GitHub Team or users with GitHub Pro. But I would definitely recommend that solution over having two different repositories.

For reference (including to myself), additional reading:

• https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment
• https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions

Let me know what you think about that solution. For the record, GitHub environments are definitely on our radar for future Nubesgen update :)

[ML72]

Thanks for the information on environments! I was previously not aware that GH environments were configurable; this sounds like a fantastic solution.

I'll try this out & get back to you on this!

[ML72]

Hello! Apologies for the delayed response; there was some work to finish before we got to fully testing this approach.

Using GitHub environments and scoped secrets with different service principals is a perfect solution. Thank you so much for the information & looking forward to seeing GitHub environments in a future Nubesgen update! 😄

[jdubois]

Oh that's exactly what I'm starting to do on the main NubesGen branch, look at this commit from a few hours ago:

NubesGen/.github/workflows/rest-server-continuous-deployment-main.yml

Line 17 in 48955e6

environment: dev

😀

This indeed seems to be the best solution, but it needs more documentation.

[ML72]

That sounds great! Would it help if I provided documentation on how my team managed branch security with environments & Azure service principals?

[jdubois]

Oh yes! I have now a first solution but it can definitely be improved.
I couldn’t find some good documentation or best practices, and GitHub environment are quite new, so we’re on new ground here. Let’s discuss: this would help a lot of people!

[ML72]

I'd be happy to discuss strategies & optimizations on this process! It's true that this is new ground - I haven't been able to find documentation on this either, so much of what I did is crude trial-and-error.

The steps I took on Azure & on GitHub are as follows.

Azure:

1. Create a limited perms app registration in AAD, in addition to the full perms app registration created by Nubesgen.
2. By default, the full perms app has subscription-level access. The limited perms app should be given access to individual resource groups (I gave it access to everything except env-prod).
   • I believe the full perms app needs to be used to create the resource groups to begin with, since subscription level perms are needed to create resource groups. For us, this wasn't a problem since we already had environments configured by Terraform by the time we started setting this up.

GitHub:

1. Create a new environment with appropriate settings (prod branch, mandatory reviewers, etc.) and create a full perms secret there.
2. Edit the gitops workflow file to trigger only on pushes to env-prod & add environment: <prod env name> to the start of production-related jobs.
   • Currently, we just created a separate copy of gitops.yml to be triggered for prod only, since the environment needs to be declared and I couldn't figure out how to conditionally add the environment: <prod env name> line.

Further Thoughts:

Some of my thoughts on the process:

• Having two separate gitops workflow files for prod and dev respectively feels wrong, since practically everything in those files is the exact same. There has to be ways to conditionally add the environment: <prod env name> declaration, but I was hoping to balance complexity with efficiency (some of the workarounds I can think of seem way overcomplicated).
• With this setup, people with write perms to the repository can technically still mess with the prod deployment. Since the limited perms app still needs access to the rg-terraform-001 resource group to be able to deploy to dev, someone can theoretically write a workflow that edits contents in the Terraform storage account. Then on the next prod deployment, resources in prod may be modified in unexpected ways.
  • One solution is to simply create two separate subscriptions with separate Terraform settings, one for prod deployment and one for dev-related deployments. This would completely isolate the environments and ensure that the app with dev-related perms cannot affect prod in any way.
  • However, that solution feels costly & a bit redundant, so my team currently runs on the assumption that any ill-intended GitHub actions runs will be caught before subsequent deployments to env-prod.

[jdubois]

Thank you so much, that's an incredibly valuable feedback!
Totally agree on the environment that should be declared conditionally so we reuse the same workflow: I still haven't found a good way to do this, but this is totally my plan.
Concerning having 2 subscriptions: I'm also starting to believe that's the best way not to mess up. Then it's more complex to set up, and indeed more costly. Another example that don't feel totally right to me: we create a Container Registry per environment. This adds cost & complexity, but we gain true isolation between environments.

[ML72]

Regarding the environment declaration - I'm playing around with it a bit too, and I'll be sure to update if/when I find a suitable solution!

As for the registry: It could work, but I agree that it may add cost & complexity on the user side. One solution I also see is Nubesgen creating a GitHub prod environment with a full-perms azure credentials; the user can specify what environments they want to create in a config file like .github/nubesgen/environments.yml, and the full-perms credential can then be used to create associated resource groups and limited-perms credentials. This could achieve perms isolation on Azure without significantly complicating users' workflow, but would likely add a lot of complexity to the code for Nubesgen's CLI.