Deploy a stack on every Pull Request using Pulumi and Github Actions

Recently I’ve been trying to incorporate infrastructure as code in my project at the early stages so that it increases the efficiency, productivity, and confidence when it comes to deployment and testing. While working on setting up the IaC, I decided on three different stages which are development, staging, and production.

One issue that I have stumbled on is when I do some changes related to the resources that I have defined in pulumi and their configurations, running pulumi preview is not enough to make sure that everything is working properly. I might have defined my resources correctly, and the preview command will let me know about that, but my configurations such as the ports that have been defined on the application load balancer, or the security groups config haven’t been set correctly. Thus, after I have merged the feature branch to one of the branches such as development, only later I would find out that not everything have been set correctly when trying to hit one of the endpoints. Remember, one of the most important things to keep in mind when it comes to DevOps is that we need to fail fast, and I need to live by that.

Many times I would do this mistake, and I need to create a commit directly on top of the development branch for instance to fix the issue, and this would ‘poison’ the commit history, also it gets messy when it comes to tracking the changes to issues/stories.

To achieve that, we would create a stack for each and every pull request that has been created for a feature, this way it’s easier to catch any errors earlier in the development cycle. Also, this helps the reviewer tremendously when reviewing a PR, not only does he have code to look at, but also a deployment to test things for himself if needed.

Let’s start with the project structure that we are going to work with

Project Structure

The infra/ directory has all the code related to pulumi and the infrastructure such as the resources definition and their configurations. Inside of infra/ I created a new folder called automation that has the automation API code that will make us achieve creating a stack automatically after each PR. This code will be run through a Github Action workflow file that I will get into a bit later. One important thing is we already have a stack defined, which is the development stack as you can see from the file Pulumi.development.yaml , this is important in my case at least, as that will be the stack which I will copy all the config from such as the secrets and environment variables that I will pass to the different resources that will be created i.e an ECS instance.

Let’s break down what are the different steps that will happen to achieve all of this to make it simpler to follow

  1. A PR on Github is created to target the development branch
  2. Github Actions will start kicking in, and the workflows will start triggering
  3. One workflow pull_request will have the steps that will cd into the automation folder and run node index.js which will run our automation api code that will create the stack for us and deploy the resources
  4. Resources are created, reviewer is satisfied with our PR, he merges the changes into the target branch
  5. One last Github Action will run destroy_pulumi_on_merge and will destroy the PR stack that has been created with its resources

Pulumi Automation API Script

Let’s start with our automation api script that will create a new stack, get the configs that have been defined by our development stack, and then deploying the resources

The script is pretty straight forward, I think. We first define our arguments such as the stack name (which we take from the env variable PR_STACK_NAME ), and the workDir which points to the directory where our Pulumi.yaml is in. We later create a new stack using the name that is set as the PR_STACK_NAME value, and copy all the secrets and config values from an already existing stack settings which is Pulumi.development.yaml in this example.

The only change I do to these configs is the AWS region. That is because there is a limit of 5 Elastic IPs per AWS region, and to get over this, I deploy the PR resources in another region which is other than the one I originally use for the different application stages development, staging, and production.

At the end we basically run pulumi up through the code.

Github Actions

There are two github action workflows that we need to define. The first being a workflow that runs when a pull request is created, this is where the new stack is created and the resources are deployed. The second workflow is to basically clean up everything that we have done, which runs when we have successfully tested everything that we need an we merge the PR to the target branch.

Pull Request Workflow

First up is the pull request workflow definition

In my case I am deploying my resources on AWS and also creating a MongoDB cluster, that is why I am passing these environment variables.

The most interesting parts of this workflow file are the last two steps. In the step before last, we create an environment variable for the stack that we will be creating for the PR. This is the bit that I am not 100% confident off for couple of reasons. One of them being is that the stack name will be <target_branch_name>_<first_15_characters_of_the_branch_to_merge>. So if you created a PR that is to merge fix_aws_resources onto the development branch, the stack will be called dev-fix-aws-resourc which I wouldn’t say is the greatest name. This is done because when we later actually merge the PR, we want to actually get this name again and be able to reference it so we could destroy the stack.

Unfortunately, this is the best combination that I could find that are not changing between commits such as the SHA that can change between commits and merges. One problem that I had in mind is the limit of characters for the AWS resource names, so I tried to keep it short, but at the same time enough to be able to look through the PRs on github and understand which one does this deployment relate to.

In the last step, we pass the just created stack name that I have explained above, and is used in the infra/automation/index.js script that does all the magic for us.

Merge Pull Request Workflow

This is the easiest part, if you understood the last workflow file and the code, the only difference in this workflow is that it triggers when the pull request is closed. The only difference is that we are passing destroy to our automation script which you can see in the last step node index.js destroy

Thank you

I hope this was helpful. There are still some doubt in me that needs to improve upon this solution, and I will be doing that for sure. If there are any changes I will make sure to update this article, and if you have suggestions and improvements I would be very grateful.

Written by

junior javascript advocate https://twitter.com/varqasim

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store