What we learned from building 60+ third party resources for AWS CloudFormation
What is AWS CloudFormation?
AWS CloudFormation is an Infrastructure as Code service which enables you to speed up cloud provisioning by modelling, provisioning and managing both AWS and third-party resources.
The public CloudFormation Registry is where users can discover, provision and manage these resources. Resources are one of the three CloudFormation extensions which are managed through the Registry:
- Resource types – model and provision custom logic as a resource, using stacks in CloudFormation.
- Modules – package resource configurations for inclusion across stack templates, in a transparent, manageable, and repeatable way.
- Hooks – proactively inspect the configuration of your AWS resources before provisioning.
We built 60+ resources for the CloudFormation Registry, covering a range of third-party providers like Fastly, GitHub, GitLab, Okta, Cloudflare, Snowflake and more.
In this blog, we’ll share the framework used for building third party resources, the gotchas to know about, and the developer experience to expect.
This blog was co-authored by Thomas Bouron, Duncan Grant and Charlotte Binstead.
Building new third party resources for AWS CloudFormation
Many use CloudFormation to manage their AWS services. In addition to AWS resource support, CloudFormation also supports managing third party services. We’ve open sourced and published several resources for commonly used third party services so these can be managed through CloudFormation. This lets you keep your configuration in code in one place, and have full visibility of changes, with reviewing and auditing. You can detect drift in case someone made manual changes in the AWS console. You can also keep secrets in the Systems Manager Parameter Store and control who can make changes with IAM.
AWS has made it easy to build your own resources too, either for private services that only you will use, or for publishing for anyone to use. Below we’ve tried to give you an insight into some of the things we learned building our first resources.
Before building anything, we needed first to identify what makes a good resource for users?
It generally boiled down to 3 sub-questions. Given a chosen third party provider:
- Is the resource commonly used by users of that provider?
- Is the resource linked to other resources of that provider?
- Does the provider API support full CRUDL (Create/Read/Update/List/Delete) operations? If not, does the provider API support CRD (Create/Read/Delete)?
If the answer to all 3 questions was ‘yes’, then it qualified as a “good resource”.
Note for (3) specifically, the CloudFormation custom resource framework requires that we can at least create, read and delete a resource of a given type, that’s why this sub-question was a must for us.
Here are some examples of resources and use cases which passed our 3 question test:
GitHub - configure users and repositories in code
We’ve created GitHub resources that allow us to configure users (GitHub::Teams::Membership) and repositories (GitHub::Repositories::Repository) in code. When a new user joins a project, you can add them to a team (GitHub::Teams::Team) quickly, and safely, in code managed by source control. Once your CI is set up you can use the webhook (GitHub::Repositories::Webhook) resource to start a build in your CI system whenever new code is merged.
PagerDuty - user access, incident management and more
Often when a developer joins a project, they need more than just GitHub access, they’ll also need access to systems like PagerDuty. So we’ve created some useful PagerDuty resources so you can also add the same developer (PagerDuty::Users::User) to the correct team (PagerDuty::Teams::Team) on PagerDuty and then add them to the relevant on-call schedule (PagerDuty::Schedules::Schedule) or have them notified of relevant issues (PagerDuty::EscalationPolicies::EscalationPolicy). We also added a resource to template how incidents are managed and apply these across lots of projects and systems (PagerDuty::ResponsePlays::ResponsePlay).
How we built and tested the new AWS CloudFormation resources
Once the resources were identified, we started implementation. In order to facilitate development and documentation, we grouped all resources from the same third party provider under the same Git repository. Each resource lives in its own folder within that repository.
Folder structure example of GitHub resources for CloudFormation
Building the AWS CloudFormation resources
The Resource Provider Development Kit (RPDK) provided by AWS currently supports 4 different languages: Java, Python, Go and TypeScript.
Choosing a programming language is very much based on personal preferences, although 2 important factors came in play when we developed our resources:
- Are there API clients or libraries already available? Are they stable?
- How much boilerplate code is required?
We were surprised to find that only a small number of the third party providers we targeted for custom resource development proposed an official and stable (i.e. not alpha or beta) client or library to communicate with their API.
It was therefore a must-have for us to be able to quickly implement a client that suited our needs. Given the fact that most of these providers communicate over HTTP through REST APIs, we could have chosen any of the 4 supported languages. However, given that the APIs all communicate with JSON payload, TypeScript does have an advantage here as it’s natively supported, there is no need to convert like Java and we can even quickly define the shape of the payload with types.
As for the boilerplate, that’s where TypeScript really shone. For virtually the same code, using TypeScript drastically reduced the amount of boilerplate that was required. Hence why most of our resources are written in TypeScript.
Testing the AWS CloudFormation resources
We pride ourselves on doing quality work and testing is an integral part of that. All of our resources use a common abstraction layer and so we wrote unit tests to avoid regressions. For the resource itself, however, we faced a dilemma. We could introduce mock data to test that our code was doing the expected HTTP requests but didn’t feel there was much value in that. What we were interested in was to test that each handler actually does what it is supposed to do: create/read/update/delete/list resources.
So, we then decided to go straight to the end-to-end testing for the resources. Luckily, AWS does provide a testing framework to help! It is called contract tests version 1 (or CTv1) and can be run locally via CloudFormation CLI which is also open source. It was great to live test each resource and validate automatically that the handlers were doing their jobs. However, a few things were missing:
- It appeared that CTv1 had a slightly different behaviour than when CloudFormation was running our resources. Some tests would fail whereas CloudFormation was perfectly happy.
- CTv1 doesn’t actually test the submit phase. This is important because even though there are some validations happening at build time, extra validations are also made during the resource submission process. We ended up a few times with a resource that passed CTv1, but failed to publish onto the CloudFormation Registry.
- Another issue was that CTv1 did not test drift detection, so we had to manually check this through real CloudFormation stacks.
Challenges
The first and probably most obvious one was credentials: how do we provide API credentials to our resources? The simplest way would be to add parameter(s) on each resource. But a better solution is to configure it at the account level with what is called Type Configuration. An admin can then configure the credentials (and more) which will get passed to each resource of that type without disclosing the secrets. The only downside at the moment is that you can have only one type configuration per resource type. It would be great to be able to set multiples, and select which one to use when deploying a CloudFormation stack.
We found out pretty quickly that almost all providers had eventually-consistent reads over writes. Although this was not apparent at first, we decided to create an abstraction layer common to all resources to tackle this issue (+ more general common behaviours). This means that each resource does have a stabilisation period built-in for all write operations with exponential back-off with jitter.
We also faced throttling issues. While we are correctly identifying them to CloudFormation, our code currently doesn’t support retries in this particular case. This is something we are aiming to fix in our abstraction layer for future versions.
Conclusion
We’re really pleased to see these AWS CloudFormation resources published and in use! You can find them in the CloudFormation Registry in your AWS Console. If you are interested in reviewing or contributing to the resources, the projects are open source and you can find them on this Github. There is also a CloudFormation community on Discord for developers to ask questions and get support.
We look forward to further developing these, and other new resources, for the public CloudFormation Registry to help AWS users better integrate their third party services with their AWS environments.
About Cloudsoft Fusion
Cloudsoft Fusion is a trusted AWS development partner, with unrivalled cloud and third party integration and connectivity experience.
We're proud to have helped launch over 60 new resources to the AWS CloudFormation registry, and we can help you to reach this rich pool of over 1 million users too.
Cloudformation Advance is a development service which takes you from design to deploy whilst reducing developer toil and getting your services infront of new users fast.
