If you’ve read the blog posts on CloudJourney.io before, you’ve likely come across the term “Continuous Verification”. If not, no worries. There’s a solid article from Dan Illson and Bill Shetti on The New Stack that explains it in detail. The short version: Continuous Verification means putting as many automated checks as possible into your CI/CD pipelines. More checks, fewer manual tasks, more data to smooth out and improve your development and deployment process.
In part one we covered the tools and technologies and in part two we covered the Continuous Integration aspect of the ACME Serverless Fitness Shop. This post is about Infrastructure as Code.
What is the ACME Serverless Fitness Shop #
Quick recap: the ACME Serverless Fitness Shop combines two of my favorite things — serverless and fitness. It has seven distinct domains, each with one or more serverless functions. Some are event-driven, others have an HTTP API, and all of them are written in Go.
Infrastructure as Code #
The Wikipedia page for Infrastructure as Code describes it as “the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.”
Put simply, Infrastructure as Code makes your infrastructure programmable. And yes, “serverless” means you shouldn’t worry about servers and VMs, but you still need to think about storage, API gateways, and databases. There are good reasons IaC is becoming the norm: faster provisioning (especially in the cloud), fewer human configuration errors (assuming the code is correct), easier multi-region deployments, and reduced risk when team members leave — the code stays behind.
Tools for Infrastructure as Code #
As I mentioned in the first part of the series, IaC means moving infrastructure creation into the CI/CD pipeline as much as possible. There are plenty of options:
- Terraform: Solid tool, but you’re writing infrastructure in a different language (HCL) than the rest of your code.
- Serverless Framework: One of the first to simplify building and deploying serverless functions, but developers still have to orchestrate different parts of their apps.
- AWS CloudFormation and SAM: AWS-native, with useful SAM templates, but it’s another syntax to learn.
- Pulumi: An open-source IaC tool that works across clouds and lets you use real programming languages.
What makes Pulumi different #
I’m a developer (or developer advocate), which means I’m definitely not a YAML expert. The languages I enjoy are TypeScript and Go. When I think about those languages, I expect loops, variables, modules, and frameworks. Pulumi is the tool that lets me mix infrastructure with actual code. To create three similar IAM roles, I write a for loop instead of copying and pasting a statement three times. That matters for developer experience.
Speaking of developer experience — we expect syntax highlighting, IDE support, and strongly typed objects. Defining infrastructure with the same concepts we use for application code is what sets Pulumi apart.
Configuration #
Each domain has a pulumi folder with the configuration and code needed to deploy its services to AWS. Pulumi uses a configuration file for setting variables:
config:
aws:region: us-west-2 ## The region you want to deploy to
awsconfig:generic:
sentrydsn: ## The DSN to connect to Sentry
accountid: ## Your AWS Account ID
awsconfig:tags:
author: retgits ## The author, you...
feature: acmeserverless ## The resources are part of a specific app (the ACME Serverless Fitness Shop)
team: vcs ## The team you're on
version: 0.2.0 ## The version of the appTo use this configuration in a Pulumi program, there are two Go structs that map the key/value pairs to strongly typed variables:
// Tags are key-value pairs to apply to the resources created by this stack
type Tags struct {
// Author is the person who created the code, or performed the deployment
Author pulumi.String
// Feature is the project that this resource belongs to
Feature pulumi.String
// Team is the team that is responsible to manage this resource
Team pulumi.String
// Version is the version of the code for this resource
Version pulumi.String
}
// GenericConfig contains the key-value pairs for the configuration of AWS in this stack
type GenericConfig struct {
// The AWS region used
Region string
// The DSN used to connect to Sentry
SentryDSN string `json:"sentrydsn"`
// The AWS AccountID to use
AccountID string `json:"accountid"`
}To populate these structs, Pulumi provides a RequireObject method that reads the configuration and returns an error if the expected YAML element isn’t found:
// Get the region
region, found := ctx.GetConfig("aws:region")
if !found {
return fmt.Errorf("region not found")
}
// Read the configuration data from Pulumi.<stack>.yaml
conf := config.New(ctx, "awsconfig")
// Create a new Tags object with the data from the configuration
var tags Tags
conf.RequireObject("tags", &tags)
// Create a new GenericConfig object with the data from the configuration
var genericConfig GenericConfig
conf.RequireObject("generic", &genericConfig)
genericConfig.Region = regionBuilding code #
You could use Make to build the Go executable and zip file that Lambda needs. But since we’re already using Go, why not use Go for that too? I built a Go module that handles it. Four lines of code to create the executable and zip file. And because Pulumi mixes infrastructure with real code, you can add loops to build multiple functions or conditions to build selectively.
fnFolder := path.Join(wd, "..", "cmd", "lambda-payment-sqs")
buildFactory := builder.NewFactory().WithFolder(fnFolder)
buildFactory.MustBuild()
buildFactory.MustZip()Finding resources #
Not all resources your app needs are in the same stack. Things like SQS queues or DynamoDB tables might live in a completely different stack, but you still need access to them.
// Lookup the SQS queues
responseQueue, err := sqs.LookupQueue(ctx, &sqs.LookupQueueArgs{
Name: fmt.Sprintf("%s-acmeserverless-sqs-payment-response", ctx.Stack()),
})
if err != nil {
return err
}
requestQueue, err := sqs.LookupQueue(ctx, &sqs.LookupQueueArgs{
Name: fmt.Sprintf("%s-acmeserverless-sqs-payment-request", ctx.Stack()),
})
if err != nil {
return err
}Here we’re looking up the two SQS queues used for payment requests and credit card validation responses. The queue names and ARNs are needed to configure IAM policies and event source mappings.
Creating IAM policies #
While I enjoy Pulumi’s Go SDK, there are areas where AWS SAM speeds things up. IAM policies are one of them. SAM lets you pick from a list of policy templates to scope Lambda permissions to the resources your app uses. To get something similar in Pulumi, I built a Go module that wraps those policy templates for use in any Go app.
// Create a factory to get policies from
iamFactory := sampolicies.NewFactory().WithAccountID(genericConfig.AccountID).WithPartition("aws").WithRegion(genericConfig.Region)
// Add a policy document to allow the function to use SQS as event source
iamFactory.AddSQSSendMessagePolicy(responseQueue.Name)
iamFactory.AddSQSPollerPolicy(requestQueue.Name)
policies, err := iamFactory.GetPolicyStatement()
if err != nil {
return err
}
_, err = iam.NewRolePolicy(ctx, "ACMEServerlessPaymentSQSPolicy", &iam.RolePolicyArgs{
Name: pulumi.String("ACMEServerlessPaymentSQSPolicy"),
Role: role.Name,
Policy: pulumi.String(policies),
})
if err != nil {
return err
}These few lines of Go create an IAM policy that lets the Lambda function send messages to and receive messages from the two queues. The Go module saves me from writing a bunch of IAM policy statements by hand.
Deploying functions #
// Create the AWS Lambda function
functionArgs := &lambda.FunctionArgs{
Description: pulumi.String("A Lambda function to validate creditcard payments"),
Runtime: pulumi.String("go1.x"),
Name: pulumi.String(fmt.Sprintf("%s-lambda-payment", ctx.Stack())),
MemorySize: pulumi.Int(256),
Timeout: pulumi.Int(10),
Handler: pulumi.String("lambda-payment-sqs"),
Environment: environment,
Code: pulumi.NewFileArchive("../cmd/lambda-payment-sqs/lambda-payment-sqs.zip"),
Role: role.Arn,
Tags: pulumi.Map(tagMap),
}
function, err := lambda.NewFunction(ctx, fmt.Sprintf("%s-lambda-payment", ctx.Stack()), functionArgs)
if err != nil {
return err
}
_, err = lambda.NewEventSourceMapping(ctx, fmt.Sprintf("%s-lambda-payment", ctx.Stack()), &lambda.EventSourceMappingArgs{
BatchSize: pulumi.Int(1),
Enabled: pulumi.Bool(true),
FunctionName: function.Arn,
EventSourceArn: pulumi.String(requestQueue.Arn),
})
if err != nil {
return err
}The function arguments look the same as they would in any other Lambda deployment tool — runtime, memory size, IAM role, etc. The Payment service is triggered by SQS messages, so it needs a NewEventSourceMapping() to connect the function to the queue. The mapping uses the IAM role from the function arguments to verify the function has permission to receive messages, and it’ll throw an error if it doesn’t.
Why use Pulumi and how does Continuous Verification play a role? #
Fair question: the Pulumi code is about twice the size of the CloudFormation template that does the same thing. So why bother?
For me, it comes down to the same arguments from earlier. Pulumi lets me write deployments in the same language as my application code, gives me strongly typed variables, and provides access to all the tooling I already use for development — IDE support, testing, loops, the works. That makes building and maintaining the serverless infrastructure a lot more natural.
Those same reasons are why Pulumi fits well with Continuous Verification. The built-in previews, the ability to verify that resources were created as expected, and the ability to iterate on your infrastructure code all help you make an informed decision about whether your code should go to production.
What’s next? #
Next up, we’ll look at the observability side of serverless with VMware Tanzu Observability by Wavefront.
Photo by panumas nikhomkhai from Pexels
