Rapid AWS Lambda development with Go and Mantil

9 minute read
Go Ecosystem
aws  ·  lambda  ·  serverless
Share:

If you need to develop an AWS Lambda function in Go, take a look at Mantil, a dev kit with staging and database connection included.

AWS Lambda was among the first FaaS services available commercially. Recently, I came across a tool that promises to make Lambda development faster and easier: Mantil. To test the tool and the docs, I created a lambda function that receives a Google Form response and stores the answers in a DynamoDB table.

Project setup

Installation was pleasantly quick and easy via Homebrew. Mantil also has a cloud-side component, but apart from creating an AWS user (if you do not have one already), I did not need to do anything on AWS to install Mantil. Mantil does this by itself through

mantil aws install

The install command takes an AWS user's API credentials and the AWS region to install the cloud components to. This, and the corresponding uninstall command, are the only places where the AWS user's API credentials are needed.

Creating a project from scratch

After the cloud-side installation was finished, I created a new project through

mantil new mantil-template-form-to-dynamodb

Inside the new project directory, a sample ping endpoint waits to be explored. The starting point of a Mantil endpoint is a struct to hold the data to send back and forth. The Ping struct holds no data – after all, ping only pongs.

Along with the struct, Mantil created also a Default()method that responds on the root path of the lambda function, /ping, and more sample methods like a Hello() method that receives and sends a string.

The data – ping has none.

type Ping struct{}

The Mantil framework invokes the constructor at startup time.

func New() *Ping {
	return &Ping{}
}

Default() responds to the endpoint /ping.

func (p *Ping) Default() string {
	return "pong"
}

Hello() responds to the endpoint /ping/hello.

func (p *Ping) Hello(ctx context.Context, name string) (string, error) {
	return "Hello, " + name, nil
}

The base project is ready to run. Calling

mantil deploy

deployed the ping API.

A cool feature of Mantil is the support for staged deployments. When mantil deploy is invoked for the first time, it asks for a default stage to create. I entered “development”.

After a few seconds, I could then test the /ping endpoint right on the command line via

mantil invoke ping

and got the result from the Default() method back. Likewise,

mantil invoke ping/hello/AppliedGo

triggers the Hello() method.

Creating a new API

Next, I wanted to create a new API that receives a Google Form response and stores the answers in a DynamoDB table.

Mantil generated the code for me via

mantil generate api form

where form is the name of the new API. This is the generated code:

A struct to receive the JSON payload of the function call

type Form struct{}

Structs for the request and response parameters.

type DefaultRequest struct{}
type DefaultResponse struct{}

Like the default Ping endpoint, the genreated endpoint also has New() and Default() methods.

func New() *Form {
	return &Form{}
}

func (f *Form) Default(ctx context.Context, req *DefaultRequest) (*DefaultResponse, error) {
	panic("not implemented")
}

How to send Google Form data to a Lambda function

A Google form was quickly created from one of the available sample forms - the party invitation form.

Mantil Party Invite

Small problem: Google forms do not send Webhooks. So I had to create a custom webhook that sends the data to the Lambda function. Luckily, I found a script and the steps to apply it to the Google form here.

I edited the script to call a new endpoint /form/save with the form data as JSON payload.

var POST_URL = "https://SOME_UINQUE_ID.execute-api.AWS_REGION.amazonaws.com";
function onSubmit(e) {
    var form = FormApp.getActiveForm();
    var allResponses = form.getResponses();
    var latestResponse = allResponses[allResponses.length - 1];
    var response = latestResponse.getItemResponses();
    var payload = {};
    for (var i = 0; i < response.length; i++) {
        var question = response[i].getItem().getTitle();
        var answer = response[i].getResponse();
        payload[question] = answer;
    }

    console.log(JSON.stringify(payload))

    var options = {
        "method": "post",
        "contentType": "application/json",
        "payload": JSON.stringify(payload)
    };
UrlFetchApp.fetch(POST_URL + "/form/save", options);
};

What is the required value for POST_URL? This depends on the active Mantil project stage. The command

mantil stage list

lists all stages along with their endpoints:

| DEFAULT | NAME | NODE |                         ENDPOINT                          |
|---------|------|------|-----------------------------------------------------------|
| *       | dev  | dev  | https://blahblah.execute-api.eu-central-1.amazonaws.com |

But at that point, I did not know exactly how the JSON payload looks. So I added a /save endpoint to the form API that did nothing but logging the received payload.

I write about the function signature later.

func (f *Form) Save(ctx context.Context, req *Form) (string, error) {
	log.Printf("Save: req is '%+v'", req)
	return "", nil
}

This way, I got hold of the raw request data.

{"What is your name?":"Cat R. Pillar","Can you attend?":"Yes,  I'll be there","How many of you are attending?":"3","What will you be bringing?":["Drinks"],"Do you have any allergies or dietary restrictions?":"Nope","What is your email address?":"[email protected]"}

Then, using Matt Holt's indispensable JSON-to-Go service to turn the JSON payload into a Go struct. The result looks kind of funny because the Javascript code at Google form side uses the form's questions 1-1 as field names.

type Form struct {
	Name         string   `json:"What is your name?"`
	CanYouAttend string   `json:"Can you attend?"`
	Count        string   `json:"How many of you are attending?"`
	Items        []string `json:"What will you be bringing?"`
	Restrictions string   `json:"Do you have any allergies or dietary restrictions?"`
	Email        string   `json:"What is your email address?"`
}

I could have pimped the JS script to substitute simpler JSON field names for the questions, but… nah. Too lazy.

Don't lose the RSVP's for your party. Or: from struct to DynamoDB

To save the received form data in DynamoDB, I needed a table. Mantil's Go package provides a ready-to-use Key-Value store but I wanted to create and use a table from scratch.

For this, I added some internal fields to the Form struct:

type Form struct {

… same fields as above… The DynamoDB client, which represents the table

	table *dynamodb.Client

The table name

	tableName string

The Mantil resource name of the table. This is needed later as input to DynamoDB operations.

	tableResourceName *string
}

Note that these are private fields. They do not get in the way when using the Form struct for receiving the JSON payload.

Then, I created a New() function that takes care of creating the table.

func New() *Form {

Mantil manages stage-specific environments where the code reads the table name from.

	tableName := os.Getenv("TABLE_NAME")
	if tableName == "" {

Use a default name if the environment variable is not set.

		tableName = "MantilPartyTable"
	}

Create the table if it does not exist. TableKey and TableSortKey are constants that I have set to “email” and “name” respectively.

	table, err := mantil.DynamodbTable(tableName, TableKey, TableSortKey)
	if err != nil {
		log.Fatalf("Cannot create table: %s", err)
	}

Now create and return the Form struct.

	return &Form{
		table:             table,
		tableResourceName: aws.String(mantil.Resource(tableName).Name),
	}
}

Side note: staged deployments made easy

A really useful feature in this context is that Mantil manages stage-specific environments. In a YAML file, I have specified two different table names for the development and production stages.

project:
  stages:
    - name: development
      env:
        TABLE_NAME: MantilPartyDev
    - name: production
      env:
        TABLE_NAME: MantilParty

This way, development and production data get out of each other's way.

Making the Save() function actually save something

Next, I extended the /form/save endpoint stub from merely logging the data to saving it to DynamoBD. Note the parameter list and return list. I can choose to pass strings or structs as request or response parameters. Mantil does the necessary conversions behind the scenes. Super convenient.

Save receives the Party form response and saves it to a DynamoDB table.

func (f *Form) Save(ctx context.Context, req *Form) (string, error) {

types is an AWS package for bridging the gap between Go's static typing and DynamoDB's quite dynamic handling of table schemas.

	items := &types.AttributeValueMemberSS{Value: req.Items}

in case the form submitter does not check off any items to bring to the party, we need to create an empty items variable for DynamoDB.

	if items == nil || len(items.Value) == 0 {
		items = &types.AttributeValueMemberSS{Value: []string{""}}
	}

This is AWS's way of adding or updating a record in a DynamoDB table.

	_, err := f.table.PutItem(context.TODO(), &dynamodb.PutItemInput{
		TableName: f.tableResourceName,
		Item: map[string]types.AttributeValue{
			"name":         &types.AttributeValueMemberS{Value: req.Name},
			"canattend":    &types.AttributeValueMemberS{Value: req.CanYouAttend},
			"count":        &types.AttributeValueMemberS{Value: req.Count},
			"items":        items,
			"restrictions": &types.AttributeValueMemberS{Value: req.Restrictions},
			"email":        &types.AttributeValueMemberS{Value: req.Email},
		},
	})

Here, we send a response and an error value back. In the context of Google Forms webhooks, the response string is not very useful, but let's keep it for the sake of showing how it's done.

	if err != nil {
		log.Printf("Cannot save form: %s", err)
		return "Cannot save form", err
	}
	return fmt.Sprintf("%s saved", req.Name), nil
}

Testing the lambda function from the command line

With all this code in place, I can now run mantil deploy, and the lambda function is ready to receive form submissions.

Because I did not want to open the AWS console every time I need to check the table, I added a /form/list endpoint that dumps the table contents as JSON data.

The list endpoint returns a list of all the records in the table.

func (f *Form) List(ctx context.Context, req *DefaultRequest) (*[]Form, error) {

DynamoDB's Scan operation, without an optional filter, returns all the records in the table.

	out, err := f.table.Scan(context.TODO(), &dynamodb.ScanInput{
		TableName: f.tableResourceName,
	})
	if err != nil {
		log.Printf("Cannot scan table: %s", err)
	}

Turn the DynamoDB output into a list of Form structs.

	var forms []Form
	for _, item := range out.Items {
		forms = append(forms, Form{
			Name:         (item["name"]).(*types.AttributeValueMemberS).Value,
			CanYouAttend: (item["canattend"]).(*types.AttributeValueMemberS).Value,
			Count:        (item["count"]).(*types.AttributeValueMemberS).Value,
			Items:        (item["items"]).(*types.AttributeValueMemberSS).Value,
			Restrictions: (item["restrictions"]).(*types.AttributeValueMemberS).Value,
			Email:        (item["email"]).(*types.AttributeValueMemberS).Value,
		})
	}

Send the list of Form structs back to the caller.

	return &forms, nil
}

This endpoind allowed me to simply run

mantil invoke form/list

to get the current contents of the table back:

200 OK
[
   {
      "What is your name?": "Cat R. Pillar",
      "Can you attend?": "Yes,  I'll be there",
      "How many of you are attending?": "3",
      "What will you be bringing?": [
         "Drinks"
      ],
      "Do you have any allergies or dietary restrictions?": "Nope",
      "What is your email address?": "[email protected]"
   }
]

I guess it's time to start the party! 🥳

Odds and ends

I tested a few more things, most notably

  • a second stage with a different table name, and
  • Mantil's built-in testing. Mantil tests follow the standard Go testing convention and can be called with mantil test.

I omit the details here as the article is already long enough, don't you think? Have a look at my Mantil-Party-Form repository for more details.

Find the repository with the complete example here:

christophberger/mantil-template-form-to-dynamodb: Receive form data and write it to a DynamoDB table

Mantil links

Homepage: Mantil

Repositories:

Mantil: mantil-io/mantil: Build your AWS Lambda-based Go backends quicker than ever

Mantil Go package: mantil-io/mantil.go: Go SDK for Mantil projects

Happy coding!

Get the Applied Go Weekly Newsletter

Stay up to date in the world of Go. Get a hand-curated, commented list of articles and projects, and more, every week right in your inbox.

By subscribing, you agree to the privacy policy.

Powered by Buttondown.

Share
Like this article? Tell your friends!