Lab Setup

Lab overview

lab setup

In the lab you will deploy a CloudFormation stack that will create resources for you in AWS. It deploys a Lambda function (Lambda Stream Function), triggered by DynamoDB table’s stream. It deploys a second Lambda function (Lambda Wiring Function) that reads the source data from S3 (#1) and loads it into the DynamoDB table (#2). The Lambda stream function, which is already in place, catches the updates to the table and inserts the records into Amazon ES (#3). Finally, you will run a third Lambda function (Lambda Streaming Function) to generate updates to the DynamoDB table (#4), which the Lambda stream function will propagate to Amazon ES.

The source data is a set of 246, Sci-Fi/Action movies, sourced from IMDb. The Lambda wiring function reads this data, adds clicks, price, purchases, and a location field, and creates the item in the DynamoDB table.

The DynamoDB schema uses the movie’s id as the primary key and adds items for the movie data’s fields.

{
    "id": "tt0088763",
    "title": "Back to the Future",
    "year": 1985,
    "rating": 8.5,
    "rank": 204,
    "genres": [
        "Adventure",
        "Comedy",
        "Sci-Fi"
    ],
    "plot": "A teenager is accidentally sent 30 years into the past in a time-traveling DeLorean invented by his friend, Dr. Emmett Brown, and must make sure his high-school-age parents unite in order to save his own existence.",
    "release_date": "1985-07-03T00:00:00Z",
    "running_time_secs": 6960,
    "actors": [
        "Christopher Lloyd",
        "Lea Thompson",
        "Michael J. Fox"
    ],
    "directors": [
        "Robert Zemeckis"
    ],
    "image_url": "http://ia.media-imdb.com/images/M/MV5BMTk4OTQ1OTMwN15BMl5BanBnXkFtZTcwOTIwMzM3MQ@@._V1_SX400_.jpg",
    "location": "44.54, -67.0 ",
    "purchases": 12460,
    "price": 29.99,
    "clicks": 1277594
}

These fields will also serve as the basis for querying the data in Elasticsearch.