Implement Auto-scaling in AWS Kinesis Stream

In this article, I am going to show you how you can implement the auto-scaling in AWS Kinesis stream as AWS does not provide that off the shelf.

For those who does not know what is AWS Kinesis. So it is AWS Managed streaming service by which you can get the real-time data over a stream and process that data.

Now lets talk why we need to implement the auto-scaling for kinesis stream. So basically a streaming is consisting of shards. A shard is having capacity around 1mbps and whenever you want to support more traffic then one need to increase the number of shard, but suppose you are having a production application setup and traffic suddenly traffic increase more than your expectation and no one increases the number of shards then this may result losing the crucial information.

To solve this particular problem I started to searching for a solution then I found a solution which AWS is providing https://github.com/awslabs/amazon-kinesis-scaling-utils. I analyzed this solution but the problem was that I had to run this application on the EC2 which is costing me. Then I further looked for different solutions but find none of the efficient solution. After all this I decided to implement my own serverless solution (to be price efficient) to handle this problem.

This solution is developed using the following services of AWS

1. Cloudwatch Alerts
2. SNS
3. Lambda Function
4. SES