Running periodical processes on AWS

by (January 27, 2017)

Posted in Clouds  Tags:AWS, Clouds, Tools, EiiaC

Sometimes there is a need to run some process on a daily basis, as an example it can be backup task.

Lets imagine that periodic process is defined in a version control system (VCS) repo and we need to run it daily having this in mind:

  1. It should be easy to implement.
  2. Reliable.
  3. Easy to maintenance.
  4. Cost-effective.
  5. Cool.

Stand alone AWS EC2 instance

Start an instance, put there a script and run it from Cron. Script should pull repo to get latest version of process it starts.

Pros

Very easy to implement.

Cons

Is it reliable? At first sight yes, but what if instance stops working - there need be a human action on it.

Is it easy to maintenance? Yes and no, not a big deal to have some instance running forever, but actually nothing runs forever and when it breaks there can be a problem because some details can be forgotten.

As for cost effectiveness it depends on a process, but usually periodical tasks runs hours not days and all other time it is just heating air/burning money.

Is it cool - no, it is boring.

Serverless using AWS Lambda

AWS Lambda is a hype technology, it allows you to write a code and forget about servers, theoretically it can be possible to wrap a process in Python or NodeJS code and run it there.

Pros

It can be cost-effective.

And it is very cool.

Cons

But what about implementation, is it easy to implement it using AWS Lambda functions? Hell no. Lambda has limitations and more suites REST API application. To solve our task we will need to build Async system to trigger some tasks somewhere and then check their statuses.

Is is reliable? More moving parts - more options to break.

Self-terminated EC2 instance in AWS Auto Scaling Group

Put a script to cloud-init, in this script:

  1. Add user which will start process.
  2. Configure ssh to access rep.
  3. Clone repo.
  4. Start process from repo.
  5. When process finishes change (using IAM Role to allow this action) number of instances in ASG to zero (self-terminate).
  6. configure Scheduled Scaling to change number of instances in ASG to 1 daily.

Block Diagram

Pros

It is a very reliable solution with self-healing options.

It is easy to support, everything is in a code (cloud-init).

And it is cost-effective, because instance will run only when periodic process runs.

And it looks very cool also.

Cons

It is a bit harder to implement comparing to standalone EC2 instance but much easier than AWS Lambda.

Summary

Latest option Self-terminated EC2 instance in AWS Auto Scaling Group seems the best among others and suites just invented technology Everything is in a Code (E2aC).

Let us know!

Contact details:

 

Services you are interested in: