Kubernetes on AWS, DevOps way: the plan which works

by (September 30, 2016)

Posted in Containers  Tags:Ansible, AWS, CloudFormation, Kubernetes, CoreOS, Docker, Alpine Linux, Clouds

Kubernetes is an open-source container orchestration system. It is fast-growing product supported by Google. If you think about using Docker it can be a good idea to do the next DevOps step and start using Kubernetes (K8s)

There is documentation about creating K8s cluster on AWS in a Hard, manual way. Also there is a great tutorial on how to create K8S cluster on CoreOS, you should definitely read both of them.

But maybe the more important question isn’t “How?”, but “What?” what benefits K8s can provide, what can I expect from it.

This article describes how K8s cluster can be created in DevOps style and what it can provide to the customer.

1 Goals

Here are targets we need to achieve when building K8s in a proper DevOps style:

  1. Infrastructure as Code (IaC).
  2. Security.
  3. High Availability (HA).
  4. Optimized Docker Images.
  5. Automate everything.
  6. Provide User Friendly interface for typical operations, like deploy new version of App.
  7. Continuous Integration.
  8. Monitor it.
  9. Be informed.

2 Tools

Tools we will use:

  1. AWS as cloud provider.
  2. Ansible to automate everything.
  3. Ansible vault to keep secrets.
  4. AWS CloudFormation for IaC.
  5. CoreOS as base system.
  6. Alpine Linux as base layer for Docker images.
  7. Fleet to orchestrate CoreOS instances.
  8. Bash scripts here and there, sorry but nova days Systemd and Docker come with scripts.
  9. Version control system to keep changes, i.e. GitLab, GitHub, etc.
  10. External services for monitoring, logs aggretation, Docker Registry, etc. (can be replaced by in-house solutions).

AWS is top #1 choice for most of the companies, however in case of need the same result (more or less) can be achieved on any platform from Google Cloud to OpenStack.

3 Plan

3.1 Infrastructure as Code

We have several options to implement IaC:

  1. Plain ansible modules.
  2. HashiCorp Terraform.
  3. AWS CloudFormation.

All of them have pros and cons, check this blog and in the future there will be post comparing these options. For example, in case of AWS CloudFormation we would:

  1. Put all cluster details in Ansible’s variables.
  2. Do not hardcode in CloudFormation templates, instead make them Jinja2 templates.
  3. Use nested stacks.

3.2 Security

3.2.1 AWS design

AWS VPC design

Good idea is to follow AWS VPC design recommendations. All of our EC2 CoreOS instances go to private subnets (no public ip). For administrative access we will use VPN. We will use ELBs to provide access from the Internet to Application(s) inside K8S.

3.2.2 Kubernetes

Actually K8S has lots of security options, but we can start from something simple, like using Cluster TLS as it described here. Definitely it should be automated with ansible and private keys will be Vaulted. Also we will need RBAC for authorization.

3.2.3 Ansible

All secrets should be kept in Ansible Vault. This way no clear text password or private key will be exposed in VCS.

3.2.4 Docker

In many situations we may need to run Private Docker Registry with storage on AWS S3. And we should consider to have all images we are using in this private repo. Also we can audit images for potential security threats, for example we can use Clair for it.

3.3 High Availability

AWS VPC design

Some Cloud provides such as Amazon AWS have concept of Availability Zone or AZ. Idea is - you can distribute your virtual circuit (VPC) among several AZs, this is called as multi-az design. It allows you to run several copies of services in different AZ, so in case of failure in one AZ you will still have working nodes in others. We will use it, also we will put everything to AWS AutoScalling groups. We will are going to create at least two managed NAT Gateways, also we can use two openvpn instances for administrative access in different zones. Depending on Production cluster size we can use 3 or more Core OS master instances (where etcd lives). Remote access to: fleet, K8s api and Docker private Registry will be ogranized via internal ELBs.

3.4 Optimized Docker Images

Consider to use Alpine Linux as base layer for Docker Images to get smaller images and speed up operations.

Also there are tweaks and tricks to get Docker images smaller by Dockerfile optimizations.

We should follow Twelve-Factor Ap rules and separate configuration options and Docker images. So application should get its configuration from fleet/K8s. Best way to do it in K8s is to use Secrets object and in fleet we can use Environment variables or command line options.

3.5 Automate everything

To bootstrap our K8s cluster we will need these playbooks:

  1. Create CloudFormation stack on AWS.
  2. Configure VPN instances (after instances provision we should use it to get access access inside our new VPC).
  3. Start Private Docker Registry (playbook will use fleetctl to start it).
  4. Build all Docker images we will need in this cluster and push them to already started Private Docker Registry.
  5. Start all needed processes on CoreOS instances (again Ansible will call fleetctl to do it), actually there are bunch of processes we will need, like - NewRelic, Logspout, DataDog, etc, and in this step we will install K8s on CoreOS (using custom Docker Image) and also install configuration for Kubelet on all instances.
  6. Start all Infrastructure resources on K8s, like kubedns, Heapster, dashboard, etc.
  7. Start Application(s).

Actually all of these can be placed in one shell script, so when the time comes to create a new cluster, we will do this:

  1. Checkout project repo from VCS.
  2. Put configuration details about new cluster (actually there are lots of details, like really lots), this step will take most of the time.
  3. Run script.
  4. Enjoy, App is running in a freshly made K8s cluster.
  5. And don’t forget to commit/push changes back to the VCS.

3.6 User Friendly interface

Slack ChatOps

ChatOps is a very innovative and easy to use way to operate and maintain your cluster. We can create Slack ChatBot to automate most used tasks, like:

  1. Manually trigger Apps Image Build from specific git id (branch, hash, tag).
  2. Deploy App (we going to use Kubernetes Deployment for this).
  3. Scale/downscale cluster.
  4. Create database backup/snapshot.

3.7 Continuous Integration

We can have automatic App Docker image build and we can have releases to be automatically deployed to K8s cluster. If the team is ready for this we can even implement Blue-Green deployment process here (but this is where App should be really ready to allow this, database migrations is one of the biggest challenges).

3.8 Monitor it

There are some free options, like Heapster or Prometheus. Other and more easier to use options are DataDog and sysdig, both of them more or less support K8s and Docker.

3.9 Be informed

Slack rocks, Ansible should report to it, monitoring should report to it and App should report to it. But not only, also we need all logs to be aggregated in one place. We can deploy inhouse solution or we can use some external service.

4 Summary

Currently Kubernetes is mature enough to use it in production, but choosing the right path is vital for project success: all tiny details are important. Cupermind company and its friendly team is ready to help your business to complete this task right and effective.

Let us know!

Contact details:


Services you are interested in: