Kubernetes is an open-source container orchestration system. It is fast-growing product supported by Google. If you think about using Docker it can be a good idea to do the next DevOps step and start using Kubernetes (K8s)
But maybe the more important question isn’t “How?”, but “What?” what benefits K8s can provide, what can I expect from it.
This article describes how K8s cluster can be created in DevOps style and what it can provide to the customer.
Here are targets we need to achieve when building K8s in a proper DevOps style:
- Infrastructure as Code (IaC).
- High Availability (HA).
- Optimized Docker Images.
- Automate everything.
- Provide User Friendly interface for typical operations, like deploy new version of App.
- Continuous Integration.
- Monitor it.
- Be informed.
Tools we will use:
- AWS as cloud provider.
- Ansible to automate everything.
- Ansible vault to keep secrets.
- AWS CloudFormation for IaC.
- CoreOS as base system.
- Alpine Linux as base layer for Docker images.
- Fleet to orchestrate CoreOS instances.
- Bash scripts here and there, sorry but nova days Systemd and Docker come with scripts.
- Version control system to keep changes, i.e. GitLab, GitHub, etc.
- External services for monitoring, logs aggretation, Docker Registry, etc. (can be replaced by in-house solutions).
3.1 Infrastructure as Code
We have several options to implement IaC:
All of them have pros and cons, check this blog and in the future there will be post comparing these options. For example, in case of
AWS CloudFormation we would:
- Put all cluster details in Ansible’s variables.
- Do not hardcode in CloudFormation templates, instead make them
- Use nested stacks.
3.2.1 AWS design
Good idea is to follow AWS VPC design recommendations. All of our EC2 CoreOS instances go to private subnets (no public ip). For administrative access we will use VPN. We will use ELBs to provide access from the Internet to Application(s) inside K8S.
Actually K8S has lots of security options, but we can start from something simple, like using Cluster TLS as it described here. Definitely it should be automated with ansible and private keys will be Vaulted. Also we will need RBAC for authorization.
All secrets should be kept in Ansible Vault. This way no clear text password or private key will be exposed in VCS.
In many situations we may need to run Private Docker Registry with storage on AWS S3. And we should consider to have all images we are using in this private repo. Also we can audit images for potential security threats, for example we can use Clair for it.
3.3 High Availability
Some Cloud provides such as Amazon AWS have concept of
Availability Zone or AZ. Idea is - you can distribute your virtual circuit (VPC) among several AZs, this is called as
multi-az design. It allows you to run several copies of services in different AZ, so in case of failure in one AZ you will still have working nodes in others. We will use it, also we will put everything to
AWS AutoScalling groups. We will are going to create at least two managed NAT Gateways, also we can use two openvpn instances for administrative access in different zones. Depending on Production cluster size we can use 3 or more Core OS
master instances (where etcd lives). Remote access to: fleet, K8s api and Docker private Registry will be ogranized via internal ELBs.
3.4 Optimized Docker Images
Consider to use Alpine Linux as base layer for Docker Images to get smaller images and speed up operations.
Also there are tweaks and tricks to get Docker images smaller by
We should follow Twelve-Factor Ap rules and separate configuration options and Docker images. So application should get its configuration from fleet/K8s. Best way to do it in K8s is to use Secrets object and in fleet we can use Environment variables or command line options.
3.5 Automate everything
To bootstrap our K8s cluster we will need these playbooks:
- Create CloudFormation stack on AWS.
- Configure VPN instances (after instances provision we should use it to get access access inside our new VPC).
- Start Private Docker Registry (playbook will use
fleetctlto start it).
- Build all Docker images we will need in this cluster and push them to already started Private Docker Registry.
- Start all needed processes on CoreOS instances (again Ansible will call
fleetctlto do it), actually there are bunch of processes we will need, like - NewRelic, Logspout, DataDog, etc, and in this step we will install K8s on CoreOS (using custom Docker Image) and also install configuration for
Kubeleton all instances.
- Start all Infrastructure resources on K8s, like
- Start Application(s).
Actually all of these can be placed in one shell script, so when the time comes to create a new cluster, we will do this:
- Checkout project repo from VCS.
- Put configuration details about new cluster (actually there are lots of details, like really lots), this step will take most of the time.
- Run script.
- Enjoy, App is running in a freshly made K8s cluster.
- And don’t forget to commit/push changes back to the VCS.
3.6 User Friendly interface
ChatOps is a very innovative and easy to use way to operate and maintain your cluster. We can create Slack ChatBot to automate most used tasks, like:
- Manually trigger Apps Image Build from specific git id (branch, hash, tag).
- Deploy App (we going to use Kubernetes Deployment for this).
- Scale/downscale cluster.
- Create database backup/snapshot.
3.7 Continuous Integration
We can have automatic App Docker image build and we can have releases to be automatically deployed to K8s cluster. If the team is ready for this we can even implement Blue-Green deployment process here (but this is where App should be really ready to allow this, database migrations is one of the biggest challenges).
3.8 Monitor it
3.9 Be informed
Slack rocks, Ansible should report to it, monitoring should report to it and App should report to it. But not only, also we need all logs to be aggregated in one place. We can deploy inhouse solution or we can use some external service.
Currently Kubernetes is mature enough to use it in production, but choosing the right path is vital for project success: all tiny details are important. Cupermind company and its friendly team is ready to help your business to complete this task right and effective.