Ansible in masterful mode

by (January 05, 2017)

Posted in Provision  Tags:Ansible, DevOps, Tools

By default there is no place for master in Ansible world. But sometimes it would be a good idea to have one, for example if there is an Auto Scaling Group with fresh OS as base image (i.e. Ubuntu 16.04 LTS) and provision process should be automatically started when the new instance gets added to this group.

Ansible master

It can be done using knockd.

Instances

There is a Cloud-Init to start a script which will knock Ansible master. For example:

#!/bin/bash
test -e /usr/bin/knock || (apt -y update && apt install -y knockd)
MASTER='ansible.example.com'
/usr/bin/knock $MASTER port1 port2 port3 port4 port5

Where “port1” - “port5” are different secret ports. This sequence acts as a password.

Master

Ansible master

Master is running knockd with knockd.conf file:

[options]
UseSyslog

[slaves]
sequence    = port1, port2, port3, port4, port5
seq_timeout = 5
command     = /ansible/provision.sh %IP%
tcpflags    = syn

And /ansible/provision.sh can be something like:

#!/bin/bash
if [ -z "$1" ]
then
    echo "Need an IP of instance to provision"
    exit 1
fi

BASEDIR="/ansible"
LOGDIR="$BASEDIR/logs"
USER="ansible"
KEEP_LOGS_DAYS=7

NOW=`date +%Y%m%d%H%M%S`

#check if logdir exists
if [ ! -d $LOGDIR ]; then
  /bin/mkdir $LOGDIR
  chown $USER $LOGDIR
fi

#delete old logs
/usr/bin/find $LOGDIR -type f -mtime +$KEEP_LOGS_DAYS -delete

#run script
sudo -H -u $USER sh -c \
"cd /$BASEDIR && \
./venv-ansible-playbook -i clusters/example/hosts provision.yml \
    -e \"IP=$1\" > $LOGDIR/$NOW.log 2>&1"

# send provision log to syslog
logger -t "provision" -f "$LOGDIR/$NOW.log"

There is venv-ansible-playbook script used, more information about it in our blog, also there is a post describing directories structure we are using in our projects.

Static inventories will not work in this scenario, but dynamic inventory would help, for example:

- add_host:
    name={{ IP }}
    groups=knockers
    ansible_user=ubuntu
    ansible_become=yes
    ansible_ssh_private_key_file=clusters/example/files/key
  changed_when: False
  failed_when: IP is not defined or not (IP | ipaddr)
  tags:
  - always

Master-slave communication

  1. Cloud provider creates a new instance, it boots and Cloud-Init runs knock script.
  2. On master knockd get knocked and starts provision.sh script.
  3. provision.sh in turn runs ansible-playbook with IP address of slave instance as an argument.
  4. Ansible provisions instance.

Real life experience

For one of our customers we build Highly Available (HA) system with 2 Ansible masters in different Availability Zones (AZ), this system also has ChatOps interface which allows end users execute Ansible’s playbooks from Slack.

Example of using Ansible Masters

Having Ansible Masters also allows us to run periodic tasks (playbooks) in cluster.

Pros and cons

Pros

Using above steps Master-slave architecture can be built. It was already said that we are using it in production for one of our clients. It works. And it gives some possibilities.

Example of Ansible Master usage

Having Ansible Master changes the way DevOps interact with the cluster. Typical example of Ansible provision is when there is some inventory file and DevOps directly provisions instances from his PC. This is not always possible or is not optimal, i.e. when there is a large amount of instances it will take too much time (here we can mention that Ansible becomes slower each version). Instead of using Ansible to directly provision instances, DevOps can use Ansible to provision Ansible Master and tell it to actually do things over Cluster. And also we can have different interfaces to Ansible Masters such as:

  1. Slack (ChatOps).
  2. Rundeck (web interface).
  3. Webhook handlers for Continues Integration (CI).
  4. Run periodic tasks.

In case of really large configuration multi-master configuration can be used to split load.

Cons

Example of using knockd in this article is a bit simplified, for example what if by some reason Ansible didn’t get knocked from the first time. In real life we are actually knocking Ansible using cron every 5 minutes and we are using lock files to prevent Ansible master running several playbooks against the same instance at the same time.

Let us know!

Contact details:

 

Services you are interested in: