Ansible Playbook for Docker Swarm Node Rolling Updates

• Feb 2024
• Semaphore UI for Ansible v2.9.37

Objective:

• create an Ansible Playbook that will go through a cluster of Docker Swarm hosts and serially update them in a graceful manner.

Perspective

• I'm new to Ansible, so had, and still have, a lot to learn
• Lots of ways to improve this, and I expect to continue working on it.

---

Challenges

• Lots of challenges due to my inexperience - below are a few I'm highlighting

How do I register a variable and persist it between plays targeting different host groups?

• https://stackoverflow.com/questions/33896847/how-do-i-register-a-variable-and-persist-it-between-plays-targeted-on-different
• https://www.reddit.com/r/ansible/comments/g6wrj2/how_do_you_dynamically_delegate_tasks_to_a/

  • Very useful post, but got stuck on passing the active_mgr variable until I dug into using hostvars

  • hostvars.<host>.active_mgr

    Key Point:

    • the hostname you use in hostvars['hostname'] should match exactly with one of the hostnames defined in your Ansible inventory under the group you are referencing. Also, the variable you are trying to access should have been defined on the group you are referencing.
    • Not sure if there is an exact source, but Microsoft CoPilot brought that to my attention

Short vs long names

• I use FQDNs for many things in my homelab, and docker swarm just wants the short name of the node for control commands.
• So I added the ansible_host= to each host to cover this.

---

Other Resources Leveraged

• Starting point for rolling updates:
  • https://morph027.gitlab.io/blog/ansible-rolling-swarm-updates
  • Since this is from 2017, a number of the plays could be simplified, or syntax had changed.
• https://www.reddit.com/r/ansible/comments/tg8jt5/help_with_old_dockerswarm_playbook_to_upgrade/
• https://docs.ansible.com/ansible/2.9/modules/docker_node_module.html
• https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_delegation.html

---

Sample Inventory Content

[swarm_workers]
worker1 ansible_host=worker1.mydomain
worker2 ansible_host=worker2.mydomain
worker3 ansible_host=worker3.mydomain

[swarm_managers]
manager1 ansible_host=manager1.mydomain
manager2 ansible_host=manager2.mydomain
manager3 ansible_host=manager3.mydomain

Ansible Playbook

---
- hosts: swarm_managers
  tasks:
  - name: Pick an online swarm manager node to execute swarm control commands 
    set_fact:
      active_mgr: "{{ ansible_play_hosts[0] }}"

- hosts: '{{ host_list }}'
  serial: 1

  tasks:

  - name: Drain Active host - Delegate to the chosen online swarm manager node
    shell: docker node update --availability drain '{{ inventory_hostname }}'
    delegate_to:  "{{ hostvars.manager1.active_mgr }}"

  - name: apt dist-upgrade
    become: yes
    apt:
      update_cache: yes
      upgrade: dist
      autoremove: yes

  - name: Reboot the machine
    become: yes
    ansible.builtin.reboot:
      pre_reboot_delay: 60
      msg: "Rebooting machine in 60 seconds - Ansible"

  - name: Set Host to Active  - Delegate to the chosen online swarm manager node
    shell: docker node update --availability active '{{ inventory_hostname }}'
    delegate_to:  "{{ hostvars.manager1.active_mgr }}"

  - name: Pause for 60 seconds to allow swarm settling
    ansible.builtin.pause:
      seconds: 60