Ansible in production: idempotency, drift detection and safe rolling changes

A playbook is usually written and tested against a clean machine. In production it runs against machines in whatever state they ended up in: a config file edited by hand during an incident, a package upgraded by unattended-upgrades, a previous run canceled halfway. What matters there is whether the playbook converges a host from an unknown state to the declared one, and whether it tells you what it had to change on the way.

Idempotent is not the same as convergent

Idempotent means a second run changes nothing. Convergent means any starting state ends in the declared one. A task can have the first property without the second: lineinfile guarding one line won't notice another line someone added next to it by hand.

Most modules get this for free because they compare current and desired state (template, copy, service with state: started, apt with state: present), so a second run with the same arguments reports ok. It doesn't hold for command and shell. Ansible can't see inside them, so they report changed on every run unless you pass creates or removes (the task then doesn't run when the given file already exists or is already gone) or tell Ansible yourself when something changed:

- name: Run database migrations
  ansible.builtin.command: php artisan migrate --force
  args:
    chdir: /var/www/app
  register: migrate
  changed_when: "'Nothing to migrate' not in migrate.stdout"

Without changed_when, every run reports at least one change, and the change count, the cheapest drift signal you have, stops meaning anything.

Making drift visible

When Ansible quietly fixes a host, the fact that someone changed it disappears. Take a service whose autostart an engineer turned off by hand (systemctl disable) while debugging:

- name: Ensure application service is running
  ansible.builtin.service:
    name: invoicer
    state: started
    enabled: true

The task fixes the state and reports changed. The report doesn't say whether that was a first install or a manual override being reverted. If you want to know, read the state before converging it:

- name: Read current enablement state
  ansible.builtin.command: systemctl is-enabled invoicer
  register: svc_enabled
  changed_when: false
  failed_when: false
  check_mode: false

- name: Report manual override
  ansible.builtin.debug:
    msg: "invoicer is '{{ svc_enabled.stdout }}' on {{ inventory_hostname }}, expected 'enabled'"
  when: svc_enabled.stdout != 'enabled'

- name: Converge service state
  ansible.builtin.service:
    name: invoicer
    state: started
    enabled: true

check_mode: false matters here. command tasks are skipped in check mode by default, so without it the read wouldn't run during a dry run.

For the whole fleet, the practical drift detector is a scheduled dry run:

ansible-playbook site.yml --check --diff

If the playbooks have correct changed_when everywhere and no unconditional restarts in tasks, any host with changed>0 in the recap differs from what they declare, and the diff shows which lines. It isn't exact. Check mode can't predict tasks that depend on the result of an earlier task that would have changed something, and modules without check mode support are skipped. What you get is a list of hosts worth looking at.

Rolling changes with rollback

Picture a deploy job that gets interrupted while updating the nginx upstream configuration. Part of the fleet now runs the old config and the rest runs the new one. You need to converge all hosts to one version, a few at a time, and stop as soon as one host fails:

- name: Converge nginx upstream config
  hosts: api_servers
  become: true
  serial: 2
  max_fail_percentage: 0

  handlers:
    - name: Reload nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

  tasks:
    - name: Deploy, verify, roll back on failure
      block:
        - name: Check whether upstream config exists
          ansible.builtin.stat:
            path: /etc/nginx/conf.d/upstream.conf
          register: upstream_before

        - name: Render upstream config
          ansible.builtin.template:
            src: templates/upstream.conf.j2
            dest: /etc/nginx/conf.d/upstream.conf
            owner: root
            group: root
            mode: '0644'
            backup: true
          register: upstream_conf
          notify: Reload nginx

        - name: Test full nginx configuration
          ansible.builtin.command: nginx -t
          changed_when: false

        - name: Reload now instead of at the end of the play
          ansible.builtin.meta: flush_handlers

        - name: Wait for health endpoint
          ansible.builtin.uri:
            url: "http://127.0.0.1:{{ nginx_port }}/health"
            status_code: 200
            timeout: 5
          register: health
          until: health.status == 200
          retries: 5
          delay: 2

      rescue:
        - name: Restore previous config
          ansible.builtin.copy:
            src: "{{ upstream_conf.backup_file }}"
            dest: /etc/nginx/conf.d/upstream.conf
            remote_src: true
            mode: '0644'
          when: upstream_conf.backup_file is defined

        - name: Remove config that did not exist before
          ansible.builtin.file:
            path: /etc/nginx/conf.d/upstream.conf
            state: absent
          when: not upstream_before.stat.exists

        - name: Reload nginx with restored config
          ansible.builtin.service:
            name: nginx
            state: reloaded

        - name: Mark host as failed
          ansible.builtin.fail:
            msg: "upstream.conf rolled back on {{ inventory_hostname }}"

What each part does:

• serial: 2 processes the inventory in batches of two hosts. With eight hosts, at most two are being changed at any moment.
• max_fail_percentage: 0 stops the rollout after the first task on which a host counts as failed. A failure inside block is rescued, so here that only happens at the explicit fail, and by then the other host in the batch has usually finished the block with the new config. The next batch never starts. Without it, Ansible only stops when every host in a batch fails, so a 1 of 2 failure would let the next batch proceed.
• nginx -t tests the whole configuration with the rendered file included. A conf.d snippet with an upstream block isn't a valid standalone config, so the validate option of template doesn't work for it. validate is fine for files like nginx.conf that nginx can test on their own.
• Handlers normally run at the end of each section of the play (pre_tasks, roles and tasks, post_tasks), per batch. meta: flush_handlers reloads nginx before the health check, which goes through nginx on nginx_port. Otherwise the check would test the old config.
• backup: true keeps a timestamped copy and returns its path in backup_file. The rescue section restores it, reloads, and then fails the host explicitly, so max_fail_percentage stops the rollout. If the file didn't exist before, there is no backup, so rescue deletes the new file and nginx reloads without it. The host still ends up failed, so the rollout stops.
• reloaded instead of restarted: new connections go to the new workers while the old ones finish the requests already in progress, so a reload doesn't interrupt them. With a broken config nginx stays on the old one.

This playbook doesn't drain hosts from the load balancer. For a plain nginx reload that's usually unnecessary. If you restart the application, take the host out of the balancer in pre_tasks and put it back in post_tasks.

Secrets

Secrets that belong to the playbook go into an encrypted vars file, referenced from a plain one. The vault_ prefix convention from the Ansible documentation makes it obvious where a value comes from:

# group_vars/prod/vars.yml
db_password: "{{ vault_db_password }}"

# group_vars/prod/vault.yml, encrypted with: ansible-vault encrypt group_vars/prod/vault.yml
vault_db_password: "..."

Secrets shared with other systems are better read at runtime from a secret manager through a lookup plugin (the amazon.aws and community.hashi_vault collections provide them), so the value lives in one place.

Any task that renders or passes a secret needs no_log: true. It hides the task result in logs and in --diff output. Use diff: false if you only want to suppress the diff:

- name: Render database config
  ansible.builtin.template:
    src: templates/database.env.j2
    dest: /var/www/app/.env.database
    owner: www-data
    mode: '0640'
  no_log: true

Set owner and mode explicitly, even when they seem obvious. In CookieDomain, my cookie consent system, the deploy script created a config file with mktemp, which gives it mode 600. mv kept that mode and rsync -a carried it to production. The web server runs as www-data and couldn't read the file, so the require in the bootstrap failed and both production and dev returned 500 on every route. The application logs had nothing, the fatal error was only in the nginx log. The fix was chmod 644 before mv, rsync --chmod=F644,D755, and is_readable() instead of is_file(). For a file with secrets 644 is too open, because any user on the system can read it. There 0640 with the web server's group, or www-data as the owner as in the example above, is the better choice. Running the deploy again didn't fix it, because every run created the file the same way.

If a secret has already been committed in plain text, rewriting git history isn't enough, because clones and CI caches still have it. Rotate the secret first, then clean up.

Testing before production

For roles, Molecule creates a container or VM, applies the role, runs it a second time to check idempotency, and runs a verifier. Since Molecule 3.0 the default verifier is Ansible, so Testinfra has to be set explicitly. With Testinfra, the tests check the state the host actually ended up in, whatever Ansible itself reported:

# molecule/default/tests/test_nginx.py
def test_nginx_running_and_enabled(host):
    nginx = host.service("nginx")
    assert nginx.is_running
    assert nginx.is_enabled


def test_nginx_config_valid(host):
    assert host.run("nginx -t").rc == 0

On a host with systemd, is_running and is_enabled ask systemd. In a plain container without it, Testinfra falls back to the service command and the links in /etc/rc?.d/, so it checks something other than the systemd you run in production. To test against systemd, you need an image with systemd or a VM.

Hand-written deploy scripts and checks are easy to get wrong. In one of my deploys docker kill turned out not to test the restart policy at all; I needed kill -9 on the process PID from the host. Another deploy script ran composer install --no-dev and so removed PHPUnit before the tests got to run. And in a script started with ssh ... bash -s, a plain read from stdin swallowed the rest of the script, and garbage ended up in .env.

Before a production run, a dry run against one representative host is fast and shows the exact template diffs:

ansible-playbook site.yml --check --diff --limit 'api_servers[0]'

Before approving a playbook change

• changed_when on every command and shell task, and failed_when where the exit code means nothing.
• No ignore_errors: true on infrastructure tasks. A failure should stop the play so the host stays in a state you can see in the report. If you need explicit recovery, that's what block/rescue is for.
• become: true only where root is needed.
• serial and a deliberately chosen max_fail_percentage on anything that touches more than one host.