OS Upgrade

This is a list of steps as guidelines for OS upgrades.

  • review APT pinnings in Ansible
  • create a temporary group_vars/<new-suite>/system.yml in a git branch:
      codename: bookworm
      codename: bookworm

  • check if there other places where the old suite name is used and update them
  • (I often choose Elwing first) ansible-playbook --diff -l Elwing -t apt playbooks/common.yml | tee /tmp/dc.log
  • run selected playbooks that update pinnings and the other suite names (run -t certificates playbooks/common.yml for bookworm)
  • apt upgrade and then apt full-upgrade (check if removed packages are problematic):
    • accept new version of:
      • /etc/services and copy-paste content of local services from /etc/services.dpkg-old to avoid having to redeploy all services
      • /etc/grub.d/10_linux but see warning below
      • /etc/ssh/ssh_config (we use ssh_config.d in Ansible now) BUT NOT sshd_config!
    • do not accept new versions for:
      • /etc/apt-cacher-ng/acng.conf
      • /etc/apache2/*
      • /etc/chrony/chrony.conf
      • /etc/default/grub
      • /etc/logrotate.d/*
      • /etc/oidentd.conf
      • /etc/smartd.conf
      • /etc/snmp/snmp.conf
      • /etc/sudoers
      • /etc/ulogd.conf
    • check the diff manually for other files
  • apt autoremove
  • apt clean
  • if PHP FPM: (to avoid having to redeploy all vhosts) (example for PHP 7.4->8.2)
    • rm /etc/php/8.2/fpm/pool.d/www.conf
    • cp /etc/php/7.4/fpm/pool.d/* /etc/php/8.2/fpm/pool.d/
    • sed -i 's/7\.4/8.2/g' /etc/php/8.2/fpm/pool.d/*
    • systemctl restart php8.2-fpm.service
    • sed -i 's/7\.4/8.2/g' /etc/apache2/sites-enabled/*.conf.d/php.conf
    • systemctl restart apache2
    • purge facts_cache/<host> before running Ansible to detect the new major version
    • run common web playbook playbooks/tenants/duckcorp/web.yml with -t web-common
  • run the playbooks/common.yml playbook
  • on MX1 servers, run the playbooks/tenants/duckcorp/mail.yml playbook with -t antispam (to switch the Rspamd repo to the new suite)
  • run the playbooks/tenants/duckcorp/accounts.yml playbook
  • run the playbooks/common_post.yml playbook
  • run the playbooks/tenants/duckcorp/monitoring.yml playbook
  • run the playbooks/cleanup.yml playbook

It is critical that the common playbook is run successfully before rebooting. Especially /etc/grub.d/10_linux must contain the --unrestricted option and the GRUB config must be regenerated or the server will block at the GRUB screen waiting for a login.

  • reboot
  • check failed services: systemctl --failed


  • check apt-forktracer output

Updated by Marc Dequènes 9 months ago · 28 revisions