OS Upgrade¶

This is a list of steps as guidelines for OS upgrades.

• review APT pinnings in Ansible
• create a temporary group_vars/<new-suite>/system.yml in a git branch:

---
system:
  distribution:
    debian:
      codename: bookworm
    dc:
      codename: bookworm

• check if there other places where the old suite name is used and update them
• (I often choose Elwing first) ansible-playbook --diff -l Elwing -t apt playbooks/common.yml | tee /tmp/dc.log
• run selected playbooks that update pinnings and the other suite names (run -t certificates playbooks/common.yml for bookworm)
• apt upgrade and then apt full-upgrade (check if removed packages are problematic):
  • accept new version of:
    • /etc/services and copy-paste content of local services from /etc/services.dpkg-old to avoid having to redeploy all services
    • /etc/grub.d/10_linux but see warning below
    • /etc/ssh/ssh_config (we use ssh_config.d in Ansible now) BUT NOT sshd_config!
  • do not accept new versions for:
    • /etc/apt-cacher-ng/acng.conf
    • /etc/apache2/*
    • /etc/chrony/chrony.conf
    • /etc/default/grub
    • /etc/logrotate.d/*
    • /etc/oidentd.conf
    • /etc/smartd.conf
    • /etc/snmp/snmp.conf
    • /etc/sudoers
    • /etc/ulogd.conf
  • check the diff manually for other files
• apt autoremove
• apt clean
• if PHP FPM: (to avoid having to redeploy all vhosts) (example for PHP 7.4->8.2)
  • rm /etc/php/8.2/fpm/pool.d/www.conf
  • cp /etc/php/7.4/fpm/pool.d/* /etc/php/8.2/fpm/pool.d/
  • sed -i 's/7\.4/8.2/g' /etc/php/8.2/fpm/pool.d/*
  • systemctl restart php8.2-fpm.service
  • sed -i 's/7\.4/8.2/g' /etc/apache2/sites-enabled/*.conf.d/php.conf
  • systemctl restart apache2
  • purge facts_cache/<host> before running Ansible to detect the new major version
  • run common web playbook playbooks/tenants/duckcorp/web.yml with -t web-common
• run the playbooks/common.yml playbook
• on MX1 servers, run the playbooks/tenants/duckcorp/mail.yml playbook with -t antispam (to switch the Rspamd repo to the new suite)
• run the playbooks/tenants/duckcorp/accounts.yml playbook
• run the playbooks/common_post.yml playbook
• run the playbooks/tenants/duckcorp/monitoring.yml playbook
• run the playbooks/cleanup.yml playbook

It is critical that the common playbook is run successfully before rebooting. Especially /etc/grub.d/10_linux must contain the --unrestricted option and the GRUB config must be regenerated or the server will block at the GRUB screen waiting for a login.

• reboot
• check failed services: systemctl --failed

Later:

• check apt-forktracer output