DuckCorp » DuckCorp Infrastructure

h1. OS Upgrade

This is a list of steps as guidelines for OS upgrades.

* review APT pinnings in Ansible

* create a temporary _group_vars/<new-suite>/system.yml_ in a git branch:

<pre>

---

system:

  distribution:

    debian:

      codename: bookworm

    dc:

      codename: bookworm

</pre>

* check if there other places where the old suite name is used and update them

* (I often choose Elwing first) ansible-playbook --diff -l Elwing -t apt playbooks/common.yml | tee /tmp/dc.log

* run selected playbooks that update pinnings and the other suite names (run _-t certificates playbooks/common.yml_ for bookworm)

* _apt upgrade_ and then _apt full-upgrade_ (check if removed packages are problematic):

** accept new version of:

*** /etc/services and copy-paste content of _local services_ from _/etc/services.dpkg-old_ to avoid having to redeploy all services

*** /etc/grub.d/10_linux but see warning below

*** /etc/ssh/ssh_config (we use ssh_config.d in Ansible now) BUT NOT sshd_config!

** do not accept new versions for:

*** /etc/apt-cacher-ng/acng.conf

*** /etc/apache2/*

*** /etc/chrony/chrony.conf

*** /etc/default/grub

*** /etc/logrotate.d/*

*** /etc/oidentd.conf

*** /etc/smartd.conf

*** /etc/snmp/snmp.conf

*** /etc/sudoers

*** /etc/ulogd.conf

** check the diff manually for other files

* _apt autoremove_

* _apt clean_

* if PHP FPM: (to avoid having to redeploy all vhosts) (example for PHP 7.4->8.2)

** rm /etc/php/8.2/fpm/pool.d/www.conf

** cp /etc/php/7.4/fpm/pool.d/* /etc/php/8.2/fpm/pool.d/

** sed -i 's/7\.4/8.2/g' /etc/php/8.2/fpm/pool.d/*

** systemctl restart php8.2-fpm.service

** sed -i 's/7\.4/8.2/g' /etc/apache2/sites-enabled/*.conf.d/php.conf

** systemctl restart apache2

** purge facts_cache/<host> before running Ansible to detect the new major version

** run common web playbook _playbooks/tenants/duckcorp/web.yml_ with _-t web-common_

* run the _playbooks/common.yml_ playbook

* on MX1 servers, run the _playbooks/tenants/duckcorp/mail.yml_ playbook with _-t antispam_ (to switch the Rspamd repo to the new suite)

* run the _playbooks/tenants/duckcorp/accounts.yml_ playbook

* run the _playbooks/common_post.yml_ playbook

* run the _playbooks/tenants/duckcorp/monitoring.yml_ playbook

* run the _playbooks/cleanup.yml_ playbook

It is critical that the common playbook is run successfully before rebooting. Especially _/etc/grub.d/10_linux_ must contain the _--unrestricted_ option and the GRUB config must be regenerated or the server will block at the GRUB screen waiting for a login.

* reboot

* check failed services: _systemctl --failed_

Later:

* check _apt-forktracer_ output