Project

General

Profile

Actions
Copy link

Enhancement #782

closed

Migrate to Debian Bookworm

Added by Marc Dequènes almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Low
Assignee:
Marc Dequènes
Category:
-
Start date:
2023-06-28
Due date:
% Done:

100%

Estimated time:
Patch Available:
Confirmed:
No
Branch:
bookworm
Entity:
DuckCorp
Security:
Help Needed:
No

Description

Quack,

This ticket is tracking the migration steps:
  • ✅ custom/backported packages:
    • ✅ identify packages that are still needed => spoolinger, python-certbot-dns-rfc2136
    • ✅ setup suite in custom repo
    • ✅ update packages and upload them
  • ✅ add non-free-firmware component to get CPU microcode updates; Nicecity also needs firmware-realtek unfortunately)
  • 🔳 needed software/config changes (to update in Ansible):
    • ✅ apache2: none
    • ✅ atheme-services: none
    • ✅ dovecot: expire plugin removed: use mailbox/autoexpunge instead
    • ✅ icecast2: none
    • ✅ inspircd: pcre module is now pcre2
    • 🔳 logcheck: rsyslog now defaults to “high precision timestamps” and we need to update regexs in our custom filters -> see #785
    • ✅ lxd: remove, we'll be using podman+quadlets instead (not needed for redmine anymore)
    • ✅ mailman3: none
    • ✅ mediawiki: config check needed
    • ✅ mariadb-server: innodb_large_prefix and innodb_file_format are deprecated and can simply be removed (we used the new default already)
    • ✅ matrix-synapse: missing, hopefully should come as backports like it did for Bullseye
    • ✅ openldap: none
    • ✅ passenger: none
    • ✅ postfix: none
    • ✅ postgresql: none
    • ✅💡 proftpd: missing, maybe it's time to deprecate it for good
    • ✅ prometheus: lots of changes but current config should be fine
    • ✅ redis-server: none
    • ✅ redmine: switch back to the package
    • ✅ roundcube: config has changed slightly
    • ✅ xl2tpd: control command renamed
    • …
  • ✅ upgrade: (follow the OS_Upgrade procedure) (suggested order)
    • ✅ Elwing
    • ✅ Nicecity
    • ✅ Orhos
    • ✅ Thorfinn
    • ✅ Jinta
    • ✅ Toushirou
    • ✅ Orfeo
  • 🔳 post-check: (possibly moved into specific ticket later)
    • 🔳 check services still missing systemd config (we might be able to remove some workaround in Ansible) -> see #786
    • 🔳 do we still need rsyslog? maybe for logcheck -> see #784
    • 🔳 remove obsolete logcheck filters (maybe start again from scratch and pull back rules we had when we hit them?) -> see #785
    • ✅ remove obsolete log files:
      • /var/log/mail.{info,warn,err}
      • /var/log/lpr.log
      • /var/log/{messages,debug,daemon.log}
    • anacron: check is it needs reenabling for software that did not switch to systemd
Actions
Copy link
 #1

Updated by Marc Dequènes almost 2 years ago

  • Subject changed from Migrate to Bookworm to Migrate to Debian Bookworm
  • Description updated (diff)
Actions
Copy link
 #2

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #5

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #6

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • Branch set to bookworm
Actions
Copy link
 #7

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #8

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #9

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #10

Updated by Marc Dequènes almost 2 years ago

On Nicecity we have a kernel trace:

Jun 29 11:15:57 Nicecity kernel: i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
Jun 29 11:15:57 Nicecity kernel: [drm] Skipping LVDS initialization for Gigabyte GA-D525TUD
Jun 29 11:15:57 Nicecity kernel: ------------[ cut here ]------------
Jun 29 11:15:57 Nicecity kernel: i915 0000:00:02.0: Useless DMI match. Internal LVDS support disabled by VBT
Jun 29 11:15:57 Nicecity kernel: WARNING: CPU: 0 PID: 336 at drivers/gpu/drm/i915/display/intel_lvds.c:840 intel_lvds_init+0x59b/0x760 [i915]
Jun 29 11:15:57 Nicecity kernel: Modules linked in: i915(+) evdev drm_buddy video wmi iTCO_wdt intel_pmc_bxt coretemp drm_display_helper iTCO_vendor_support sha512_ssse>
Jun 29 11:15:57 Nicecity kernel: CPU: 0 PID: 336 Comm: (udev-worker) Not tainted 6.1.0-9-amd64 #1  Debian 6.1.27-1
Jun 29 11:15:57 Nicecity kernel: Hardware name: Gigabyte Technology Co., Ltd. D525TUD/D525TUD, BIOS F9 12/14/2012
Jun 29 11:15:57 Nicecity kernel: RIP: 0010:intel_lvds_init+0x59b/0x760 [i915]
Jun 29 11:15:57 Nicecity kernel: Code: 08 75 af 48 8b 7b 08 48 8b 5f 50 48 85 db 75 03 48 8b 1f e8 b7 f3 91 c6 48 89 da 48 c7 c7 70 85 e4 c0 48 89 c6 e8 55 f4 2d c6 <0f>
Jun 29 11:15:57 Nicecity kernel: RSP: 0018:ffffbf2f006b3a48 EFLAGS: 00010282
Jun 29 11:15:57 Nicecity kernel: RAX: 0000000000000000 RBX: ffff96df00b54eb0 RCX: 0000000000000000
Jun 29 11:15:57 Nicecity kernel: RDX: 0000000000000002 RSI: ffffffff8833fa66 RDI: 00000000ffffffff
Jun 29 11:15:57 Nicecity kernel: RBP: ffff96df03cd02b8 R08: 0000000000000000 R09: ffffbf2f006b38b8
Jun 29 11:15:57 Nicecity kernel: R10: 0000000000000003 R11: ffffffff88ad43a8 R12: ffff96df03cd02c0
Jun 29 11:15:57 Nicecity kernel: R13: 0000000000000001 R14: 0000000000000000 R15: ffff96df00c2b000
Jun 29 11:15:57 Nicecity kernel: FS:  00007fe6e3ac28c0(0000) GS:ffff96df3bc00000(0000) knlGS:0000000000000000
Jun 29 11:15:57 Nicecity kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 29 11:15:57 Nicecity kernel: CR2: 00007f6a1979ae20 CR3: 0000000103f5c000 CR4: 00000000000006f0
Jun 29 11:15:57 Nicecity kernel: Call Trace:
Jun 29 11:15:57 Nicecity kernel:  <TASK>
Jun 29 11:15:57 Nicecity kernel:  ? delay_tsc+0x8f/0xc0
Jun 29 11:15:57 Nicecity kernel:  intel_modeset_init_nogem+0x10ba/0x1150 [i915]
Jun 29 11:15:57 Nicecity kernel:  ? _raw_spin_unlock_irq+0x1b/0x40
Jun 29 11:15:57 Nicecity kernel:  ? intel_irq_postinstall+0x43a/0x610 [i915]
Jun 29 11:15:57 Nicecity kernel:  i915_driver_probe+0x5c5/0xe50 [i915]
Jun 29 11:15:57 Nicecity kernel:  ? i915_pci_probe+0x94/0x1d0 [i915]
Jun 29 11:15:57 Nicecity kernel:  local_pci_probe+0x3e/0x80
Jun 29 11:15:57 Nicecity kernel:  pci_device_probe+0xc3/0x230
Jun 29 11:15:57 Nicecity kernel:  really_probe+0xdb/0x380
Jun 29 11:15:57 Nicecity kernel:  ? pm_runtime_barrier+0x50/0x90
Jun 29 11:15:57 Nicecity kernel:  __driver_probe_device+0x78/0x120
Jun 29 11:15:57 Nicecity kernel:  driver_probe_device+0x1f/0x90
Jun 29 11:15:57 Nicecity kernel:  __driver_attach+0xce/0x1c0
Jun 29 11:15:57 Nicecity kernel:  ? __device_attach_driver+0x110/0x110
Jun 29 11:15:57 Nicecity kernel:  bus_for_each_dev+0x84/0xd0
Jun 29 11:15:57 Nicecity kernel:  bus_add_driver+0x1ae/0x200
Jun 29 11:15:57 Nicecity kernel:  driver_register+0x89/0xe0
Jun 29 11:15:57 Nicecity kernel:  i915_init+0x1f/0x7f [i915]
Jun 29 11:15:57 Nicecity kernel:  ? 0xffffffffc0f4e000
Jun 29 11:15:57 Nicecity kernel:  do_one_initcall+0x56/0x220
Jun 29 11:15:57 Nicecity kernel:  do_init_module+0x4a/0x200
Jun 29 11:15:57 Nicecity kernel:  __do_sys_finit_module+0xac/0x120
Jun 29 11:15:57 Nicecity kernel:  do_syscall_64+0x58/0xc0
Jun 29 11:15:57 Nicecity kernel:  ? syscall_exit_to_user_mode+0x17/0x40
Jun 29 11:15:57 Nicecity kernel:  ? do_syscall_64+0x67/0xc0
Jun 29 11:15:57 Nicecity kernel:  ? exit_to_user_mode_prepare+0x147/0x1d0
Jun 29 11:15:57 Nicecity kernel:  ? syscall_exit_to_user_mode+0x17/0x40
Jun 29 11:15:57 Nicecity kernel:  ? do_syscall_64+0x67/0xc0
Jun 29 11:15:57 Nicecity kernel:  ? do_syscall_64+0x67/0xc0
Jun 29 11:15:57 Nicecity kernel:  ? do_syscall_64+0x67/0xc0
Jun 29 11:15:57 Nicecity kernel:  entry_SYSCALL_64_after_hwframe+0x63/0xcd
Jun 29 11:15:57 Nicecity kernel: RIP: 0033:0x7fe6e3d4d4f9
Jun 29 11:15:57 Nicecity kernel: Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48>
Jun 29 11:15:57 Nicecity kernel: RSP: 002b:00007ffec70cfe08 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
Jun 29 11:15:57 Nicecity kernel: RAX: ffffffffffffffda RBX: 000055af12abe3e0 RCX: 00007fe6e3d4d4f9
Jun 29 11:15:57 Nicecity kernel: RDX: 0000000000000000 RSI: 00007fe6e3ee0efd RDI: 0000000000000016
Jun 29 11:15:57 Nicecity kernel: RBP: 00007fe6e3ee0efd R08: 0000000000000000 R09: 000055af12aba430
Jun 29 11:15:57 Nicecity kernel: R10: 0000000000000016 R11: 0000000000000246 R12: 0000000000020000
Jun 29 11:15:57 Nicecity kernel: R13: 0000000000000000 R14: 000055af12b7dfe0 R15: 000055af116eae4f
Jun 29 11:15:57 Nicecity kernel:  </TASK>
Jun 29 11:15:57 Nicecity kernel: ---[ end trace 0000000000000000 ]---
Jun 29 11:15:57 Nicecity kernel: i915 0000:00:02.0: [drm] Initialized overlay support.
Jun 29 11:15:57 Nicecity kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0

I can't se the display so maybe there's an impact but the rest of the initialization goes on and the machine seem to work fine. Could not find the same exact problem on the Internet, so I'm not sure what to do with that.

Actions
Copy link
 #11

Updated by Marc Dequènes almost 2 years ago

On Nicecity openipmi fails to start:

Jun 29 11:18:32 Nicecity kernel: ipmi_si: IPMI System Interface driver
Jun 29 11:18:32 Nicecity kernel: ipmi_si: Unable to find any System Interface(s)
Jun 29 11:18:32 Nicecity openipmi[3581]: Starting ipmi drivers ipmi failed!

Why do we even have this installed in the first place?

Actions
Copy link
 #12

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
  • % Done changed from 0 to 20
Actions
Copy link
 #13

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)

On Orthos the ninjabot Python venv needed to be upgraded to the new Python major version:

python3 -m venv --upgrade /opt/ninjabot//venv

Actions
Copy link
 #14

Updated by Marc Dequènes almost 2 years ago

On Orthos same problem as Nicecity with openipmi.

Actions
Copy link
 #15

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #16

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #17

Updated by Marc Dequènes almost 2 years ago

On Elwing logrotate failed, affecting Orfeo, Jinta and Thorfinn too:

un 30 00:00:01 Elwing systemd[1]: Starting logrotate.service - Rotate log files...
Jun 30 00:00:01 Elwing logrotate[84610]: /usr/sbin/invoke-rc.d: 538: /etc/init.d/rsyslog: not found
Jun 30 00:00:01 Elwing logrotate[84607]: invoke-rc.d: initscript rsyslog, action "rotate" failed.
Jun 30 00:00:01 Elwing logrotate[84524]: error: error running non-shared postrotate script for /var/log/burp-client.log of '/var/log/burp-client.log '
Jun 30 00:00:02 Elwing systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Jun 30 00:00:02 Elwing systemd[1]: logrotate.service: Failed with result 'exit-code'.
Jun 30 00:00:02 Elwing systemd[1]: Failed to start logrotate.service - Rotate log files.

In /etc/logrotate.d/burp is does a invoke-rc.d rsyslog rotate. Now that the sysv config has vanished this cannot work therefore I deleted the file on all hosts.

This config is not in the "new" Ansible repo and I cannot updte the rules accordingly.

Actions
Copy link
 #18

Updated by Marc Dequènes almost 2 years ago

  • Description updated (diff)
  • % Done changed from 20 to 40
Actions
Copy link
 #19

Updated by Marc Dequènes over 1 year ago

  • Description updated (diff)
Actions
Copy link
 #20

Updated by Marc Dequènes over 1 year ago

Investing problems with the Apache on Toushirou: it is very slow and often times out. It can be experienced from another Internet access, is visible in the monitoring, and can even be reproduced locally to some extent, therefore this is not network related. There is no resource shortage, no lack of entropy, and nothing useful in the logs so far. I'll keep investigating.

Actions
Copy link
 #21

Updated by Marc Dequènes over 1 year ago

  • Description updated (diff)
Actions
Copy link
 #22

Updated by Marc Dequènes over 1 year ago

On Toushirou this was a DNS problem: Hivane resolvers did not allow Toushirou's Nerim IPs to resolve but with the nameserver rotation I fail to see it during my tests.

Actions
Copy link
 #23

Updated by Marc Dequènes over 1 year ago

Fixed redmine registration, this is package bug and I'm preparing an official upload but in the meanwhile I built a package for us.

Actions
Copy link
 #24

Updated by Marc Dequènes over 1 year ago

  • Description updated (diff)
  • % Done changed from 40 to 70

Fixed a few things (see repo).

openipmi is not needed anymore, therefore I removed it.

Toushirou should be in good shape now.

Actions
Copy link
 #25

Updated by Marc Dequènes over 1 year ago

I had to update the database scheme s/postgres/postgresql/ in mailman.cfg in database/url because the scheme was deprecated in sqlalchemy. I'm not sure if this is my fault or if the package used it by mistake since it seems this part was generated (we do not own this file in Ansible, only make a small change).

Actions
Copy link
 #26

Updated by Marc Dequènes over 1 year ago

  • Description updated (diff)
  • % Done changed from 70 to 90

phppgadmin does not support PG15 in Bookworm, what a shame.

Actions
Copy link
 #27

Updated by Marc Dequènes over 1 year ago

Orfeo has one disk down:

ioc0 vol_id 2 type IM, 2 phy, 67 GB, state DEGRADED, flags ENABLED
ioc0 phy 0 scsi_id 4 SEAGATE  ST973402SS       S229, 68 GB, state ONLINE, flags NONE
ioc0 phy 1 scsi_id 3 SEAGATE  ST973401LSUN72G  0556, 68 GB, state MISSING, flags OUT_OF_SYNC

I'll defer rebooting the machine.

Actions
Copy link
 #28

Updated by Marc Dequènes over 1 year ago

Made some more changes and merged the branch.

Actions
Copy link
 #29

Updated by Marc Dequènes over 1 year ago

We won't reboot Orfeo and Pilou will take care of preparing Orfeo's new body (#652).

On my side I'll be moving services just in case (#783).

Actions
Copy link
 #30

Updated by Marc Dequènes over 1 year ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved
  • % Done changed from 90 to 100
Actions
Copy link

Also available in: Atom PDF