DuckCorp » DuckCorp Infrastructure

Today Toushirou was restarted unexpectedly. It seems that this restart wasn't due a command.

The server was restarted after Dec 13 10:07:03 (UTC+1). I unlocked the encrypted encryption around 13h15 (UTC+1).

syslog contains:

Dec 13 10:06:52 Toushirou postfix/smtpd[1353160]: disconnect from <redacted> ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7
Dec 13 10:07:03 Toushirou stunnel: LOG5[8632]: Connection closed: 182 byte(s) sent to TLS, 20 byte(s) sent to socket
@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
[...]
@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Dec 13 13:18:38 Toushirou systemd-udevd[631]: Using default interface naming scheme 'v247'.
Dec 13 13:18:38 Toushirou systemd-udevd[630]: Using default interface naming scheme 'v247'.
Dec 13 13:18:38 Toushirou lvm[578]:   3 logical volume(s) in volume group "extra" monitored

The filesystem journals were recovered:

Dec 13 13:18:38 Toushirou systemd-fsck[791]: /dev/md0 was not cleanly unmounted, check forced.

Dec 13 13:18:38 Toushirou systemd-fsck[790]: /dev/mapper/main-ldap: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[790]: /dev/mapper/main-ldap: clean, 14/23616 files, 9468/94208 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-ldap.

Dec 13 13:18:38 Toushirou systemd-fsck[787]: /dev/mapper/main-ftp: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[787]: /dev/mapper/main-ftp: clean, 1042/1966080 files, 4094072/7864320 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-ftp.

Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: Clearing orphaned inode 524490 (uid=0, gid=4, mode=0100640, size=186)
Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: Clearing orphaned inode 525136 (uid=0, gid=4, mode=0100640, size=2261619)
[...]
Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: clean, 3025/915712 files, 701679/3661824 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-logs.

Dec 13 13:18:38 Toushirou systemd-fsck[797]: /dev/mapper/main-mysql: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[797]: /dev/mapper/main-mysql: clean, 1706/305216 files, 302945/1220608 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-mysql.

Dec 13 13:18:38 Toushirou systemd-fsck[801]: /dev/mapper/main-projects: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[801]: /dev/mapper/main-projects: clean, 15384/977280 files, 2501362/3932160 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-projects.

Dec 13 13:18:38 Toushirou systemd-fsck[805]: /dev/mapper/main-stuffcloud: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[805]: /dev/mapper/main-stuffcloud: clean, 184647/8519680 files, 22560629/34078720 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-stuffcloud.

Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: Clearing orphaned inode 136445 (uid=0, gid=0, mode=0100664, size=11567160)
Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: Clearing orphaned inode 136045 (uid=0, gid=0, mode=0100664, size=9253600)
[...]
Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: clean, 43941/305216 files, 677459/1220608 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-var.

Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: Clearing orphaned inode 20 (uid=0, gid=0, mode=0100666, size=0)
Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: Clearing orphaned inode 50 (uid=128, gid=136, mode=0100600, size=0)
[...]
Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: clean, 3380/121920 files, 20791/487424 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-tmp.

Dec 13 13:18:38 Toushirou systemd-fsck[814]: /dev/mapper/main-vcs: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[814]: /dev/mapper/main-vcs: clean, 62639/183264 files, 334140/732160 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-vcs.

Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: Clearing orphaned inode 1314229 (uid=5111, gid=5111, mode=0100600, size=2543956)
[...]
Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: clean, 38189/1966080 files, 3862291/7864320 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-vmail.

Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/extra-lxd.

Dec 13 13:18:38 Toushirou systemd-fsck[827]: /dev/mapper/extra-home: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[827]: /dev/mapper/extra-home: clean, 576437/19660800 files, 60022856/78643200 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/extra-home.

Dec 13 13:18:38 Toushirou systemd-fsck[791]: /dev/md0: 348/64000 files (23.9% non-contiguous), 63264/255936 blocks

Dec 13 13:18:38 Toushirou systemd-fsck[819]: /dev/mapper/main-www: recovering journal
Dec 13 13:18:38 Toushirou systemd-fsck[819]: /dev/mapper/main-www: clean, 417149/9175040 files, 7579187/36700160 blocks
Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-www.

Thanks to GuiHome and Victor for letting me know that the NextCloud service was unavailable.

Once the server has been restarted there was an error with the hivane network link. Hence some service were unavailable. The nerim link worked.

root@Toushirou:~# systemctl --failed
  UNIT                              LOAD   ACTIVE SUB    DESCRIPTION
● apache2.service                   loaded failed failed The Apache HTTP Server
● ifup@eth\x2dwan\x2dhivane.service loaded failed failed ifup for eth-wan-hivane
● matrix-appservice-irc.service     loaded failed failed Matrix AppService IRC
● networking.service                loaded failed failed Raise network interfaces

root@Toushirou:~# ifdown --force eth-wan-hivane
RTNETLINK answers: Cannot assign requested address
RTNETLINK answers: Cannot assign requested address
root@Toushirou:~# ifup --force eth-wan-hivane
Waiting for DAD... Timed out
ifup: failed to bring up eth-wan-hivane

I remember the timed out issue occurred when the last time the server was moved from a rack to another. I tried the ifdown/ifup commands several times (until the Timed out disappeared).

The logs show that the timed out issue occurred at boot:

Dec 13 13:18:45 Toushirou sh[1562]: Waiting for DAD... Timed out
Dec 13 13:18:45 Toushirou sh[1496]: ifup: failed to bring up eth-wan-hivane

Next I restarted apache2.service and matrix-appservice-irc.service, then I updated /lib/systemd/system/lxd.socket in order to fix a typo:

Dec 13 15:48:22 Toushirou systemd[1]: /lib/systemd/system/lxd.socket:8: Unit must be of type service, ignoring: lxd.servcie

After that i ran systemctl daemon-reload and lxc list then the redmine LXC container restarted.

At this time I tried to create this issue using redmine:https://projects.duckcorp.org/ but an issue occurred after i tried to authenticate: the redmine web interface showed an error: "Cannot assign requested address - connect(2) for [2001:67c:1740:9001::c1c8:2ab1]:636".

The restart of the slapd service (which was listening on IPv6 but not IPv4) fixed this issue.