Bug #746
closedunexpected restart of Toushirou host
50%
Description
Today Toushirou was restarted unexpectedly. It seems that this restart wasn't due a command.
The server was restarted after Dec 13 10:07:03
(UTC+1). I unlocked the encrypted encryption around 13h15 (UTC+1).
syslog
contains:
Dec 13 10:06:52 Toushirou postfix/smtpd[1353160]: disconnect from <redacted> ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7 Dec 13 10:07:03 Toushirou stunnel: LOG5[8632]: Connection closed: 182 byte(s) sent to TLS, 20 byte(s) sent to socket @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ [...] @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ Dec 13 13:18:38 Toushirou systemd-udevd[631]: Using default interface naming scheme 'v247'. Dec 13 13:18:38 Toushirou systemd-udevd[630]: Using default interface naming scheme 'v247'. Dec 13 13:18:38 Toushirou lvm[578]: 3 logical volume(s) in volume group "extra" monitored
The filesystem journals were recovered:
Dec 13 13:18:38 Toushirou systemd-fsck[791]: /dev/md0 was not cleanly unmounted, check forced. Dec 13 13:18:38 Toushirou systemd-fsck[790]: /dev/mapper/main-ldap: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[790]: /dev/mapper/main-ldap: clean, 14/23616 files, 9468/94208 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-ldap. Dec 13 13:18:38 Toushirou systemd-fsck[787]: /dev/mapper/main-ftp: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[787]: /dev/mapper/main-ftp: clean, 1042/1966080 files, 4094072/7864320 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-ftp. Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: Clearing orphaned inode 524490 (uid=0, gid=4, mode=0100640, size=186) Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: Clearing orphaned inode 525136 (uid=0, gid=4, mode=0100640, size=2261619) [...] Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: clean, 3025/915712 files, 701679/3661824 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-logs. Dec 13 13:18:38 Toushirou systemd-fsck[797]: /dev/mapper/main-mysql: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[797]: /dev/mapper/main-mysql: clean, 1706/305216 files, 302945/1220608 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-mysql. Dec 13 13:18:38 Toushirou systemd-fsck[801]: /dev/mapper/main-projects: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[801]: /dev/mapper/main-projects: clean, 15384/977280 files, 2501362/3932160 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-projects. Dec 13 13:18:38 Toushirou systemd-fsck[805]: /dev/mapper/main-stuffcloud: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[805]: /dev/mapper/main-stuffcloud: clean, 184647/8519680 files, 22560629/34078720 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-stuffcloud. Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: Clearing orphaned inode 136445 (uid=0, gid=0, mode=0100664, size=11567160) Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: Clearing orphaned inode 136045 (uid=0, gid=0, mode=0100664, size=9253600) [...] Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: clean, 43941/305216 files, 677459/1220608 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-var. Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: Clearing orphaned inode 20 (uid=0, gid=0, mode=0100666, size=0) Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: Clearing orphaned inode 50 (uid=128, gid=136, mode=0100600, size=0) [...] Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: clean, 3380/121920 files, 20791/487424 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-tmp. Dec 13 13:18:38 Toushirou systemd-fsck[814]: /dev/mapper/main-vcs: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[814]: /dev/mapper/main-vcs: clean, 62639/183264 files, 334140/732160 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-vcs. Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: Clearing orphaned inode 1314229 (uid=5111, gid=5111, mode=0100600, size=2543956) [...] Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: clean, 38189/1966080 files, 3862291/7864320 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-vmail. Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/extra-lxd. Dec 13 13:18:38 Toushirou systemd-fsck[827]: /dev/mapper/extra-home: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[827]: /dev/mapper/extra-home: clean, 576437/19660800 files, 60022856/78643200 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/extra-home. Dec 13 13:18:38 Toushirou systemd-fsck[791]: /dev/md0: 348/64000 files (23.9% non-contiguous), 63264/255936 blocks Dec 13 13:18:38 Toushirou systemd-fsck[819]: /dev/mapper/main-www: recovering journal Dec 13 13:18:38 Toushirou systemd-fsck[819]: /dev/mapper/main-www: clean, 417149/9175040 files, 7579187/36700160 blocks Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-www.
Thanks to GuiHome and Victor for letting me know that the NextCloud service was unavailable.
Once the server has been restarted there was an error with the hivane network link. Hence some service were unavailable. The nerim link worked.
root@Toushirou:~# systemctl --failed UNIT LOAD ACTIVE SUB DESCRIPTION ● apache2.service loaded failed failed The Apache HTTP Server ● ifup@eth\x2dwan\x2dhivane.service loaded failed failed ifup for eth-wan-hivane ● matrix-appservice-irc.service loaded failed failed Matrix AppService IRC ● networking.service loaded failed failed Raise network interfaces
root@Toushirou:~# ifdown --force eth-wan-hivane RTNETLINK answers: Cannot assign requested address RTNETLINK answers: Cannot assign requested address root@Toushirou:~# ifup --force eth-wan-hivane Waiting for DAD... Timed out ifup: failed to bring up eth-wan-hivane
I remember the timed out issue occurred when the last time the server was moved from a rack to another. I tried the ifdown
/ifup
commands several times (until the Timed out
disappeared).
The logs show that the timed out issue occurred at boot:
Dec 13 13:18:45 Toushirou sh[1562]: Waiting for DAD... Timed out Dec 13 13:18:45 Toushirou sh[1496]: ifup: failed to bring up eth-wan-hivane
Next I restarted apache2.service
and matrix-appservice-irc.service
, then I updated /lib/systemd/system/lxd.socket
in order to fix a typo:
Dec 13 15:48:22 Toushirou systemd[1]: /lib/systemd/system/lxd.socket:8: Unit must be of type service, ignoring: lxd.servcie
After that i ran
systemctl daemon-reload
and lxc list
then the redmine LXC container restarted.
At this time I tried to create this issue using redmine:https://projects.duckcorp.org/ but an issue occurred after i tried to authenticate: the redmine web interface showed an error: "Cannot assign requested address - connect(2) for [2001:67c:1740:9001::c1c8:2ab1]:636"
.
The restart of the slapd
service (which was listening on IPv6 but not IPv4) fixed this issue.
Updated by Pierre-Louis Bonicoli almost 3 years ago
- Description updated (diff)
- Status changed from New to In Progress
- % Done changed from 0 to 50
- Security set to No
Updated by Marc Dequènes over 1 year ago
- Status changed from In Progress to Rejected
I guess we'll never know what happened…