Project

General

Profile

Bug #746

Updated by Pierre-Louis Bonicoli about 2 years ago

Today Toushirou was restarted unexpectedly. It seems that this restart wasn't due a command. 

 The server was restarted after @Dec 13 10:07:03@ (UTC+1). I unlocked the encrypted encryption around 13h15 (UTC+1). 

 @syslog@ contains: 
 <pre> 
 Dec 13 10:06:52 Toushirou postfix/smtpd[1353160]: disconnect from <redacted> ehlo=2 starttls=1 mail=1 rcpt=1 bdat=1 quit=1 commands=7 
 Dec 13 10:07:03 Toushirou stunnel: LOG5[8632]: Connection closed: 182 byte(s) sent to TLS, 20 byte(s) sent to socket 
 @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ 
 [...] 
 @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ 
 Dec 13 13:18:38 Toushirou systemd-udevd[631]: Using default interface naming scheme 'v247'. 
 Dec 13 13:18:38 Toushirou systemd-udevd[630]: Using default interface naming scheme 'v247'. 
 Dec 13 13:18:38 Toushirou lvm[578]:     3 logical volume(s) in volume group "extra" monitored 
 </pre> 

 The filesystem journals were recovered: 
 <pre> 
 Dec 13 13:18:38 Toushirou systemd-fsck[791]: /dev/md0 was not cleanly unmounted, check forced. 

 Dec 13 13:18:38 Toushirou systemd-fsck[790]: /dev/mapper/main-ldap: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[790]: /dev/mapper/main-ldap: clean, 14/23616 files, 9468/94208 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-ldap. 

 Dec 13 13:18:38 Toushirou systemd-fsck[787]: /dev/mapper/main-ftp: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[787]: /dev/mapper/main-ftp: clean, 1042/1966080 files, 4094072/7864320 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-ftp. 

 Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: Clearing orphaned inode 524490 (uid=0, gid=4, mode=0100640, size=186) 
 Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: Clearing orphaned inode 525136 (uid=0, gid=4, mode=0100640, size=2261619) 
 [...] 
 Dec 13 13:18:38 Toushirou systemd-fsck[794]: /dev/mapper/main-logs: clean, 3025/915712 files, 701679/3661824 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-logs. 

 Dec 13 13:18:38 Toushirou systemd-fsck[797]: /dev/mapper/main-mysql: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[797]: /dev/mapper/main-mysql: clean, 1706/305216 files, 302945/1220608 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-mysql. 

 Dec 13 13:18:38 Toushirou systemd-fsck[801]: /dev/mapper/main-projects: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[801]: /dev/mapper/main-projects: clean, 15384/977280 files, 2501362/3932160 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-projects. 

 Dec 13 13:18:38 Toushirou systemd-fsck[805]: /dev/mapper/main-stuffcloud: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[805]: /dev/mapper/main-stuffcloud: clean, 184647/8519680 files, 22560629/34078720 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-stuffcloud. 

 Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: Clearing orphaned inode 136445 (uid=0, gid=0, mode=0100664, size=11567160) 
 Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: Clearing orphaned inode 136045 (uid=0, gid=0, mode=0100664, size=9253600) 
 [...] 
 Dec 13 13:18:38 Toushirou systemd-fsck[810]: /dev/mapper/main-var: clean, 43941/305216 files, 677459/1220608 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-var. 

 Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: Clearing orphaned inode 20 (uid=0, gid=0, mode=0100666, size=0) 
 Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: Clearing orphaned inode 50 (uid=128, gid=136, mode=0100600, size=0) 
 [...] 
 Dec 13 13:18:38 Toushirou systemd-fsck[811]: /dev/mapper/main-tmp: clean, 3380/121920 files, 20791/487424 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-tmp. 

 Dec 13 13:18:38 Toushirou systemd-fsck[814]: /dev/mapper/main-vcs: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[814]: /dev/mapper/main-vcs: clean, 62639/183264 files, 334140/732160 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-vcs. 

 Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: Clearing orphaned inode 1314229 (uid=5111, gid=5111, mode=0100600, size=2543956) 
 [...] 
 Dec 13 13:18:38 Toushirou systemd-fsck[817]: /dev/mapper/main-vmail: clean, 38189/1966080 files, 3862291/7864320 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-vmail. 

 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/extra-lxd. 

 Dec 13 13:18:38 Toushirou systemd-fsck[827]: /dev/mapper/extra-home: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[827]: /dev/mapper/extra-home: clean, 576437/19660800 files, 60022856/78643200 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/extra-home. 

 Dec 13 13:18:38 Toushirou systemd-fsck[791]: /dev/md0: 348/64000 files (23.9% non-contiguous), 63264/255936 blocks 

 Dec 13 13:18:38 Toushirou systemd-fsck[819]: /dev/mapper/main-www: recovering journal 
 Dec 13 13:18:38 Toushirou systemd-fsck[819]: /dev/mapper/main-www: clean, 417149/9175040 files, 7579187/36700160 blocks 
 Dec 13 13:18:38 Toushirou systemd[1]: Finished File System Check on /dev/mapper/main-www. 
 </pre> 

 Thanks to GuiHome and Victor for letting me know that the NextCloud service was unavailable. 

 Once the server has been restarted there was an error with the hivane network link. Hence some service were unavailable. The nerim link worked.  
 <pre> 
 root@Toushirou:~# systemctl --failed 
   UNIT                                LOAD     ACTIVE SUB      DESCRIPTION 
 ● apache2.service                     loaded failed failed The Apache HTTP Server 
 ● ifup@eth\x2dwan\x2dhivane.service loaded failed failed ifup for eth-wan-hivane 
 ● matrix-appservice-irc.service       loaded failed failed Matrix AppService IRC 
 ● networking.service                  loaded failed failed Raise network interfaces 
 </pre> 

 <pre> 
 root@Toushirou:~# ifdown --force eth-wan-hivane 
 RTNETLINK answers: Cannot assign requested address 
 RTNETLINK answers: Cannot assign requested address 
 root@Toushirou:~# ifup --force eth-wan-hivane 
 Waiting for DAD... Timed out 
 ifup: failed to bring up eth-wan-hivane 
 </pre> 

 I remember the timed out issue occurred when the last time the server was moved from a rack to another. I tried the @ifdown@/@ifup@ commands several times (until the @Timed out@ disappeared). 

 The logs show that the timed out issue occurred at boot: 
 <pre> 
 Dec 13 13:18:45 Toushirou sh[1562]: Waiting for DAD... Timed out 
 Dec 13 13:18:45 Toushirou sh[1496]: ifup: failed to bring up eth-wan-hivane 
 </pre> 

 Next I restarted @apache2.service@ and @matrix-appservice-irc.service@, then I updated @/lib/systemd/system/lxd.socket@ in order to fix a typo: 
 <pre>Dec 13 15:48:22 Toushirou systemd[1]: /lib/systemd/system/lxd.socket:8: Unit must be of type service, ignoring: lxd.servcie 
 </pre> 
 After that i ran @systemctl daemon-reload@ and @lxc list@ then the redmine LXC container restarted. 

 At this time I tried to create this issue using redmine:https://projects.duckcorp.org/ but an issue occurred after i tried to authenticate: the redmine web interface showed an error: @"Cannot assign requested address - connect(2) for [2001:67c:1740:9001::c1c8:2ab1]:636"@. 

 The restart of the @slapd@ service (which was listening on IPv6 but not IPv4) fixed this issue.

Back