DuckCorp Projects: Issueshttps://projects.duckcorp.org/https://projects.duckcorp.org/favicon.ico?16699090422023-07-09T13:51:58ZDuckCorp Projects
Redmine DuckCorp Infrastructure - Bug #783 (In Progress): Move Services out of Orfeohttps://projects.duckcorp.org/issues/7832023-07-09T13:51:58ZMarc Dequènesduck@duckcorp.org
Orfeo's RAID ! has one disk down, so let's move certain services out of it for now:
<ul>
<li>✅ PostgreSQL database -> Toushirou</li>
<li>✅ webmail -> Toushirou</li>
<li>✅ mailing-lists -> Toushirou</li>
<li>✅ XMPP -> Jinta</li>
<li>🔳 IRC services</li>
<li>🔳 (maybe, or later if things gets bad) NS1 & DDNS -> Toushirou</li>
</ul> DuckCorp Infrastructure - Enhancement #743 (In Progress): Switching to Prometheus?https://projects.duckcorp.org/issues/7432021-11-17T11:56:09ZMarc Dequènesduck@duckcorp.org
<p>With exporters gaining TLS support there is no obvious major problem left and we can do some testing.</p>
<p>I've started a new playbook and role to experiment and so far it is working well.</p>
Some though, in no order:
<ul>
<li>zabbix: hard to configure all in the slow UI</li>
<li>zabbix: certain features are slow to come (<a class="issue tracker-2 status-6 priority-3 priority-lowest closed" title="Enhancement: Maybe use LLD Stacking graph script in forum (Rejected)" href="https://projects.duckcorp.org/issues/495">#495</a>, native systemd support, LLD web checks…)</li>
<li>prometheus_ansible_role: no service autodetection anymore, I found very easy to map "features" to inventory groups or variables; it's now easy to manually disable or force-enable if needed</li>
<li>prometheus: <a href="https://github.com/prometheus/prometheus/issues/8543" class="external">nice feature coming to help split the config</a>, but in the meanwhile I might be able to use <em>file_sd_configs</em> and avoid passing inventory vars directly into the role to work around the problem</li>
<li>grafana: I would have preferred if grafana was packaged in Debian but in the end it's very handy to make use of their dashboard libraries and avoid spending hours and hours designing every little graph</li>
<li>prometheus: using textfiles collector can be an alternative to the lack of exporter or when it's not packaged (used for NTP/chrony)</li>
</ul>
What we have so far:
<ul>
<li>node basic and all the hardware goodies, temperature etc seem to be there too</li>
<li>poller stats</li>
<li>Bind</li>
<li>Postfix</li>
<li>Apache</li>
<li>PG</li>
<li>LXD</li>
<li>blackbox with checks of almost all public services endpoints, with TLS and protocol checks when possible too</li>
<li>Prosody but no grafana dashboard and the amount of stats are limited; there are additional modules called measure_* to complement but they are not packaged</li>
<li>MySQL</li>
<li>NTP</li>
<li>Nextcloud</li>
</ul>
I was able to setup several exporters and borrow various alerts from <a class="external" href="https://awesome-prometheus-alerts.grep.to/">https://awesome-prometheus-alerts.grep.to/</a> but even if we have more than before in certain areas I'd like to check if we're missing something important (compared to our Zabbix installation):
<ul>
<li><del>time sync is checked but NTPd stats are missing; there is an exporter but it is not packaged</del></li>
<li>no maps, but if that was cute that was also utterly useless</li>
<li>ProFTPd, but I'm not sure it's worth it now</li>
<li>SNMP checks for my internal switches, more out of curiosity</li>
<li>SNMP checks for my printer, but I don't use it very often so it's not critical</li>
<li>OpenLDAP stats, more out of curiosity</li>
<li><del>MDA, this is important</del></li>
<li><del>MySQL, also important</del></li>
<li><del>alerts via mail, IRC and XMPP</del></li>
</ul>
What I plan to look at:
<ul>
<li>[WIP] make the role generic and split it form our main repo (and use it at OSCI)</li>
<li><del>generation of alerter contacts and alert methods (Matrix, XMPP, Mail)</del></li>
<li><del>blackbox, maybe replace smokeping? add check for certs, DNSSEC etc</del></li>
<li>[WIP] grafana base config generation</li>
<li><del>MySQL exporter</del></li>
<li>SNMP for my internal switches</li>
<li>could we make certain graphs public? (like pings etc?)</li>
<li><del><a href="https://github.com/kumina/dovecot_exporter" class="external">Dovecot exporter</a>, but not packaged in Debian</del></li>
<li><del><a href="https://github.com/xperimental/nextcloud-exporter" class="external">Nextcloud exporter</a></del> backported and bumped to 0.5.0 for token auth support</li>
<li><a href="https://github.com/jaywink/matrix-alertmanager" class="external">Matrix alert hook</a>, but not packaged in Debian</li>
<li><a href="https://github.com/prometheus/node_exporter/issues/1136" class="external">node exporter maintainers do not want to add systemd service stats</a> but there is a <a href="https://github.com/povilasv/systemd_exporter" class="external">systemd exporter</a> that would help get per-service resource consumption stats</li>
<li>the IRC relay displays only limited info, no severity coloring, and sometimes disconnect and is unable to reconnect; <a href="https://gitlab.crans.org/esum/NinjaBot" class="external">NinjaBot</a> seems to be a nice alternative</li>
<li>SSH checks on non-standard port (currently Orthos and Nicecity checks only check the gateway…)</li>
</ul> DuckCorp Infrastructure - Bug #692 (New): Elwing cannot be unlocked remotelyhttps://projects.duckcorp.org/issues/6922020-04-12T15:28:35ZMarc Dequènesduck@duckcorp.org
<p>The server is not listening on SSH port as expected.</p> DuckCorp Infrastructure - Tracking #677 (New): roundcube/twofactor_gauthenticator: bug in 2FA QR ...https://projects.duckcorp.org/issues/6772019-09-21T08:52:57ZMarc Dequènesduck@duckcorp.org
<p><a class="external" href="https://github.com/alexandregz/twofactor_gauthenticator/issues/99">https://github.com/alexandregz/twofactor_gauthenticator/issues/99</a></p>
<p>According to the BR it is possible to get the QR code simply by switching back to the <em>Larry</em> theme temporarily.</p> DuckCorp Infrastructure - Enhancement #675 (In Progress): Publish DANE/TLSA records for Let's Enc...https://projects.duckcorp.org/issues/6752019-09-20T17:42:05ZMarc Dequènesduck@duckcorp.orgDuckCorp Infrastructure - Enhancement #647 (In Progress): Patched softwares, how to handle them?https://projects.duckcorp.org/issues/6472019-04-21T09:54:52ZMarc Dequènesduck@duckcorp.org
The following files were patched:
<ul>
<li>Jinta:
<ul>
<li>/usr/lib/python3/dist-packages/wikitrans/wikimarkup.py: fix for dico</li>
</ul>
</li>
<li>Orfeo:
<ul>
<li>/usr/lib/ruby/vendor_ruby/mail/fields/content_type_field.rb: fix for cyborghood_postman</li>
<li>/var/lib/aspell/br.{compat,rws}: affected by <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690216" class="external">Debian#690216</a></li>
<li>/usr/lib/python3/dist-packages/postorius/templates/postorius/lists/held_messages.html: fix for held messages popup not visible</li>
</ul>
</li>
<li>Thorfinn:
<ul>
<li>/var/lib/aspell/br.{compat,rws}: affected by <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690216" class="external">Debian#690216</a></li>
</ul>
</li>
<li>Toushirou:
<ul>
<li>/usr/share/redmine/app/controllers/activities_controller.rb: missing API to get activities on the website</li>
<li>/usr/share/redmine/lib/redmine/themes.rb: per-project theme</li>
<li>/var/lib/aspell/br.{compat,rws}: affected by <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=690216" class="external">Debian#690216</a></li>
<li>/usr/share/redmine/app/helpers/application_helper.rb: <a class="issue tracker-6 status-3 priority-4 priority-default closed" title="Tracking: redmine: disable usage of non-free gravatar service (Resolved)" href="https://projects.duckcorp.org/issues/731">#731</a></li>
<li>/usr/share/redmine/lib/plugins/gravatar/lib/gravatar.rb: <a class="issue tracker-6 status-3 priority-4 priority-default closed" title="Tracking: redmine: disable usage of non-free gravatar service (Resolved)" href="https://projects.duckcorp.org/issues/731">#731</a></li>
</ul></li>
</ul> DuckCorp Infrastructure - Enhancement #602 (In Progress): Deploy Content Security Policy (CSP) an...https://projects.duckcorp.org/issues/6022017-09-30T07:02:53ZMarc Dequènesduck@duckcorp.org
<p>We should have a look at this: <a class="external" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP</a></p>
<p>Full specification: <a class="external" href="https://www.w3.org/TR/CSP/">https://www.w3.org/TR/CSP/</a></p> DuckCorp Infrastructure - Enhancement #564 (In Progress): Unused accounts spring cleanuphttps://projects.duckcorp.org/issues/5642017-06-24T03:51:16ZMarc Dequènesduck@duckcorp.org
<p>We probably have several unused accounts, so let's check usage, contact users, and adapt or close account as needed.</p>
<p>Sub-tasks are to track specific resources usage, which would allow for per-service cleanup and help account-level cleanup.</p>
<p>It would be nice if the per-service reports could be parsable in order to generate an aggregate report for admin.</p> DuckCorp Infrastructure - Bug #550 (In Progress): Internet is slow (and probably ever was)https://projects.duckcorp.org/issues/5502017-06-13T08:21:39ZMarc Dequènesduck@duckcorp.org
<pre>
# speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from OCN (114.158.197.252)...
Selecting best server based on latency...
Hosted by h3zjp (Nerima) [14.71 km]: 13.441 ms
Testing download speed........................................
Download: 247.74 Mbits/s
Testing upload speed..................................................
Upload: 123.57 Mbits/s
</pre><br /><pre>
# speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from OCN (114.158.197.252)...
Selecting best server based on latency...
Hosted by TB (Tokyo) [8.86 km]: 6.79 ms
Testing download speed........................................
Download: 204.83 Mbits/s
Testing upload speed..................................................
Upload: 183.24 Mbits/s
</pre>
<p>Google speedtest seems to confirm.</p> DuckCorp Infrastructure - Bug #531 (In Progress): Logcheck on Elwing seems unable to send mailhttps://projects.duckcorp.org/issues/5312017-05-02T15:31:47ZPierre-Louis Bonicolipierre-louis.bonicoli@ir5.eu
<p>On <code>Orfeo</code>, in <code>/var/log/mail.log</code><br /><pre>
May 1 12:55:34 orfeo postfix/smtpd[14064]: NOQUEUE: reject: RCPT from unknown[2400:4030:9f9b:e901::1]: 450 4.7.1 Client host rejected: cannot find your hostname, [2400:4030:9f9b:e901::1]; from=<daemon@hq.duckcorp.org> to=<root@hq.duckcorp.org> proto=ESMTP helo=<Elwing.hq.duckcorp.org>
[...]
May 1 12:55:34 orfeo postfix/smtpd[19450]: NOQUEUE: reject: RCPT from unknown[2400:4030:9f9b:e901::1]: 450 4.7.1 Client host rejected: cannot find your hostname, [2400:4030:9f9b:e901::1]; from=<duck@hq.duckcorp.org> to=<duck@hq.duckcorp.org> proto=ESMTP helo=<Elwing.hq.duckcorp.org>
</pre></p> DuckCorp Infrastructure - Review #519 (In Progress): Review burp rolehttps://projects.duckcorp.org/issues/5192017-04-03T12:00:00ZPierre-Louis Bonicolipierre-louis.bonicoli@ir5.eu
<p>The Burp role is available here: <a class="external" href="https://gitlab.com/pilou-/ansible-role-burp">https://gitlab.com/pilou-/ansible-role-burp</a>.</p> DuckCorp Infrastructure - Enhancement #515 (In Progress): Orfeo: increase log retentionhttps://projects.duckcorp.org/issues/5152017-03-02T11:35:12ZPierre-Louis Bonicolipierre-louis.bonicoli@ir5.eu
<pre>
Filesystem Size Used Avail Use% Mounted on
/dev/sdb7 2.7G 2.2G 588M 79% /var/log
</pre>
The current log retention delays are:
<ul>
<li>7 days
<ul>
<li><code>/var/log/syslog</code> <ins>223 Mo</ins></li>
</ul>
</li>
<li>14 days
<ul>
<li><code>/var/log/apache2</code> 11 Mo</li>
</ul>
</li>
<li>4 weeks
<ul>
<li><code>/var/log/mail.*</code> <strong><ins>1.5 Go</ins></strong></li>
<li><code>/var/log/auth.log*</code> 58 Mo</li>
<li><code>/var/log/user.log*</code> 248 Ko</li>
<li><code>/var/log/daemon.log*</code> 23 Mo</li>
<li><code>/var/log/fail2ban</code> 5 Mo</li>
</ul>
</li>
<li>7 months
<ul>
<li><code>/var/log/vmail_clamdscan.log*</code> 28 Ko</li>
</ul>
</li>
<li>1 year
<ul>
<li><code>/var/log/roundcube/</code> 1.6 Mo</li>
</ul></li>
</ul>
<ul>
<li>Not using logrotate:
<ul>
<li><code>/var/log/named</code> 204Mo</li>
</ul></li>
</ul>
Propositions:
<ul>
<li>either rotate <code>/var/log/mail.log</code> everyday or compress <code>/var/log/mail.log.1</code> (it would use ~120 Mo instead of 800 Mo)</li>
<li><code>/var/log/apache2</code>: increase log retention to 2 months</li>
<li>compress <code>/var/log/syslog.log.1</code></li>
</ul> DuckCorp Infrastructure - Bug #463 (In Progress): Replace our Ancient Galleryhttps://projects.duckcorp.org/issues/4632015-07-12T21:18:23ZMarc Dequènesduck@duckcorp.org
<p>Gallery2 is not supported anymore, it is old and ugly, probably with security issues. It also uses old libraries, probably having security issues too, like Smarty.</p>
<p>We need to find a proper replacement.</p>
<p>Requirements:<br /> - not too ugly and some kind of responsive JS slideshow<br /> - method to hook into LDAP for auth (direct LDAP support or PAM)<br /> - direct access to the media files in the filesystem<br /> - a webapp using a daemon for background tasks and inotify support would be nice<br /> - an Android app would be nice</p>
<p>Pyoto from Kilobug had some of these features but not all. Maybe we could work with him to improve it.</p> DuckCorp Infrastructure - Bug #242 (In Progress): Mails to unmanaged domains are delivered locall...https://projects.duckcorp.org/issues/2422011-08-25T07:53:39ZMarc Dequènesduck@duckcorp.org
<p>The problem has been spotted with the user <em>pilou</em>. He has an emailAccount only to use the SMTP relay. When a mail is sent to its primary address, which is not on a managed domain, it is not relayed outside but delivered locally.</p> LdapShadows - Enhancement #33 (New): Design and Rework the external APIhttps://projects.duckcorp.org/issues/332010-04-05T18:16:15ZMarc Dequènesduck@duckcorp.org
<p>Work with the CyborgHood project to design an usuable and well-abstracted API for their use, ti should be a good start.</p>