DuckCorp » DuckCorp Infrastructure

With exporters gaining TLS support there is no obvious major problem left and we can do some testing.

I've started a new playbook and role to experiment and so far it is working well.

Some though, in no order:

• zabbix: hard to configure all in the slow UI
• zabbix: certain features are slow to come (#495, native systemd support, LLD web checks…)
• prometheus_ansible_role: no service autodetection anymore, I found very easy to map "features" to inventory groups or variables; it's now easy to manually disable or force-enable if needed
• prometheus: nice feature coming to help split the config, but in the meanwhile I might be able to use file_sd_configs and avoid passing inventory vars directly into the role to work around the problem
• grafana: I would have preferred if grafana was packaged in Debian but in the end it's very handy to make use of their dashboard libraries and avoid spending hours and hours designing every little graph
• prometheus: using textfiles collector can be an alternative to the lack of exporter or when it's not packaged (used for NTP/chrony)

What we have so far:

• node basic and all the hardware goodies, temperature etc seem to be there too
• poller stats
• Bind
• Postfix
• Apache
• PG
• LXD
• blackbox with checks of almost all public services endpoints, with TLS and protocol checks when possible too
• Prosody but no grafana dashboard and the amount of stats are limited; there are additional modules called measure_* to complement but they are not packaged
• MySQL
• NTP
• Nextcloud

I was able to setup several exporters and borrow various alerts from https://awesome-prometheus-alerts.grep.to/ but even if we have more than before in certain areas I'd like to check if we're missing something important (compared to our Zabbix installation):

• time sync is checked but NTPd stats are missing; there is an exporter but it is not packaged
• no maps, but if that was cute that was also utterly useless
• ProFTPd, but I'm not sure it's worth it now
• SNMP checks for my internal switches, more out of curiosity
• SNMP checks for my printer, but I don't use it very often so it's not critical
• OpenLDAP stats, more out of curiosity
• MDA, this is important
• MySQL, also important
• alerts via mail, IRC and XMPP

What I plan to look at:

• [WIP] make the role generic and split it form our main repo (and use it at OSCI)
• generation of alerter contacts and alert methods (Matrix, XMPP, Mail)
• blackbox, maybe replace smokeping? add check for certs, DNSSEC etc
• [WIP] grafana base config generation
• MySQL exporter
• SNMP for my internal switches
• could we make certain graphs public? (like pings etc?)
• Dovecot exporter, but not packaged in Debian
• Nextcloud exporter backported and bumped to 0.5.0 for token auth support
• Matrix alert hook, but not packaged in Debian
• node exporter maintainers do not want to add systemd service stats but there is a systemd exporter that would help get per-service resource consumption stats
• the IRC relay displays only limited info, no severity coloring, and sometimes disconnect and is unable to reconnect; NinjaBot seems to be a nice alternative
• SSH checks on non-standard port (currently Orthos and Nicecity checks only check the gateway…)