Project

General

Profile

README

Introduction

We strive to modernize, and make rebuilding of the services reliable and reproducible. After quite some time to deliberate Ansible was chosen to maintain our infrastructure.

Requirements

These playbooks are made to run with Ansible >=2.7, <2.8, preferably with Python 3.

You might need this for our custom plugins:

  • python3-unidecode

You should love YAML, as the rules and most of the configuration are written in this format. This is not difficult to learn though.

Configuration

A ansible.cfg configuration file is provided with all needed settings and to be sure we all use the same and achieve the same results. The only controversial setting is hash_behaviour = merge, it is very practical to allow partial variable overrides using groups of hosts (if you disagree, try to convince Duck, good luck).

The lists of hosts and groups (the inventory) is held by hosts.yml. All hosts should be listed in the all group, and then in various groups as appropriate; failing to do so will most probably result in dropping hosts out of the inventory when removing them from groups. We do not maintain variables in this file, this is not very practical.

The files in host_vars/<host>/ and group_vars/<group>/ for hosts and groups hold specific variables. We use groups for hosts holding the save service, or in the same geographic zone… That's were the magical hash_behaviour = merge shines.

Most of the infrastructure parameters are common information stored in group_vars/all: package repositories, DNS settings, users, entities…

Some files are also stored in data:

  • DuckCorp-specific files which are used by a role but not included in it to keep it generic
  • isolated tasks in a play manipulating data (copy/template/…); creating a higher level role may be a cleaner solution though

Playbooks

Here is a list of the playbooks and their goal:

  • dc_check: run various tests on machines to check for problems; this is WIP and currently only checks for unapplied upgrades, package diff and obsolete packages
  • dc: (partially, WIP) deploy the DuckCorp infrastructure
  • regen_ldap_content: generate the whole LDAP database content
  • regen_ssh_keys: on shell boxes, create missing user home directories and add their SSH keys to their authorized_keys file (preserving local changes)

Please don't use roles sections in plays, use include_role or include_role tasks instead, it is more powerful and you can order it as you wish (as any task).

Roles

We should try our best to modularize the rules to ease readability and maintenance.

Except for basic system settings (dc-base role) which should be kept small, and DuckCorp-specific needs, all roles are maintained in separate repositories (WIP for legacy roles) and should be kept generic (without trying to address each and every possible need in the world), or clearly state their scope limitation. They should all be documented and bear meta information.

Roles should present a clear API. We also use more and more multiple entrypoints using the include_role action and tasks_from parameter. This allows to factorize various functions based on the same logic and variables.

Limiting Scope

It is possible to limit the scope of the run to preview on a particular machine or shorten the run when you're sure changes affect only specific host or tasks.

To limit the scope of the machines, use the -l option.

To run only specific groups of tasks, use --tags and --skip-tags. A partial (due to dynamic includes) but sufficient list of tags can be found using:

ansible-playbook --list-tags playbooks/*.yml

Tests

Validation

The provided validate_playbooks.sh script should be run before every commit to avoid mistakes.

This script depends on:

  • flake8
  • ansible-lint

When we switch to a newer Ansible version, dependencies like ansible-lint which are tied to Ansible should be upgraded too. There might be changes in the reports after migration and we should strive to fix them quickly. A new commit is not responsible for the previous state of the rules and should not mix topic changes with unrelated fixes, they should be handled in separate commits.

Simulation

The --check option is available but there is no effort yet to make problematic tasks (command/shell/…) handled better.

Improving Ansible Speed

Ansible is slow, but there's a nice project to improve its performance. It still has glitches so it's not enabled by default, but it's easy to enable it.

First install the library (it is not yet packaged):

pip install mitogen

Then you just need to run playbooks this way:

ANSIBLE_STRATEGY=mitogen_linear ansible-playbook …

Statistics
| Branch: | Revision:

duckcorp-infra / ansible @ 802177b7

# Date Author Comment
802177b7 2018-12-30 14:44 Marc Dequènes

Orfeo: needs accounts resolution

It was removed in e8b3717 because Orfeo does not have historical VIP
shell accounts anymore, but certain services needs it (like
cyborghood_postman).

aad53888 2018-12-30 13:41 Marc Dequènes

dns: adjust rate limiting

f498ead5 2018-12-30 10:33 Marc Dequènes

Pond: experimenting with Bind DNSSEC support

55ee712d 2018-12-30 10:32 Marc Dequènes

Pond: update list of reverse zones

a7da91d8 2018-12-29 10:18 Marc Dequènes

dns: add rate limiting

6e731173 2018-12-29 08:31 Marc Dequènes

added duckcorp/www.milkypond.org

9ab70ad9 2018-12-29 07:59 Marc Dequènes

dc-postfix: rework TLS security

  • enforce server cipher order
  • be more restrictive with mandatory secured connections
  • smtpd?_tls_session_cache_database is not needed anymore, RFC 5077 TLS
    session tickets is recommended instead
  • share TLS settings among server types
7dcda2d6 2018-12-28 16:35 Marc Dequènes (Duck)

www.duckcorp.org: webstats are no more

Removed Piwik/Matomo from the CSP.

0cf9ae03 2018-12-28 16:27 Marc Dequènes (Duck)

www.duckcorp.org: forgot alias to TLD

e8b37173 2018-12-28 07:35 Marc Dequènes (Duck)

Orfeo: not a shell server anymore

f968f7df 2018-12-27 07:04 Marc Dequènes

unlock_host_encryption: failed when facts caching is expired

`system.boot.options` could not be defined because `_ip` depends on
facts, which prevented using any of the `system.boot.*` variables. So
moved initramfs SSH IP option in a separate top-level variable.

112ade40 2018-12-27 07:03 Marc Dequènes

unlock_host_encryption: do not log passphrases

84711404 2018-12-19 08:17 Marc Dequènes

static.perso.duckcorp.org: ensure 'proxy_wstunnel' Apache module is loaded

df6e330d 2018-12-19 08:08 Marc Dequènes

static.perso.duckcorp.org: no need for TLS to reach weechat on localhost

A recent update changed the behavior and the weechat certificate was not
accepted anymore. Weechat can bind on localhost only, and the port is
not opened anyway, so we do not need TLS between the proxy and Weechat.

75a83248 2018-12-19 08:00 Marc Dequènes

static.perso.duckcorp.org: weechat port was hardcoded

f7f0da45 2018-12-14 02:47 Marc Dequènes

dc-web: remove manually managed vhost confdir

b18c4706 2018-12-14 02:47 Marc Dequènes

added duckcorp/lists.duckcorp.org (web only)

c1c28bf2 2018-12-14 02:33 Marc Dequènes

added duckland/calibre.hq.duckcorp.org (web only)

3c210693 2018-12-14 02:13 Marc Dequènes

www.hq.duckcorp.org: fix DN for 'dc-duckland' group

cbeb6a63 2018-12-14 02:09 Marc Dequènes

added duckcorp/photos-ng.duckcorp.org (web only)

c8355c25 2018-12-13 14:03 Marc Dequènes

added duckcorp/wiki.duckcorp.org (web only)

cd87ba9a 2018-12-13 12:50 Marc Dequènes

www.duckcorp.org: prune useless files

77ff0dab 2018-12-13 12:11 Marc Dequènes

dc-web: install indexoverride data and config

50d2c386 2018-12-13 11:51 Marc Dequènes

dc-web: install shared data

3ea6b94f 2018-12-13 11:04 Marc Dequènes

added DL/www.hq.duckcorp.org

c6d062d6 2018-12-13 10:42 Marc Dequènes

added DL/printer.hq.duckcorp.org

5b710d38 2018-12-13 10:10 Marc Dequènes

radio.duckcorp.org: fix ProxyPassReverse

01e5f906 2018-12-13 09:57 Marc Dequènes

dc-icecast: add ansible_managed tags

eea73e50 2018-12-13 09:53 Marc Dequènes

dc-ftp: fix mode

4f25165c 2018-12-13 09:52 Marc Dequènes

added radio

41dda944 2018-12-13 09:03 Marc Dequènes

dc-ftp: add ansible_managed tags

af35a124 2018-12-13 08:58 Marc Dequènes

added FTP installation

70ea0c6c 2018-12-11 15:12 Marc Dequènes

dc-dovecot: replace obsolete 'antispam' plugin with 'imapsieve'

- update dc-spoolinger config as 'imapsieve' does not normalize line
endings like 'antispam' did
- also fix global SIEVE scripts not properly recompiled

fixes #630

520b44df 2018-12-07 02:49 Marc Dequènes

add playbook to unlock encrypted filesystems

2194c5f6 2018-12-06 19:11 Marc Dequènes

Elwing: enable initramfs_ssh and enable decrypting Elwing_data volume

cdcd99e0 2018-12-06 18:52 Marc Dequènes

dc-base: move 'initramfs_ssh' parameter in the tree

Also fix the default value btw.

d4a206f1 2018-12-06 18:46 Marc Dequènes

dc-base: take care of encrypted filesystems

66944da3 2018-12-06 18:19 Marc Dequènes

dc-base: update initramfs after generating dropbear's host keys

daf34d8f 2018-12-06 18:10 Marc Dequènes

add special vars to prepare Toushirou-NG

b51060f1 2018-12-06 18:10 Marc Dequènes

dc-base: reuse OpenSSH keys for dropbear-initramfs

394eced5 2018-12-06 16:32 Marc Dequènes

web: vhosts cleanup in accounts

9726d378 2018-12-06 09:58 Marc Dequènes

dc-dovecot: enable vacation-seconds SIEVE rules

5ee863e9 2018-12-06 09:57 Marc Dequènes

dc-dovecot: enable spamtest SIEVE rules

76c95ab8 2018-12-06 09:07 Marc Dequènes

webmail.duckcorp.org: replace unmaintained Roundcube 'sieverules' plugin by 'managesieve'

7badb0ca 2018-12-06 06:10 Marc Dequènes

dc-dovecot: workaround for Debian#915687

b14d3d60 2018-12-06 05:29 Marc Dequènes

dc-dovecot: enable IMAP METADATA

66f1a4bc 2018-12-06 04:41 Marc Dequènes

shizuka-STAR.duckcorp.org: not need for document root

5bded476 2018-12-05 16:22 Marc Dequènes

update submodules

67d85a02 2018-12-05 14:33 Marc Dequènes

stuff.milkypond.org: avoid duplicate headers

a67bcabf 2018-12-05 09:43 Marc Dequènes

update submodules

18ef5eab 2018-12-05 09:22 Marc Dequènes

update submodules

30980af1 2018-12-05 09:22 Marc Dequènes

dc-web: lowercase status page vhost name to avoid duplicates

218e55fb 2018-12-05 07:56 Marc Dequènes

dc-accounts: fix hosts resolution order, broke FQDN resolution with Python/Ansible

9c8196bf 2018-12-05 07:14 Marc Dequènes

switch the controller to Python 3

4de9bb8d 2018-12-05 07:13 Marc Dequènes

fix E502

e622a1c6 2018-12-04 19:13 Marc Dequènes

ignore more pedantic flake8 tests

64ca35b6 2018-12-04 19:12 Marc Dequènes

added duckcorp/www.duckcorp.org

c44433bb 2018-12-04 13:02 Marc Dequènes

fix E122

948b37ed 2018-12-04 13:01 Marc Dequènes

fix E265

10db18a8 2018-12-04 13:00 Marc Dequènes

vcs: prepare partition and directories

9b1801d6 2018-12-04 12:59 Marc Dequènes

fix ANSIBLE0011

5ad2d234 2018-12-04 12:58 Marc Dequènes

typo

16c6bf14 2018-12-04 12:45 Marc Dequènes

mail: create mail storage partition

9499819b 2018-12-04 12:28 Marc Dequènes

added duckcorp/vcs-git-viewer.duckcorp.org

06993aa2 2018-12-04 05:01 Marc Dequènes

add DICT server and client and duckcorp/dico.duckcorp.org web client

068e7350 2018-12-03 10:48 Marc Dequènes

dc-postfix: fix IPv6 quoting

52672286 2018-12-03 10:28 Marc Dequènes

dl/l2tp: fix tunnel auth

a58b7f15 2018-12-03 10:21 Marc Dequènes

dl/l2tp: remove 'noccp' option

1b91365b 2018-12-03 04:19 Marc Dequènes

redmine: move vaulted variables in a specific file, easier to maintain

18fece2a 2018-12-03 04:09 Marc Dequènes

projects.duckcorp.org: setup local repositories

a712a5d1 2018-12-01 09:29 Marc Dequènes

added duckcorp/stuff.milkypond.org

134acfc1 2018-11-26 18:53 Marc Dequènes

mail: more quota for gorou

089976a7 2018-11-26 03:08 Arnaud Fontaine

add email addresses for 'arnau' user account.

95250e4e 2018-11-20 18:04 Marc Dequènes

dc-rspamd: repository only supports amd64

697df182 2018-11-20 06:59 Marc Dequènes

update submodules

673d673d 2018-11-20 05:23 Marc Dequènes

dc-dovecot: global SIEVE include dir used ancient parameter name

This caused failure to compile the default SIEVE script.

7e369cbf 2018-11-19 19:05 Marc Dequènes

dc-dovecot: create 'vmail' user before using it

0f895c6a 2018-11-19 18:57 Marc Dequènes

mail: order fix, mda before antispam (user 'vmail' needed first)

ae2521f0 2018-11-19 18:56 Marc Dequènes

mail: make 'vmail' user dynamic

5ff4f640 2018-11-19 18:45 Marc Dequènes

mail: order fix, install mta before antispam

9a109588 2018-11-19 18:44 Marc Dequènes

mail: import_role merges params from two calls in the same play

cd87f7fb 2018-11-19 09:27 Marc Dequènes

gossip.duckcorp.org: fix service restarting

2df15926 2018-11-19 09:26 Marc Dequènes

gossip.duckcorp.org: git is needed

b5d08d1c 2018-11-19 09:26 Marc Dequènes

perso.duckcorp.org: enable 'userdir' module

de1464d6 2018-11-19 09:25 Marc Dequènes

dc-mediawiki: fix editing of Apache Mediawiki config

1c84088e 2018-11-19 07:20 Marc Dequènes

web: install Mediwiki instances' config

27890035 2018-11-19 07:20 Marc Dequènes

dc-mediawiki: Apache config for Mediawiki has moved, adaptation needed

de6e6b5d 2018-11-19 06:41 Marc Dequènes

dc-mediawiki: remove useless allowed path

4ff77069 2018-11-19 05:26 Marc Dequènes

update submodules

faea1ba4 2018-11-19 05:23 Marc Dequènes

dc-irconweb: enable proxy module

df7f66b5 2018-11-19 04:21 Marc Dequènes

dc-web: PHP setting depends on module being present

af4aea74 2018-11-19 02:19 Pierre-Louis Bonicoli

dc-base: allow to install and configure dropbear

fa8d9e61 2018-11-19 02:19 Pierre-Louis Bonicoli

dc-base: allow to set GRUB_CMDLINE_LINUX

cd1e6920 2018-11-19 01:55 Marc Dequènes

vaulted file not named properly

6698577c 2018-11-19 01:32 Arnaud Fontaine

add/remove email addresses for 'arnau' user account.

02344cbd 2018-11-18 09:44 Marc Dequènes

added duckcorp/webmail.duckcorp.org

e7091a7a 2018-11-03 05:28 Marc Dequènes (Duck)

supervision: enable db config local backup

61d0a329 2018-11-03 04:09 Marc Dequènes (Duck)

dc-dovecot: split SQL databases

fixes #631

ae87a1a7 2018-11-03 04:06 Arnaud Fontaine

add/remove email addresses for 'arnau' user account.

356ead29 2018-11-02 06:20 Marc Dequènes (Duck)

updated required Ansible version