https://projects.duckcorp.org/https://projects.duckcorp.org/favicon.ico?16699090422021-11-02T12:09:02ZDuckCorp ProjectsDuckCorp Infrastructure - Bug #741: Redmine: "Incomplete response received from application"https://projects.duckcorp.org/issues/741?journal_id=22982021-11-02T12:09:02ZPierre-Louis Bonicolipierre-louis.bonicoli@ir5.eu
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/2298/diff?detail_id=2637">diff</a>)</li></ul> DuckCorp Infrastructure - Bug #741: Redmine: "Incomplete response received from application"https://projects.duckcorp.org/issues/741?journal_id=23022021-11-16T04:33:27ZMarc Dequènesduck@duckcorp.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>Pierre-Louis Bonicoli</i> to <i>Marc Dequènes</i></li></ul><p>I see these errors on the reverse proxy:<br /><pre>
AH01102: error reading status line from remote server 10.0.7.2:80
AH00898: Error reading from remote server returned by /projects/dc-ducklings-volunteer-activities/issues.atom
</pre></p>
<p>In the container I see those:<br /><pre>
[ 2021-11-16 05:13:05.9810 2586983/7f84aaffd700 age/Cor/Con/InternalUtils.cpp:112 ]: [Client 12-6864] Sending 502 response: application did not send a complete response
App 2831951 stdout:
[ 2021-11-16 05:13:06.9367 2586983/7f8500a1c700 age/Cor/Con/InternalUtils.cpp:112 ]: [Client 2-6865] Sending 502 response: application did not send a complete response
[ 2021-11-16 05:13:07.2635 2586983/7f84eb7fe700 age/Cor/CoreMain.cpp:819 ]: Checking whether to disconnect long-running connections for process 2831931, application redmine_md
</pre></p>
<p>And those too:<br /><pre>
[ 2021-11-16 01:06:24.8866 2586983/7f84e82b4700 age/Cor/Spa/SmartSpawner.h:740 ]: The application preloader seems to have crashed, restarting it and trying again...
</pre></p>
<p>I just found out the source of the problem:<br /><pre>
[Tue Nov 16 06:17:13 2021] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=lxc.payload.redmine,mems_allowed=0-1,oom_memcg=/lxc.payload.redmine,task_memcg=/lxc.payload.redmine/system.slice/apache2.service,task=ruby,pid=2171217,uid=1065569
[Tue Nov 16 06:17:13 2021] Memory cgroup out of memory: Killed process 2171217 (ruby) total-vm:414356kB, anon-rss:129448kB, file-rss:8932kB, shmem-rss:0kB, UID:1065569 pgtables:444kB oom_score_adj:0
[Tue Nov 16 06:17:13 2021] oom_reaper: reaped process 2171217 (ruby), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
</pre></p>
<p>I increased the memory limit for the container (the default stays at 512MB).</p>
<p>Current usage (after increase):<br /><pre>
# lxc info redmine
…
Memory usage:
Memory (current): 620.11MiB
…
</pre><br />But it was limited to 512MB. Now we have a better understanding of how (too) much memory is being used by this service ;-P.</p> DuckCorp Infrastructure - Bug #741: Redmine: "Incomplete response received from application"https://projects.duckcorp.org/issues/741?journal_id=23032021-11-16T04:34:18ZMarc Dequènesduck@duckcorp.org
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="redmine: increase memory limit fixes #741" href="https://projects.duckcorp.org/projects/dc-admin/repository/duckcorp-infra/revisions/5e72c249b020aeb0dc20f03a59390f0a1ff2a573">duckcorp-infra|5e72c249b020aeb0dc20f03a59390f0a1ff2a573</a>.</p>