Intermittent 502 proxy error to Zope / Plone “error reading status line from remote server”

Intermittent 502 proxy error to Zope / Plone “error reading status line from remote server”

We’re currently running Plone 3.3.6 on one instance of Zope with 4 threads behind Apache 2.2. We have a number of subsites all managed from one Plone site.
I’m currently testing a new set-up running 2 Zope instances, 2 threads each, and load balanced by Apache. We have a tool that spiders our sites to check for errors and this is reporting a number of 502 proxy errors. It runs fine on our live sites and usually also on our test sites.
As expected we see a 502 error in the Apache access log.
10.10.11.1 – – [04/Feb/2012:14:34:46 +0000] “GET /family-strengthening HTTP/1.1” 502 345 “-” “Mozilla”

We also get an entry in the Apache error log
[Sat Feb 04 14:34:46 2012] [error] [client 10.10.11.1] proxy: error reading status line from remote server dev:18082
[Sat Feb 04 14:34:46 2012] [error] [client 10.10.11.1] proxy: Error reading from remote server returned by /family-strengthening

Sometimes Zope access log shows the request with 200 response but with 0 bytes served
./instance2-Z2.log:10.10.11.4 – Anonymous [04/Feb/2012:14:34:47 +0100] “GET /VirtualHostBase/http/test.street-children.org.uk:80/sos/soschildrensvillages_org_uk/street-children_org_uk/VirtualHostRoot/family-strengthening HTTP/1.1” 200 0 “” “Mozilla”

Other times the request does not show up in the Zope access log.
There are no errors in the Zope instance logs.
When I request the same pages through a browser I never receive the error. Running our spider again results in a similar number of 502 errors but for different pages.
I think our Apache configuration for our sites is pretty standard:
ProxyPreserveHost On

BalancerMember http://dev:18081
BalancerMember http://dev:18082

RewriteRule ^/(.*) balancer://zopes/VirtualHostBase/http/%{SERVER_NAME}:80/sos/soschildrensvillages_org_uk/%{ENV:SOS_PLONE_FOLDER_SHORTNAME}/VirtualHostRoot/$1 [last,proxy]

Related:  Nginx upstream reconfiguration while one is processing request

I don’t know where to start trying to debug this so any help would be much appreciated.

Solutions/Answers:

Solution 1:

Update

I thought I’d resolved this by doing an Apache stop and then Apache start

apachectl stop
apachectl start

The problem went away on the test system only to appear on live once I switched to two Zopes there.

I had been restarting Apache gracefully. We had also been getting errors like the following in our apache error log every time we gracefully restarted

[Tue Feb 07 19:41:07 2012] [error] proxy: ap_get_scoreboard_lb(97) failed in child 411 for worker http://dev:18081
[Tue Feb 07 19:41:07 2012] [error] proxy: ap_get_scoreboard_lb(99) failed in child 411 for worker http://dev:18082

I had read that these were harmless but maybe not? They no longer appear on graceful restart after the stop / start

References