Did you create a healthmonitor and associate it with you pool? It is possible to observe slow responses when one of the pool members is down and no healthmonitor is associated with the pool. If the selected backend server is down, haproxy will retry N times before it the request is dispatched to a different backend server. When a healthmonitor is associated with the pool, no member that is down will be considered for the request.

It definitely too slow too be true, even by Quantum standards!

This kind of timeout makes me suspect your instances or haproxy are trying to perform some kind of dns resolution before sending the response. You might want to try with tcpdump on the server side.