Revision history [back]

click to hide/show revision 1
initial version

Perhaps its the load balancer. How do you load balance the requests?
I can make a __guess__ as follows:
When you try to execute a command, the credentials are sent to controller1, it generates a token and gives back to you. You use this token to execute a command, but this time the request is sent to controller2, who thinks the token is invalid (as it was generated by controller1). On the second attempt, controller2 generates a token, but it is sent to controller1 (who rejects it). But now, your subsequent commands succeed because you have tokens from both controllers (and I think haproxy somehow keeps track of connections) and they are sent to the correct nodes. Again you face the problem when the token expires.

One thing to prevent would be to synchronize tokens depending on the backend used. Of course, the sync should be fast enough so that both the nodes have the token before the request is sent. Or maybe, you can place the service in active/passive mode? However, you might face the same issue when the active node fails.