Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

ldap authentication on Liberty only working partially

please offer some advice for what/where to look next. (this is on Liberty)

I got ldap authentication working well on our Newton environment https://ask.openstack.org/en/question/107448/password-authentication-thru-ldap/

now when I reproduce the same exact settings on our Liberty environment I see it only partially work. once everything is configured I use "openstack user list --domain mydom" to see my ldap users and to indicate my configuration is pretty much correct.

when I run that command 2 times it returns nothing. The 3rd time it returns the users. So every 3 times I get the users.

the strange thing is when the command is successful admin.log will show "WARNING keystone.common.wsgi Could not find domain: mydom" !!! when the command isn't successful the logging doesn't show anything.

using the ldap tool ldapsearch I can get info from the ldap server every single time

The kicker is I think something is wrong with keystone logging, I can't figure out what but keystone.log is months old and the only files updating is admin.log and main.log

thanks!

ldap authentication on Liberty only working partially

please offer some advice for what/where to look next. (this is on Liberty)

I got ldap authentication working well on our Newton environment https://ask.openstack.org/en/question/107448/password-authentication-thru-ldap/

now when I reproduce the same exact settings on our Liberty environment I see it only partially work. once everything is configured I use "openstack user list --domain mydom" to see my ldap users and to indicate my configuration is pretty much correct.

when I run that command 2 times it returns nothing. The 3rd time it returns the users. So every 3 times I get the users.

the strange thing is when the command is successful admin.log will show "WARNING keystone.common.wsgi Could not find domain: mydom" !!! when the command isn't successful the logging doesn't show anything.

using the ldap tool ldapsearch I can get info from the ldap server every single time

The kicker is I think something is wrong with keystone logging, I can't figure out what but keystone.log is months old and the only files updating is admin.log and main.log

thanks!

UPDATE:

here are log snippets from running the command 3 different times. It looks to me like the same command line is calling 3 different commands under the hood!!? any ideas how come?

openstack user list --domain dialogic (1st time: see Authorizing identity:list_domains())

2017-07-28 13:31:57.748 100836 DEBUG dogpile.core.dogpile [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:31:57.751 100836 DEBUG keystone.middleware.core [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=wrlWmmRlRmKk2xkN1p1KfA, audit_chain_id=wrlWmmRlRmKk2xkN1p1KfA) at 0x7f04b5c60d08>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:314
2017-07-28 13:31:57.754 100836 INFO keystone.common.wsgi [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] GET http://192.168.0.2:35357/v3/domains?name=mydom
2017-07-28 13:31:57.754 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: Authorizing identity:list_domains() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:62
2017-07-28 13:31:57.754 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:67
2017-07-28 13:31:57.754 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: Adding query filter params (name=mydom) wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:194
2017-07-28 13:31:57.754 100836 DEBUG keystone.policy.backends.rules [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] enforce identity:list_domains: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=wrlWmmRlRmKk2xkN1p1KfA, audit_chain_id=wrlWmmRlRmKk2xkN1p1KfA) at 0x7f04b5c60d08>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2017-07-28 13:31:57.755 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: Authorization granted wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:204
2017-07-28 13:32:03.985 100835 DEBUG keystone.middleware.core [req-5b71c115-23c6-400b-8723-397d2ec9f2a1 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:310

openstack user list --domain dialogic (2nd time: see Authorizing identity:get_domain(domain_id=mydom)

2017-07-28 13:33:31.528 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:33:31.530 100835 DEBUG keystone.middleware.core [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=ovBNYDXaQCS1X-Ep8G3B8g, audit_chain_id=ovBNYDXaQCS1X-Ep8G3B8g) at 0x7f04b5ba6868>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:314
2017-07-28 13:33:31.532 100835 INFO keystone.common.wsgi [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] GET http://192.168.0.2:35357/v3/domains/mydom
2017-07-28 13:33:31.533 100835 DEBUG keystone.common.controller [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] RBAC: Authorizing identity:get_domain(domain_id=mydom) _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:62
2017-07-28 13:33:31.533 100835 DEBUG keystone.common.controller [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:67
2017-07-28 13:33:31.534 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2017-07-28 13:33:31.534 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2017-07-28 13:33:31.534 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f04b5c2ebd0> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2017-07-28 13:33:31.535 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2017-07-28 13:33:31.539 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:33:31.539 100835 WARNING keystone.common.wsgi [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Could not find domain: mydom
2017-07-28 13:33:33.703 100836 DEBUG keystone.middleware.core [req-5dbb75b7-3938-4c9f-8de3-c8e9e2ba1a0e - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:310

openstack user list --domain dialogic (3rd time, which works, see Authorizing identity:list_users())

2017-07-28 13:39:00.373 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2017-07-28 13:39:00.373 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2017-07-28 13:39:00.373 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f04b5bb3750> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2017-07-28 13:39:00.374 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2017-07-28 13:39:00.375 100832 INFO keystone.token.providers.fernet.utils [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys
2017-07-28 13:39:00.404 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:39:00.406 100832 DEBUG keystone.middleware.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=AJdHHcdeRSiWyrF0Q4vfLQ, audit_chain_id=AJdHHcdeRSiWyrF0Q4vfLQ) at 0x7f04bee29be0>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:314
2017-07-28 13:39:00.408 100832 INFO keystone.common.wsgi [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] GET http://192.168.0.2:35357/v3/users?domain_id=5974fb34ae0a4c7089d9579a92ba1a48
2017-07-28 13:39:00.408 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: Authorizing identity:list_users() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:62
2017-07-28 13:39:00.408 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:67
2017-07-28 13:39:00.409 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: Adding query filter params (domain_id=5974fb34ae0a4c7089d9579a92ba1a48) wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:194
2017-07-28 13:39:00.409 100832 DEBUG keystone.policy.backends.rules [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] enforce identity:list_users: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=AJdHHcdeRSiWyrF0Q4vfLQ, audit_chain_id=AJdHHcdeRSiWyrF0Q4vfLQ) at 0x7f04bee29be0>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2017-07-28 13:39:00.409 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: Authorization granted wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:204
2017-07-28 13:39:00.410 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP init: url=ldap://10.10.10.2 _common_ldap_initialization /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:576
2017-07-28 13:39:00.410 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP init: use_tls=False tls_cacertfile=None tls_cacertdir=None tls_req_cert=2 tls_avail=1 _common_ldap_initialization /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:580
2017-07-28 13:39:00.410 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP bind: who=ldapadmin@mydom.com simple_bind_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:899
2017-07-28 13:39:00.412 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP search: base=ou=Corp,dc=mydom,dc=com scope=2 filterstr=(&(memberOf=CN=Openstack Users,OU=Groups,DC=mydom,DC=com)(objectClass=person)(cn=*)) attrs=['', 'cn', 'enabled', 'mail', 'sAMAccountName'] attrsonly=0 search_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:934
2017-07-28 13:39:00.415 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP unbind unbind_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:907
2017-07-28 13:39:00.416 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.416 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.417 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: Eric Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.420 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: a126e02f8906240d2bab25254ce3624f1c5d7db41c910b43a2904c54c3d6341e _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:00.421 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.421 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: James Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.423 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: cb0ca1a026dfae79beda8b944313dd07c9179e84add1d2765a0b86501e383637 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:00.426 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.426 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: Stephen Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.429 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: cac1a57549b3622977d3e29f5c418293768a8d71dd93395a95a007bf55a54cb1 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:00.429 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.429 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: Timothy Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.431 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: b1aee9ac4e26e8418bb3197f28b2488ab847ccb8aa713576d975e4c1fb9fa98c _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:04.647 100831 DEBUG keystone.middleware.core [req-3c34a983-959a-4ff9-8767-41b640b34f00 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:310

ldap authentication on Liberty only working partially

please offer some advice for what/where to look next. (this is on Liberty)

I got ldap authentication working well on our Newton environment https://ask.openstack.org/en/question/107448/password-authentication-thru-ldap/

now when I reproduce the same exact settings on our Liberty environment I see it only partially work. once everything is configured I use "openstack user list --domain mydom" to see my ldap users and to indicate my configuration is pretty much correct.

when I run that command 2 times it returns nothing. The 3rd time it returns the users. So every 3 times I get the users.

the strange thing is when the command is successful admin.log will show "WARNING keystone.common.wsgi Could not find domain: mydom" !!! when the command isn't successful the logging doesn't show anything.

using the ldap tool ldapsearch I can get info from the ldap server every single time

The kicker is I think something is wrong with keystone logging, I can't figure out what but keystone.log is months old and the only files updating is admin.log and main.log

thanks!

UPDATE:

here are log snippets from running the command 3 different times. It looks to me like the same command line is calling 3 different commands under the hood!!? any ideas how come?

openstack user list --domain dialogic mydom (1st time: see Authorizing identity:list_domains())

2017-07-28 13:31:57.748 100836 DEBUG dogpile.core.dogpile [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:31:57.751 100836 DEBUG keystone.middleware.core [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=wrlWmmRlRmKk2xkN1p1KfA, audit_chain_id=wrlWmmRlRmKk2xkN1p1KfA) at 0x7f04b5c60d08>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:314
2017-07-28 13:31:57.754 100836 INFO keystone.common.wsgi [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] GET http://192.168.0.2:35357/v3/domains?name=mydom
2017-07-28 13:31:57.754 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: Authorizing identity:list_domains() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:62
2017-07-28 13:31:57.754 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:67
2017-07-28 13:31:57.754 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: Adding query filter params (name=mydom) wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:194
2017-07-28 13:31:57.754 100836 DEBUG keystone.policy.backends.rules [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] enforce identity:list_domains: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=wrlWmmRlRmKk2xkN1p1KfA, audit_chain_id=wrlWmmRlRmKk2xkN1p1KfA) at 0x7f04b5c60d08>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2017-07-28 13:31:57.755 100836 DEBUG keystone.common.controller [req-c650ebca-a84a-4143-8547-aa91e05b67af - - - - -] RBAC: Authorization granted wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:204
2017-07-28 13:32:03.985 100835 DEBUG keystone.middleware.core [req-5b71c115-23c6-400b-8723-397d2ec9f2a1 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:310

openstack user list --domain dialogic mydom (2nd time: see Authorizing identity:get_domain(domain_id=mydom)

2017-07-28 13:33:31.528 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:33:31.530 100835 DEBUG keystone.middleware.core [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=ovBNYDXaQCS1X-Ep8G3B8g, audit_chain_id=ovBNYDXaQCS1X-Ep8G3B8g) at 0x7f04b5ba6868>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:314
2017-07-28 13:33:31.532 100835 INFO keystone.common.wsgi [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] GET http://192.168.0.2:35357/v3/domains/mydom
2017-07-28 13:33:31.533 100835 DEBUG keystone.common.controller [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] RBAC: Authorizing identity:get_domain(domain_id=mydom) _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:62
2017-07-28 13:33:31.533 100835 DEBUG keystone.common.controller [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:67
2017-07-28 13:33:31.534 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2017-07-28 13:33:31.534 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2017-07-28 13:33:31.534 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f04b5c2ebd0> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2017-07-28 13:33:31.535 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2017-07-28 13:33:31.539 100835 DEBUG dogpile.core.dogpile [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:33:31.539 100835 WARNING keystone.common.wsgi [req-85140726-90b0-4d97-a00a-0623f81aec4f - - - - -] Could not find domain: mydom
2017-07-28 13:33:33.703 100836 DEBUG keystone.middleware.core [req-5dbb75b7-3938-4c9f-8de3-c8e9e2ba1a0e - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:310

openstack user list --domain dialogic mydom (3rd time, which works, see Authorizing identity:list_users())identity:list_users())

2017-07-28 13:39:00.373 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] NeedRegenerationException _enter /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:94
2017-07-28 13:39:00.373 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] no value, waiting for create lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:127
2017-07-28 13:39:00.373 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] value creation lock <dogpile.cache.region._LockWrapper object at 0x7f04b5bb3750> acquired _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:131
2017-07-28 13:39:00.374 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Calling creation function _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:148
2017-07-28 13:39:00.375 100832 INFO keystone.token.providers.fernet.utils [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Loaded 2 encryption keys (max_active_keys=3) from: /etc/keystone/fernet-keys
2017-07-28 13:39:00.404 100832 DEBUG dogpile.core.dogpile [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Released creation lock _enter_create /usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py:154
2017-07-28 13:39:00.406 100832 DEBUG keystone.middleware.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=AJdHHcdeRSiWyrF0Q4vfLQ, audit_chain_id=AJdHHcdeRSiWyrF0Q4vfLQ) at 0x7f04bee29be0>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:314
2017-07-28 13:39:00.408 100832 INFO keystone.common.wsgi [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] GET http://192.168.0.2:35357/v3/users?domain_id=5974fb34ae0a4c7089d9579a92ba1a48
2017-07-28 13:39:00.408 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: Authorizing identity:list_users() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:62
2017-07-28 13:39:00.408 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:67
2017-07-28 13:39:00.409 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: Adding query filter params (domain_id=5974fb34ae0a4c7089d9579a92ba1a48) wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:194
2017-07-28 13:39:00.409 100832 DEBUG keystone.policy.backends.rules [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] enforce identity:list_users: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'19d9fea8ad7744779ba2c81e1ad36e76', 'roles': [u'_member_', u'admin'], 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=AJdHHcdeRSiWyrF0Q4vfLQ, audit_chain_id=AJdHHcdeRSiWyrF0Q4vfLQ) at 0x7f04bee29be0>, 'project_id': u'fcff98f1c7a34a86be56c0a3d069c280', 'trust_id': None} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2017-07-28 13:39:00.409 100832 DEBUG keystone.common.controller [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] RBAC: Authorization granted wrapper /usr/lib/python2.7/dist-packages/keystone/common/controller.py:204
2017-07-28 13:39:00.410 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP init: url=ldap://10.10.10.2 _common_ldap_initialization /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:576
2017-07-28 13:39:00.410 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP init: use_tls=False tls_cacertfile=None tls_cacertdir=None tls_req_cert=2 tls_avail=1 _common_ldap_initialization /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:580
2017-07-28 13:39:00.410 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP bind: who=ldapadmin@mydom.com simple_bind_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:899
2017-07-28 13:39:00.412 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP search: base=ou=Corp,dc=mydom,dc=com scope=2 filterstr=(&(memberOf=CN=Openstack Users,OU=Groups,DC=mydom,DC=com)(objectClass=person)(cn=*)) attrs=['', 'cn', 'enabled', 'mail', 'sAMAccountName'] attrsonly=0 search_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:934
2017-07-28 13:39:00.415 100832 DEBUG keystone.common.ldap.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] LDAP unbind unbind_s /usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py:907
2017-07-28 13:39:00.416 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.416 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.417 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: Eric Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.420 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: a126e02f8906240d2bab25254ce3624f1c5d7db41c910b43a2904c54c3d6341e _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:00.421 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.421 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: James Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.423 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: cb0ca1a026dfae79beda8b944313dd07c9179e84add1d2765a0b86501e383637 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:00.426 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.426 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: Stephen Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.429 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: cac1a57549b3622977d3e29f5c418293768a8d71dd93395a95a007bf55a54cb1 _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:00.429 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] ID Mapping - Domain ID: 5974fb34ae0a4c7089d9579a92ba1a48, Default Driver: False, Domains: False, UUIDs: False, Compatible IDs: True _set_domain_id_and_mapping /usr/lib/python2.7/dist-packages/keystone/identity/core.py:587
2017-07-28 13:39:00.429 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Local ID: Timothy Bob _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:605
2017-07-28 13:39:00.431 100832 DEBUG keystone.identity.core [req-69ce1ac1-aac5-4cfe-bcf4-c768d4fee2b0 - - - - -] Found existing mapping to public ID: b1aee9ac4e26e8418bb3197f28b2488ab847ccb8aa713576d975e4c1fb9fa98c _set_domain_id_and_mapping_for_single_ref /usr/lib/python2.7/dist-packages/keystone/identity/core.py:618
2017-07-28 13:39:04.647 100831 DEBUG keystone.middleware.core [req-3c34a983-959a-4ff9-8767-41b640b34f00 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. process_request /usr/lib/python2.7/dist-packages/keystone/middleware/core.py:310

furthermore: one time i went into the actual openstack CLI and typed the command in the CLI: user list --domian mydom and it returned the users correctly 39 out of 40 times! Then I exited the CLI and went back in, and that command failed to return the users 40 out of 40 times!

what gives here guys?

thanks