Ask Your Question
1

Remote debugging OpenStack Kilo "nova-scheduler" with Eclipse-Pydev ends in "ThreadSuspended" error messsage

asked 2015-07-11 02:43:53 -0600

holger-king gravatar image

updated 2015-07-13 06:32:29 -0600

Dear community,

my current development environment is:

  • Eclipse Luna, version 4.4.2
  • PyDev 4.2.0 (latest version)
  • RDO OpenStack Kilo (latest version)
  • Red Hat Enterprise Linux 7.1 (latest version)

when trying to remote debug a nova-scheduler related Python source file name "filter_scheduler.py" located in "/usr/lib/python2.7/site-packages/nova/scheduler".

I followed the following guidelines:

  • http://pydev.org/manual_adv_remote_debugger.html (http://pydev.org/manual_adv_remote_de...)
  • http://ilearnstack.com/2013/05/30/debug-openstack-code-local-remote-with-eclipse-and-pydev-plug-in/ (http://ilearnstack.com/2013/05/30/deb...)
  • http://debugopenstack.blogspot.de/2012/12/debugopenstack-code-local-remote-with.html (http://debugopenstack.blogspot.de/201...)
  • https://ask.openstack.org/en/question/815/how-do-i-debug-nova-service-with-eclipse-and-pydev/ (https://ask.openstack.org/en/question...)

So, I did the following steps:

  1. copied the Python source of "/usr/lib/python2.7/site-packages/nova/*" to my localhost and imported the Python sources in my PyDev Eclipse project
  2. copied the Eclipse "pydevd" Python debugger client implementation from "C:\Program Files (x86)\eclipse\dropins\plugins\org.python.pydev_4.0.0.201504132356" to the OpenStack remote host in "/tmp/pysrc"
  3. adapted the "/tmp/pysrc/pydevd_file_utils.py" file to match the source paths of the Pyhton code between the remote host and my local development environment (see PATHS_FROM_ECLIPSE_TO_PYTHON = [(r'c:\Workspace\PythonTest\src', r'/usr/lib/python2.7/site-packages')])
  4. modified "/usr/bin/nova-scheduler" to ensure enhancing the Python module search path for "pydevd" via "sys.path.append('/tmp/pysrc')" below "import sys"
  5. inserted the "import eventlet" and "eventlet.monkey_patch(all=False,socket=True,time=True,os=False,thread=False)" statements in "/usr/bin/nova-scheduler" to prevent getting "ThreadSuspended" error messages
  6. activated the breakpoint in "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py" by additionally importing "import pydevd" and inserting "pydevd.settrace('xxx.xxx.xxx.xxx', port=5678, stdoutToServer=True, stderrToServer=True,suspend=True)" where the debugging should start
  7. started the Eclipse PyDev debugger on standard port 5678
  8. restarted "nova-scheduler" via "systemctl restart nova-scheduler"
  9. launched a nova docker instance via "nova boot --image "docker.io/fedora:latest" --flavor m1.tiny --nic net-id=<nic_id> docker_test"
  10. checked the Eclipse PyDev debugger and always see an "Error reading ThreadSuspended" error message

Although, I inserted the famous "eventlet.monkey_patch" statement in "/usr/bin/nova-scheduler" I get this error message:

Unexpected XML error reading:<xml><thread id="pid22922_seq1" stop_reason="105" message="None"><frame id="31976272" name="allocate_lock" file="/usr/lib/python2.7/site-packages/eventlet/green/thread.py" line="78">"</frame><frame id="31416032" name="__init__" file="/usr/lib64/python2.7/threading.py" line="563">"</frame><frame id="31415280" name="Event" file="/usr/lib64/python2.7/threading.py" line="550">"</frame><frame id="31412496" name="__init__" file="/usr/lib64/python2.7/threading.py" line="690">"</frame><frame id="97266224" name="__init__" file="/tmp/pysrc42/pydevd_comm.py" line="250">"</frame><frame id="97358208" name="__init__" file="/tmp/pysrc42/pydevd.py" line="205">"</frame><frame id="97235984" name="_locked_settrace" file="/tmp/pysrc42/pydevd.py" line="1967">"</frame><frame id="97255072" name="settrace" file="/tmp/pysrc42/pydevd.py" line="1876">"</frame><frame id="97254496" name="select_destinations" file="/usr/lib/python2 ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-07-15 04:47:21 -0600

holger-king gravatar image

updated 2015-07-15 06:33:21 -0600

The PROBLEM could be SOLVED by starting "nova-compute" setting it in debug mode. Please follow these steps:

  1. copy "pydev" Python API from PyDev Eclipse plugin to remote host where OpenStack is running
  2. guarantee that the Python source code you want to debug is imported in your local Eclipse environment
  3. do guarantee that "pydevd_file_utils.py" is correctly setup regarding the "PATHS_FROM_ECLIPSE_TO_PYTHON" directive (see example above)
  4. in "/usr/lib/python2.7/site-packages/nova/debugger.py" do tell this module where to find "pydevd" by appending the module search path after the "import sys" using "sys.path.append('<where_to_find_pydevd>') or extending "PYTHONPATH" environment variable (this prevents changing the source code for the "pydevd" imports)
  5. start "nova-compute" manually using: "/usr/bin/nova-compute --config-file /etc/nova/nova.conf --remote_debug-host <ip_address_of_your_eclipse_environment> --remote_debug-port <pydev_debug_port_in_eclipse>"
  6. set a "pydevd.settrace()" command (see above) where you want to start debugging

For further details see http://novaremotedebug.blogspot.de/2013/12/debugging-nova-using-remote-debugger.html (this web page)!

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-07-11 02:43:53 -0600

Seen: 790 times

Last updated: Jul 15 '15