Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Nova-manage can't sync DB

I am trying to set up an two node openstack, on fresh ubuntu 12.04 installations. As beginner following strictly the guide at http://openstack-folsom-install-guide.readthedocs.org/en/latest/, only replacing the ip's from the both nodes.

The controller/network-node (192.168.80.151) seems to be okay/ready/no errors, the compute-node (cn01, 192.168.80.152) throws some errors during setup of nova....

After configuration holding tight to the guide (see at the bottom), first errors (my fix-intentions and following error further down the page) appeared, while trying to "nova-manage db sync":

    """""" error #1 """""""""
    root@cn01:~# nova-manage db sync
    2013-06-18 15:30:31 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.migration' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.pyc'> from (pid=6360) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:663
    Command failed, please check log for more info
    2013-06-18 15:30:31 CRITICAL nova [-] <VerNum(161)>
    2013-06-18 15:30:31 TRACE nova Traceback (most recent call last):
    2013-06-18 15:30:31 TRACE nova   File "/usr/bin/nova-manage", line 1746, in <module>
    2013-06-18 15:30:31 TRACE nova     main()
    2013-06-18 15:30:31 TRACE nova   File "/usr/bin/nova-manage", line 1733, in main
    2013-06-18 15:30:31 TRACE nova     fn(*fn_args, **fn_kwargs)
    2013-06-18 15:30:31 TRACE nova   File "/usr/bin/nova-manage", line 1102, in sync
    2013-06-18 15:30:31 TRACE nova     return migration.db_sync(version)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 30, in db_sync
    2013-06-18 15:30:31 TRACE nova     return IMPL.db_sync(version=version)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 53, in db_sync
    2013-06-18 15:30:31 TRACE nova     versioning_api.upgrade(FLAGS.sql_connection, repo_path, version)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
    2013-06-18 15:30:31 TRACE nova     return _migrate(url, repository, version, upgrade=True, err=err, **opts)
    2013-06-18 15:30:31 TRACE nova   File "<string>", line 2, in _migrate
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
    2013-06-18 15:30:31 TRACE nova     return f(*a, **kw)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 345, in _migrate
    2013-06-18 15:30:31 TRACE nova     changeset = schema.changeset(version)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 80, in changeset
    2013-06-18 15:30:31 TRACE nova     changeset = self.repository.changeset(database, start_ver, version)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/repository.py", line 225, in changeset
    2013-06-18 15:30:31 TRACE nova     changes = [self.version(v).script(database, op) for v in versions]
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/repository.py", line 189, in version
    2013-06-18 15:30:31 TRACE nova     return self.versions.version(*p, **k)
    2013-06-18 15:30:31 TRACE nova   File "/usr/lib/python2.7/dist-packages/migrate/versioning/version.py", line 140, in version
    2013-06-18 15:30:31 TRACE nova     return self.versions[VerNum(vernum)]
    2013-06-18 15:30:31 TRACE nova KeyError: <VerNum(161)>
    2013-06-18 15:30:31 TRACE nova 
    """"""""""""""""""""""""""""""

I searched all support platforms for my error lines, tried some other configurations, without succes, setting them back. After some further browsing here (particularly here: /nova/+question/146204), I decided to try to solve the DEBUG message first, related to SqlALchemy, hoping to solve the rest... the action:

"""""""""""""""""""""""""""""

apt-get install python-pip

root@cn01:~# pip install sqlalchemy-migrate Requirement already satisfied (use --upgrade to upgrade): sqlalchemy-migrate in /usr/local/lib/python2.7/dist-packages Requirement already satisfied (use --upgrade to upgrade): SQLAlchemy>=0.6 in /usr/local/lib/python2.7/dist-packages (from sqlalchemy-migrate) Requirement already satisfied (use --upgrade to upgrade): decorator in /usr/local/lib/python2.7/dist-packages (from sqlalchemy-migrate) Requirement already satisfied (use --upgrade to upgrade): Tempita>=0.4 in /usr/local/lib/python2.7/dist-packages (from sqlalchemy-migrate) Requirement already satisfied (use --upgrade to upgrade): distribute in /usr/local/lib/python2.7/dist-packages (from sqlalchemy-migrate) Cleaning up... root@cn01:~# pip install sqlalchemy-migrate --upgrade Downloading/unpacking sqlalchemy-migrate Downloading sqlalchemy-migrate-0.7.2.tar.gz (105Kb): 105Kb downloaded Running setup.py egg_info for package sqlalchemy-migrate

warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*pyc' found anywhere in distribution
warning: no previously-included files found matching '.hgtags'
warning: no previously-included files matching '*' found under directory 'docs/_build'

Downloading/unpacking SQLAlchemy>=0.6 (from sqlalchemy-migrate) Downloading SQLAlchemy-0.8.1.tar.gz (3.8Mb): 3.8Mb downloaded Running setup.py egg_info for package SQLAlchemy

warning: no files found matching '*.jpg' under directory 'doc'
no previously-included directories found matching 'doc/build/output'

Downloading/unpacking decorator (from sqlalchemy-migrate) Downloading decorator-3.4.0.tar.gz Running setup.py egg_info for package decorator

warning: no previously-included files found matching 'Makefile'

Downloading/unpacking Tempita>=0.4 (from sqlalchemy-migrate) Downloading Tempita-0.5.1.tar.gz Running setup.py egg_info for package Tempita

Downloading/unpacking distribute (from sqlalchemy-migrate) Downloading distribute-0.6.45.tar.gz (723Kb): 723Kb downloaded Running setup.py egg_info for package distribute

Installing collected packages: sqlalchemy-migrate, SQLAlchemy, decorator, Tempita, distribute Found existing installation: sqlalchemy-migrate 0.7.2 Uninstalling sqlalchemy-migrate: Successfully uninstalled sqlalchemy-migrate Running setup.py install for sqlalchemy-migrate

warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*pyc' found anywhere in distribution
warning: no previously-included files found matching '.hgtags'
warning: no previously-included files matching '*' found under directory 'docs/_build'
Installing migrate script to /usr/local/bin
Installing migrate-repository script to /usr/local/bin

Found existing installation: SQLAlchemy 0.8.1 Uninstalling SQLAlchemy: Successfully uninstalled SQLAlchemy Running setup.py install for SQLAlchemy building 'sqlalchemy.cprocessors' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-2.7/lib/sqlalchemy/cextension/processors.o lib/sqlalchemy/cextension/processors.c:10:20: schwerwiegender Fehler: Python.h: Datei oder Verzeichnis nicht gefunden Kompilierung beendet. ************************ command 'gcc' failed with exit status 1 WARNING: The C extension could not be compiled, speedups are not enabled. Failure information, if any, is above. Retrying the build without the C extension now. ************************

warning: no files found matching '*.jpg' under directory 'doc'
no previously-included directories found matching 'doc/build/output'
***************************************************************************
WARNING: The C extension could not be compiled, speedups are not enabled.
Plain-Python build succeeded.
***************************************************************************

Found existing installation: decorator 3.4.0 Uninstalling decorator: Successfully uninstalled decorator Running setup.py install for decorator

warning: no previously-included files found matching 'Makefile'

Found existing installation: Tempita 0.5.1 Uninstalling Tempita: Successfully uninstalled Tempita Running setup.py install for Tempita

Found existing installation: distribute 0.6.45 Uninstalling distribute: Successfully uninstalled distribute Running setup.py install for distribute Before install bootstrap. Scanning installed packages Setuptools installation detected at /usr/local/lib/python2.7/dist-packages Non-egg installation Moving elements out of the way... Already patched. /usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info already patched.

Installing easy_install script to /usr/local/bin
Installing easy_install-2.7 script to /usr/local/bin
After install bootstrap.
/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info already exists

Successfully installed sqlalchemy-migrate SQLAlchemy decorator Tempita distribute Cleaning up... """""""""""""""""""""""""""""

So there were some error while compiling (PYthon.h not found), now I am not sure, what's up with sqlalchemy-migrate... Trying another db sync give's a new error:

"""""""""error 2"""""""""""""""""""" root@cn01:/var/log/nova# nova-manage db sync Command failed, please check log for more info 2013-06-18 16:28:42 CRITICAL nova [-] cannot import name exceptions 2013-06-18 16:28:42 TRACE nova Traceback (most recent call last): 2013-06-18 16:28:42 TRACE nova File "/usr/bin/nova-manage", line 1746, in <module> 2013-06-18 16:28:42 TRACE nova main() 2013-06-18 16:28:42 TRACE nova File "/usr/bin/nova-manage", line 1733, in main 2013-06-18 16:28:42 TRACE nova fn(fn_args, *fn_kwargs) 2013-06-18 16:28:42 TRACE nova File "/usr/bin/nova-manage", line 1102, in sync 2013-06-18 16:28:42 TRACE nova return migration.db_sync(version) 2013-06-18 16:28:42 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 30, in db_sync 2013-06-18 16:28:42 TRACE nova return IMPL.db_sync(version=version) 2013-06-18 16:28:42 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 667, in __getattr__ 2013-06-18 16:28:42 TRACE nova backend = self.__get_backend() 2013-06-18 16:28:42 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 662, in __get_backend 2013-06-18 16:28:42 TRACE nova self.__backend = __import__(name, None, None, fromlist) 2013-06-18 16:28:42 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 28, in <module> 2013-06-18 16:28:42 TRACE nova from migrate.versioning import api as versioning_api 2013-06-18 16:28:42 TRACE nova File "/usr/local/lib/python2.7/dist-packages/migrate/versioning/api.py", line 33, in <module> 2013-06-18 16:28:42 TRACE nova from migrate.versioning import (repository, schema, version, 2013-06-18 16:28:42 TRACE nova File "/usr/local/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 10, in <module> 2013-06-18 16:28:42 TRACE nova from sqlalchemy import exceptions as sa_exceptions 2013-06-18 16:28:42 TRACE nova ImportError: cannot import name exceptions 2013-06-18 16:28:42 TRACE nova """""""""""""""""""""""""""""

I read some older posts about issues about different migrate versions... but I am not sure if it's still a problem. So, right now, I don't know what to do anymore, maybe some kind & wise men is crazy enough to read all this and may help me along in any little way?

At the bottom I added some further information related to my problem. If I can give you more detailed infos/logs, please give me me a hint :)

Thank you thousand times in advance,

Daniel


Post scriptum:

This is what I did on a fresh Ubuntu:

apt-get install vlan bridge-utils
echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/99-openstack-ipv4-forwarding.conf
service procps start
sysctl -p
apt-get install -y kvm libvirt-bin pm-utils
vi /etc/libvirt/qemu.conf
        cgroup_device_acl = [
        "/dev/null", "/dev/full", "/dev/zero",
        "/dev/random", "/dev/urandom",
        "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
        "/dev/rtc", "/dev/hpet", "/dev/net/tun"
        ]
virsh net-destroy default
virsh net-undefine default
vi /etc/libvirt/libvirtd.conf
        listen_tls = 0
        listen_tcp = 1
        auth_tcp = "none"
vi /etc/init/libvirt-bin.conf
        env libvirtd_opts="-d -l"
vi /etc/default/libvirt-bin 
        libvirtd_opts="-d -l"
service libvirt-bin restart

apt-get install nova-network bridge-utils
vi /etc/network/interfaces
        auto lo
        iface lo inet loopback
        # The primary network interface
        auto br100
        iface br100 inet static
             address 192.168.80.152
             netmask 255.255.255.0
             network 192.168.80.0
             broadcast 192.168.80.255
             gateway 192.168.80.254
             # dns-* options are implemented by the resolvconf package, if installed
             dns-nameservers 8.8.8.8
             dns-search mtv.nimbula.org
             bridge_ports eth0
             bridge_stp off
             bridge_maxwait 0
             bridge_fd 0
brctl addbr br100; /etc/init.d/networking restart
""""
root@cn01:~# ifconfig
    br100     Link encap:Ethernet  Hardware Adresse b8:ac:6f:14:ff:aa  
              inet Adresse:192.168.80.152  Bcast:192.168.80.255  Maske:255.255.255.0
              inet6-Adresse: fe80::baac:6fff:fe14:ffaa/64 Gültigkeitsbereich:Verbindung
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metrik:1
              RX packets:93 errors:0 dropped:0 overruns:0 frame:0
              TX packets:67 errors:0 dropped:0 overruns:0 carrier:0
              Kollisionen:0 Sendewarteschlangenlänge:0 
              RX-Bytes:6046 (6.0 KB)  TX-Bytes:6302 (6.3 KB)

    eth0      Link encap:Ethernet  Hardware Adresse b8:ac:6f:14:ff:aa  
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metrik:1
              RX packets:90 errors:0 dropped:0 overruns:0 frame:0
              TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
              Kollisionen:2 Sendewarteschlangenlänge:1000 
              RX-Bytes:7268 (7.2 KB)  TX-Bytes:5914 (5.9 KB)
              Interrupt:36 Speicher:d6000000-d6012800 

    eth1      Link encap:Ethernet  Hardware Adresse b8:ac:6f:14:ff:ac  
              inet Adresse:10.10.20.1  Bcast:10.10.20.255  Maske:255.255.255.0
"""""
apt-get install nova-api-metadata nova-compute-kvm
vi /etc/nova/api-paste.ini
    [filter:authtoken]
    paste.filter_factory = keystone.middleware.auth_token:filter_factory
    auth_host = 192.168.80.151
    auth_port = 35357
    auth_protocol = http
    admin_tenant_name = service
    admin_user = nova
    admin_password = service_pass
    signing_dirname = /tmp/keystone-signing-nova

vi /etc/nova/nova-compute.conf
    [DEFAULT]
    libvirt_type=kvm
vi /etc/nova/nova.conf
    [DEFAULT]
    logdir=/var/log/nova
    state_path=/var/lib/nova
    lock_path=/run/lock/nova
    verbose=True
    api_paste_config=/etc/nova/api-paste.ini
    scheduler_driver=nova.scheduler.simple.SimpleScheduler
    s3_host=192.168.80.151
    ec2_host=192.168.80.151
    ec2_dmz_host=192.168.80.151
    rabbit_host=192.168.80.151
    cc_host=192.168.80.151
    metadata_host=192.168.80.152
    metadata_listen=0.0.0.0
    nova_url=http://192.168.80.151:8774/v1.1/
    sql_connection=mysql://novaUser:novaPass@10.32.14.232/nova
    ec2_url=http://192.168.80.151:8773/services/Cloud
    root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
    # Auth
    use_deprecated_auth=false
    auth_strategy=keystone
    keystone_ec2_url=http://192.168.80.151:5000/v2.0/ec2tokens
    # Imaging service
    glance_api_servers=192.168.80.151:9292
    image_service=nova.image.glance.GlanceImageService
    # Vnc configuration
    novnc_enabled=true
    novncproxy_base_url=http://192.168.80.151 10.32.14.232:6080/vnc_auto.html
    novncproxy_port=6080
    vncserver_proxyclient_address=192.168.80.152
    vncserver_listen=0.0.0.0
    # NETWORK
    network_manager=nova.network.manager.FlatDHCPManager
    force_dhcp_release=True
    dhcpbridge=/usr/bin/nova-dhcpbridge
    dhcpbridge_flagfile=/etc/nova/nova.conf
    firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
    # Change my_ip to match each host
    my_ip=192.168.80.152
    public_interface=br100
    vlan_interface=eth0
    flat_network_bridge=br100
    flat_interface=eth0
    #Note the different pool, this will be used for instance range
    fixed_range=10.10.20.0/24
    # Compute #
    compute_driver=libvirt.LibvirtDriver
    # Cinder #
    volume_api_class=nova.volume.cinder.API
    osapi_volume_listen_port=5900

    nova-manage db sync
    >> and here started my problems, the first pasted error.

PS: i tried alread "nova-manage --flagfile=/etc/nova/nova.conf db sync" without success.

PPS:

apt-get install -y ubuntu-cloud-keyring
echo deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/grizzly main >> /etc/apt/sources.list.d/grizzly.list
apt-get update -y
apt-get upgrade -y
apt-get dist-upgrade -y
apt-get install -y ntp
sed -i 's/server ntp.ubuntu.com/server 192.168.80.151/g' /etc/ntp.conf
service ntp restart
apt-get install -y vlan bridge-utils
sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf
sysctl net.ipv4.ip_forward=1


# i first followed an multiple-node-instruction, so I also did following:
apt-get install -y openvswitch-switch openvswitch-datapath-dkms
#br-int will be used for VM integration
ovs-vsctl add-br br-int
apt-get -y install quantum-plugin-openvswitch-agent
""" shell log:
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut       
Statusinformationen werden eingelesen... Fertig
quantum-plugin-openvswitch-agent ist schon die neueste Version.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
2 nicht vollständig installiert oder entfernt.
Nach dieser Operation werden 0 B Plattenplatz zusätzlich benutzt.
quantum-common (2012.1-0ubuntu4) wird eingerichtet ...
chown: Zugriff auf »/etc/quantum/“ nicht möglich: Datei oder Verzeichnis nicht gefunden
dpkg: Fehler beim Bearbeiten von quantum-common (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
dpkg: Abhängigkeitsprobleme verhindern Konfiguration von quantum-plugin-openvswitch-agent:
 quantum-plugin-openvswitch-agent hängt ab von quantum-common; aber:
  Paket quantum-common ist noch nicht konfiguriert.
dpkg: Fehler beim Bearbeiten von quantum-plugin-openvswitch-agent (--configure):
 Abhängigkeitsprobleme - verbleibt unkonfiguriert
Es wurde kein Apport-Bericht verfasst, da die Fehlermeldung darauf hindeutet, dass dies lediglich ein Folgefehler eines vorherigen Problems ist.
  Fehler traten auf beim Bearbeiten von:
 quantum-common
 quantum-plugin-openvswitch-agent
E: Sub-process /usr/bin/dpkg returned an error code (1)
""""" 
# so the installation failed by quantum-common cause missing access, and i stopped this way... source of error?:-/ and I switched to: http://openstack-folsom-install-guide.readthedocs.org/en/latest/

#

PPPS: other "nova-manage"-commands seem also to be broken, here "service list": """""""""""""""""""" root@cn01:/var/log/nova# nova-manage service list 2013-06-18 16:56:45 DEBUG nova.utils [req-d7595500-12f6-4ea4-b11b-c8581d813385 None None] backend <module 'nova.db.sqlalchemy.api'="" from="" '="" usr="" lib="" python2.7="" dist-packages="" nova="" db="" sqlalchemy="" api.pyc'=""> from (pid=6931) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:663 Command failed, please check log for more info 2013-06-18 16:56:45 CRITICAL nova [req-d7595500-12f6-4ea4-b11b-c8581d813385 None None] (OperationalError) (1054, "Unknown column 'services.availability_zone' in 'field list'") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.binary AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone \nFROM services \nWHERE services.deleted = false' () 2013-06-18 16:56:45 TRACE nova Traceback (most recent call last): 2013-06-18 16:56:45 TRACE nova File "/usr/bin/nova-manage", line 1746, in <module> 2013-06-18 16:56:45 TRACE nova main() 2013-06-18 16:56:45 TRACE nova File "/usr/bin/nova-manage", line 1733, in main 2013-06-18 16:56:45 TRACE nova fn(fn_args, *fn_kwargs) 2013-06-18 16:56:45 TRACE nova File "/usr/bin/nova-manage", line 964, in list 2013-06-18 16:56:45 TRACE nova services = db.service_get_all(ctxt) 2013-06-18 16:56:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 107, in service_get_all 2013-06-18 16:56:45 TRACE nova return IMPL.service_get_all(context, disabled) 2013-06-18 16:56:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 102, in wrapper 2013-06-18 16:56:45 TRACE nova return f(args, *kwargs) 2013-06-18 16:56:45 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 273, in service_get_all 2013-06-18 16:56:45 TRACE nova return query.all() 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2104, in all 2013-06-18 16:56:45 TRACE nova return list(self) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2216, in __iter__ 2013-06-18 16:56:45 TRACE nova return self._execute_and_instances(context) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2231, in _execute_and_instances 2013-06-18 16:56:45 TRACE nova result = conn.execute(querycontext.statement, self._params) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute 2013-06-18 16:56:45 TRACE nova params) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement 2013-06-18 16:56:45 TRACE nova compiled_sql, distilled_params 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context 2013-06-18 16:56:45 TRACE nova context) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception 2013-06-18 16:56:45 TRACE nova exc_info 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 163, in raise_from_cause 2013-06-18 16:56:45 TRACE nova reraise(type(exception), exception, tb=exc_tb) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context 2013-06-18 16:56:45 TRACE nova context) 2013-06-18 16:56:45 TRACE nova File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute 2013-06-18 16:56:45 TRACE nova cursor.execute(statement, parameters) 2013-06-18 16:56:45 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute 2013-06-18 16:56:45 TRACE nova self.errorhandler(self, exc, value) 2013-06-18 16:56:45 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler 2013-06-18 16:56:45 TRACE nova raise errorclass, errorvalue 2013-06-18 16:56:45 TRACE nova OperationalError: (OperationalError) (1054, "Unknown column 'services.availability_zone' in 'field list'") 'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.host AS services_host, services.binary AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone \nFROM services \nWHERE services.deleted = false' () 2013-06-18 16:56:45 TRACE nova """"""""""""""""""""""""""""