Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

I can`t get images from glance server

Hi all~

I have four physical servers and there is nova modules.

First node: nova-api,nova-objectstore,nova-scheduler,glance IP:10.250.0.1 Second node: nova-network IP:10.250.0.2 Third node: nova-compute IP:10.250.0.3 Fourth node: nova-volume IP:10.250.0.4

The problem is when I start run up a new instance, the Third node can`t get images from First node.

In Third node /var/lib/nova/instances/_base, there is only 0 byte file that name is 00000001.

drwxr-xr-x 2 nova nogroup 4096 2011-06-18 23:13 ./ drwxr-xr-x 5 nova root 4096 2011-06-18 23:13 ../

-rw-r--r-- 1 nova nogroup 0 2011-06-18 23:13 00000001

Here is Third node nova-compute.log

2011-06-18 23:13:27,313 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:578 2011-06-18 23:13:27,313 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:583 2011-06-18 23:13:27,313 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:27,319 INFO nova.virt.libvirt_conn [-] Instance instance-00000014 destroyed successfully. 2011-06-18 23:13:27,324 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:27,334 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:27,347 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:27,377 DEBUG nova.virt.libvirt.firewall [-] The nwfilter(nova-instance-instance-00000014-secgroup) for instance-00000014 is not found. from (pid=1620) unfilter_instance /usr/lib/pymodules/python2.7/nova/virt/libvirt/firewall.py:310 2011-06-18 23:13:27,378 INFO nova.virt.libvirt_conn [-] instance instance-00000014: deleting instance files /var/lib/nova/instances/instance-00000014 2011-06-18 23:13:37,440 DEBUG nova.rpc [-] received {u'_context_request_id': u'MUSPYCO5H9ZZNKG-9XPW', u'_context_read_deleted': False, u'args': {u'instance_id': 21, u'request_spec': {u'instance_properties': {u'state_description': u'scheduling', u'availability_zone': None, u'ramdisk_id': u'2', u'instance_type_id': 2, u'user_data': u'', u'vm_mode': None, u'reservation_id': u'r-qqoo8059', u'user_id': u'novaadmin', u'display_description': None, u'key_data': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDduHppPsr0FdQXO8immKC2Su5/LfRHEgfjbJd1Nw03TkZ+gMymGOw2eyXFLjLsHiXk3/fimicRu0fY1HcnAO0p8fjgMP13EDm/77qkzwbVe6WqxSxYHeT6HzBbNk/AfNMKIAayWAk5KQe9sVjWGYLN+rA0euRCRUGX7Z6W+LC48Q== nova@ubuntu-api001\n', u'state': 0, u'project_id': u'novaproject', u'metadata': {}, u'kernel_id': u'1', u'key_name': u'park', u'display_name': None, u'local_gb': 0, u'locked': False, u'launch_time': u'2011-06-18T14:13:36Z', u'memory_mb': 512, u'vcpus': 1, u'image_ref': 3, u'os_type': None}, u'instance_type': {u'rxtx_quota': 0, u'deleted_at': None, u'name': u'm1.tiny', u'deleted': False, u'created_at': None, u'updated_at': None, u'memory_mb': 512, u'vcpus': 1, u'rxtx_cap': 0, u'swap': 0, u'flavorid': 1, u'id': 2, u'local_gb': 0}, u'num_instances': 1, u'filter': u'nova.scheduler.host_filter.InstanceTypeFilter', u'blob': None}, u'admin_password': None, u'injected_files': None, u'availability_zone': None}, u'_context_is_admin': True, u'_context_timestamp': u'2011-06-18T14:13:36Z', u'_context_user': u'novaadmin', u'method': u'run_instance', u'_context_project': u'novaproject', u'_context_remote_address': u'10.250.0.1'} from (pid=1620) process_data /usr/lib/pymodules/python2.7/nova/rpc.py:202 2011-06-18 23:13:37,441 DEBUG nova.rpc [-] unpacked context: {'timestamp': u'2011-06-18T14:13:36Z', 'msg_id': None, 'remote_address': u'10.250.0.1', 'project': u'novaproject', 'is_admin': True, 'user': u'novaadmin', 'request_id': u'MUSPYCO5H9ZZNKG-9XPW', 'read_deleted': False} from (pid=1620) _unpack_context /usr/lib/pymodules/python2.7/nova/rpc.py:445 2011-06-18 23:13:37,488 AUDIT nova.compute.manager [MUSPYCO5H9ZZNKG-9XPW novaadmin novaproject] instance 21: starting... 2011-06-18 23:13:37,760 DEBUG nova.rpc [-] Making asynchronous call on network.ubuntu-net001 ... from (pid=1620) multicall /usr/lib/pymodules/python2.7/nova/rpc.py:475 2011-06-18 23:13:37,760 DEBUG nova.rpc [-] MSG_ID is 4a4f397a058c4d63b50ce684788b9945 from (pid=1620) multicall /usr/lib/pymodules/python2.7/nova/rpc.py:478 2011-06-18 23:13:38,194 DEBUG nova.utils [-] Attempting to grab semaphore "ensure_bridge" for method "ensure_bridge"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:578 2011-06-18 23:13:38,194 DEBUG nova.utils [-] Attempting to grab file lock "ensure_bridge" for method "ensure_bridge"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:583 2011-06-18 23:13:38,195 DEBUG nova.utils [-] Running cmd (subprocess): ip link show dev br100 from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,202 DEBUG nova.utils [-] Running cmd (subprocess): sudo route -n from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,215 DEBUG nova.utils [-] Running cmd (subprocess): sudo ip addr show dev eth1 scope global from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,229 DEBUG nova.utils [-] Running cmd (subprocess): sudo brctl addif br100 eth1 from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,242 DEBUG nova.utils [-] Result was 1 from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:159 2011-06-18 23:13:38,392 DEBUG nova.virt.libvirt_conn [-] instance instance-00000015: starting toXML method from (pid=1620) to_xml /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:996 2011-06-18 23:13:38,407 DEBUG nova.virt.libvirt_conn [-] instance instance-00000015: finished toXML method from (pid=1620) to_xml /usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py:999 2011-06-18 23:13:38,418 INFO nova [-] called setup_basic_filtering in nwfilter 2011-06-18 23:13:38,418 INFO nova [-] ensuring static filters 2011-06-18 23:13:38,443 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3b14710=""> 2011-06-18 23:13:38,444 INFO nova [-] <nova.db.sqlalchemy.models.securitygroupingressrule object="" at="" 0x3b14b90=""> 2011-06-18 23:13:38,444 DEBUG nova.utils [-] Attempting to grab semaphore "iptables" for method "apply"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:578 2011-06-18 23:13:38,444 DEBUG nova.utils [-] Attempting to grab file lock "iptables" for method "apply"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:583 2011-06-18 23:13:38,444 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t filter from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,459 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,474 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-save -t nat from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,488 DEBUG nova.utils [-] Running cmd (subprocess): sudo iptables-restore from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,517 DEBUG nova.utils [-] Running cmd (subprocess): mkdir -p /var/lib/nova/instances/instance-00000015/ from (pid=1620) execute /usr/lib/pymodules/python2.7/nova/utils.py:141 2011-06-18 23:13:38,524 INFO nova.virt.libvirt_conn [-] instance instance-00000015: Creating image 2011-06-18 23:13:38,555 DEBUG nova.utils [-] Attempting to grab semaphore "00000001" for method "call_if_not_exists"... from (pid=1620) inner /usr/lib/pymodules/python2.7/nova/utils.py:578 2011-06-18 23:13:38,556 ERROR nova.exception [-] Uncaught exception (nova.exception): TRACE: Traceback (most recent call last): (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 87, in _wrap (nova.exception): TRACE: return f(args, *kw) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 582, in spawn (nova.exception): TRACE: self._create_image(instance, xml, network_info=network_info) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 808, in _create_image (nova.exception): TRACE: project=project) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 743, in _cache_image (nova.exception): TRACE: call_if_not_exists(base, fn, args, *kwargs) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/utils.py", line 591, in inner (nova.exception): TRACE: retval = f(args, *kwargs) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 741, in call_if_not_exists (nova.exception): TRACE: fn(target=base, args, *kwargs) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/libvirt/connection.py", line 754, in _fetch_image (nova.exception): TRACE: images.fetch(image_id, target, user, project) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/virt/images.py", line 44, in fetch (nova.exception): TRACE: metadata = image_service.get(elevated, image_id, image_file) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/nova/image/glance.py", line 139, in get (nova.exception): TRACE: image_meta, image_chunks = self.client.get_image(image_id) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/glance/client.py", line 260, in get_image (nova.exception): TRACE: res = self.do_request("GET", "/images/%s" % image_id) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/glance/client.py", line 210, in do_request (nova.exception): TRACE: headers, params) (nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/glance/client.py", line 174, in do_request (nova.exception): TRACE: "server. Got error: %s" % e) (nova.exception): TRACE: ClientConnectionError: Unable to connect to server. Got error: [Errno 111] ECONNREFUSED (nova.exception): TRACE: 2011-06-18 23:13:38,588 ERROR nova.compute.manager [-] Instance '21' failed to spawn. Is virtualization enabled in the BIOS? Details: Unable to connect to server. Got error: [Errno 111] ECONNREFUSED (nova.compute.manager): TRACE: Traceback (most recent call last): (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/compute/manager.py", line 256, in run_instance (nova.compute.manager): TRACE: self.driver.spawn(instance_ref) (nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.7/nova/exception.py", line 93, in _wrap (nova.compute.manager): TRACE: raise Error(str(e)) (nova.compute.manager): TRACE: Error: Unable to connect to server. Got error: [Errno 111] ECONNREFUSED

(nova.compute.manager): TRACE:

Here is my nova.conf and all of nodes using same nova.conf

--sql_connection=mysql://root:mysqlpass@10.250.0.1/nova --s3_host=10.250.0.1 --rabbit_host=10.250.0.1 --cc_host=10.250.0.1 --ec2_url=http://10.250.0.1:8773/services/Cloud --routing_source_ip=10.250.0.2 --daemonize=1 --dhcpbridge_flagfile=/etc/nova/nova.conf --dhcpbridge=/usr/bin/nova-dhcpbridge --FAKE_subdomain=ec2 --ca_path=/var/lib/nova/CA --keys_path=/var/lib/nova/keys --networks_path=/var/lib/nova/networks --instances_path=/var/lib/nova/instances --images_path=/var/lib/nova/images --buckets_path=/var/lib/nova/buckets --libvirt_type=kvm --network_manager=nova.network.manager.FlatDHCPManager --flat_network_dhcp_start=10.0.0.2 --logdir=/var/log/nova --verbose --public_interface=eth0 --flat_interface=eth1 --flat_injected=False --lock_path=/var/lock/nova --zone_name=zone2 --glance_host=10.250.0.1 --glance_port=9292 --image_service=nova.image.glance.GlanceImageService --iscsi_ip_prefix=10.250.0.4

--volume_group=nova-volumes

Thanks.