Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Yes,the cluster created through REST was in waiting state.

I had created successfully one instance before the cluster launch. When I created my new cluster using REST having 1 master and 1 worker node ,, The master node and worker node launched successfully as instances with internal IP and cluster went into waiting state. I again launched an instance successfully.

Thus I have 4 successfully launched instances in active state.

1)before cluster 2 & 3) Cluster---> master node and slave node 4)after cluster goes into waiting state.

I created instances before and after cluster just to check whether my nova is contacting to keystone or not .

nova console-log of cluster-- master node is as follows as --->

//////////////////////////////////////////////////////////////////////////////// [......

[ 0.526863] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 0.528036] vgaarb: loaded [ 0.529064] vgaarb: bridge control possible 0000:00:02.0 [ 0.532400] SCSI subsystem initialized [ 0.534088] ACPI: bus type usb registered [ 0.535480] usbcore: registered new interface driver usbfs [ 0.536051] usbcore: registered new interface driver hub [ 0.537928] usbcore: registered new device driver usb [ 0.540386] PCI: Using ACPI for IRQ routing [ 0.542470] NetLabel: Initializing [ 0.544043] NetLabel: domain hash size = 128 [ 0.545438] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.547009] NetLabel: unlabeled traffic allowed by default [ 0.548201] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 0.550288] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.553013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.560122] Switching to clocksource kvm-clock [ 0.579695] AppArmor: AppArmor Filesystem Enabled [ 0.970489] pnp: PnP ACPI init [ 0.971648] ACPI: bus type pnp registered [ 0.974063] pnp: PnP ACPI: found 8 devices [ 0.975446] ACPI: ACPI bus type pnp unregistered [ 0.983793] NET: Registered protocol family 2 [ 0.985417] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.989392] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.998635] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 1.003642] TCP: Hash tables configured (established 262144 bind 65536) [ 1.005719] TCP: reno registered [ 1.006884] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 1.008784] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 1.010837] NET: Registered protocol family 1 [ 1.012308] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.014125] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.015896] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.017977] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 1.033502] audit: initializing netlink socket (disabled) [ 1.035208] type=2000 audit(1376481113.032:1): initialized [ 1.037043] Trying to unpack rootfs image as initramfs... [ 1.320896] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.328782] VFS: Disk quotas dquot_6.5.2 [ 1.330348] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.338159] fuse init (API version 7.19) [ 1.339641] msgmni has been set to 3963 [ 1.361148] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 1.368343] io scheduler noop registered [ 1.369800] io scheduler deadline registered (default) [ 1.377944] io scheduler cfq registered [ 1.381820] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.383633] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.454616] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.457374] ACPI: Power Button [PWRF] [ 1.544512] GHES: HEST is not enabled! [ 1.546389] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.548467] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 1.550488] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.576865] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.701117] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 1.728275] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.752534] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 1.760444] Linux agpgart interface v0.103 [ 1.769514] brd: module loaded [ 1.771549] loop: module loaded [ 2.060045] Refined TSC clocksource calibration: 2394.000 MHz. [ 2.107412] vda: vda1 [ 2.468051] vdb: unknown partition table [ 2.500090] scsi0 : ata_piix [ 2.508227] scsi1 : ata_piix [ 2.509408] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc1c0 irq 14 [ 2.511804] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc1c8 irq 15 [ 2.515247] Fixed MDIO Bus: probed [ 2.516567] tun: Universal TUN/TAP device driver, 1.6 [ 2.518165] tun: (C) 1999-2004 Max Krasnyansky maxk@qualcomm.com [ 2.524213] PPP generic driver version 2.4.2 [ 2.532062] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.534093] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.535969] uhci_hcd: USB Universal Host Controller Interface driver [ 2.544134] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 2.546053] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 2.749962] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c180 [ 2.805601] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 2.807841] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.812730] usb usb1: Product: UHCI Host Controller [ 2.814568] usb usb1: Manufacturer: Linux 3.5.0-36-generic uhci_hcd [ 2.818140] usb usb1: SerialNumber: 0000:00:01.2 [ 2.820008] hub 1-0:1.0: USB hub found [ 2.821489] hub 1-0:1.0: 2 ports detected [ 2.823189] usbcore: registered new interface driver libusual [ 2.901088] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.920456] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.922178] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.924007] mousedev: PS/2 mouse device common for all mice [ 2.926586] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.929677] rtc_cmos 00:01: RTC can wake from S4 [ 2.931946] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 2.934141] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs [ 2.936228] device-mapper: uevent: version 1.0.3 [ 2.983878] device-mapper: ioctl: 4.22.0-ioctl (2011-10-19) initialised: dm-devel@redhat.com [ 2.987132] cpuidle: using governor ladder [ 2.988793] cpuidle: using governor menu [ 2.990326] EFI Variables Facility v0.08 2004-May-17 [ 2.996416] ashmem: initialized [ 3.112828] TCP: cubic registered [ 3.114358] NET: Registered protocol family 10 [ 3.117155] NET: Registered protocol family 17 [ 3.118801] Key type dns_resolver registered [ 3.465389] usb 1-1: new full-speed USB device number 2 using uhci_hcd [ 4.333664] registered taskstats version 1 [ 4.445162] Freeing initrd memory: 18476k freed [ 4.765859] Key type trusted registered [ 5.459163] Key type encrypted registered [ 5.468823] Magic number: 9:568:883 [ 5.508555] rtc_cmos 00:01: setting system clock to 2013-08-14 11:51:57 UTC (1376481117) [ 5.532350] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 5.534329] EDD information not available. [ 5.537647] Freeing unused kernel memory: 932k freed [ 5.539729] Write protecting the kernel read-only data: 12288k [ 5.546311] Freeing unused kernel memory: 1468k freed [ 6.102661] Freeing unused kernel memory: 1132k freed Loading, please wait... [ 6.896710] usb 1-1: New USB device found, idVendor=0627, idProduct=0001 [ 6.898922] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5 [ 6.906007] usb 1-1: Product: QEMU USB Tablet [ 6.911417] usb 1-1: Manufacturer: QEMU 1.0 [ 6.912938] usb 1-1: SerialNumber: 42 [ 7.501545] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 8.187454] usbcore: registered new interface driver usbhid [ 8.189319] usbhid: USB HID core driver [ 8.459567] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 8.593574] input: QEMU 1.0 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input2 [ 8.598345] FDC 0 is a S82078B [ 8.601017] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 1.0 QEMU USB Tablet] on usb-0000:00:01.2-1/input0 [ 8.740930] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc900008ac000, fa:16:3e:4e:22:b2, IRQ 11 [ 10.282863] 8139too: 8139too Fast Ethernet driver 0.9.28 error: unexpectedly disconnected from boot status daemon Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. [ 20.346068] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) Begin: Running /scripts/local-bottom ... [ 21.583052] vda: vda1 GROWROOT: CHANGED: partition=1 start=16065 old: size=3052350 end=3068415 new: size=20948760,end=20964825 [ 24.501466] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) done. done. Begin: Running /scripts/init-bottom ... done. [ 38.061029] EXT4-fs (vda1): re-mounted. Opts: (null) mountall: Disconnected from Plymouth Cloud-init v. 0.7 running 'init-local' at Wed, 14 Aug 2013 11:53:31 +0000. Up 99.77 seconds. Cloud-init v. 0.7 running 'init' at Wed, 14 Aug 2013 11:53:41 +0000. Up 109.94 seconds. ci-info: ++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++ ci-info: +--------+------+-----------+-----------------+-------------------+ ci-info: | Device | Up | Address | Mask | Hw-Address | ci-info: +--------+------+-----------+-----------------+-------------------+ ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | ci-info: | eth0 | True | 10.1.1.8 | 255.255.255.224 | fa:16:3e:4e:22:b2 | ci-info: +--------+------+-----------+-----------------+-------------------+ ci-info: +++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++ ci-info: +-------+-------------+----------+-----------------+-----------+-------+ ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | ci-info: +-------+-------------+----------+-----------------+-----------+-------+ ci-info: | 0 | 0.0.0.0 | 10.1.1.3 | 0.0.0.0 | eth0 | UG | ci-info: | 1 | 10.1.1.0 | 0.0.0.0 | 255.255.255.224 | eth0 | U | ci-info: +-------+-------------+----------+-----------------+-----------+-------+ 2013-08-14 11:54:57,113 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [65/120s]: socket timeout [timed out] 2013-08-14 11:55:58,775 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [126/120s]: socket timeout [timed out] 2013-08-14 11:55:58,903 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 127 seconds Generating public/private rsa key pair. Your identification has been saved in /etc/ssh/ssh_host_rsa_key. Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. The key fingerprint is: c2:de:26:8b:b3:13:69:2b:23:0a:b1:37:ea:4c:21:cf root@ubuntu The key's randomart image is: +--[ RSA 2048]----+ | | | | | | | . | |o. .o S | |.=. +. o | |o.E. oo o | |=oo.+. + | |=+ oo+. | +-----------------+ Generating public/private dsa key pair. Your identification has been saved in /etc/ssh/ssh_host_dsa_key. Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub. The key fingerprint is: 95:a7:36:d8:5a:7a:9c:b7:74:f5:8a:e1:5a:d6:88:c1 root@ubuntu The key's randomart image is: +--[ DSA 1024]----+ | | | . | | o . | | = o | | S E .| | * = o ..| | o = B o .| | . * = . | | ..+ . | +-----------------+ Generating public/private ecdsa key pair. Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key. Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub. The key fingerprint is: df:1c:85:e1:14:1b:9e:cd:85:29:99:41:da:d2:4a:f7 root@ubuntu The key's randomart image is: +--[ECDSA 256]---+ | .B= o.| | B+Oo. | | + X.+ | | . + o | | S . . E | | . o . | | . o | | | | | +-----------------+ Cloud-init v. 0.7 running 'modules:config' at Wed, 14 Aug 2013 11:56:38 +0000. Up 286.90 seconds. * Starting AppArmor profiles [80G Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd [74G[ OK ] [74G[ OK ]g iSCSI initiator service iscsid [80G * Setting up iSCSI targets [80G iscsiadm: No records found [74G[ OK ] [74G[ OK ]g network filesystems [80G landscape-client is not configured, please run landscape-config. + touch /var/log/first-boot.d.log + chmod 0600 /var/log/first-boot.d.log + run-parts /etc/first-boot.d + rm -fr /etc/first-boot.d + rm -f /etc/rc.local + '[' -f /etc/rc.local.REAL ']' + mv /etc/rc.local.REAL /etc/rc.local + exit 0 Generating locales... en_US.UTF-8... done Generation complete. Cloud-init v. 0.7 running 'modules:final' at Wed, 14 Aug 2013 11:58:04 +0000. Up 372.04 seconds. ci-info: no authorized ssh keys fingerprints found for user ubuntu.ci-info: no authorized ssh keys fingerprints found for user ubuntu.ec2: ec2: ############################################################# ec2: -----BEGIN SSH HOST KEY FINGERPRINTS----- ec2: 1024 95:a7:36:d8:5a:7a:9c:b7:74:f5:8a:e1:5a:d6:88:c1 root@ubuntu (DSA) ec2: 256 df:1c:85:e1:14:1b:9e:cd:85:29:99:41:da:d2:4a:f7 root@ubuntu (ECDSA) ec2: 2048 c2:de:26:8b:b3:13:69:2b:23:0a:b1:37:ea:4c:21:cf root@ubuntu (RSA) ec2: -----END SSH HOST KEY FINGERPRINTS----- ec2: ############################################################# -----BEGIN SSH HOST KEY KEYS----- ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBluwxdspR0pFKw3xPHt4cdLQYkO6KR57LCa4pfxy0t+nK8txHibKfLghPrzq2qwCPsMIWMRsXTYK3eKvRi8O8A= root@ubuntu ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmK436gQGlasR5yRATJwcZ7uEaB2ddndjszUKwbrjcZXPwHrlxa0SlpFf/n+YSW6v4Zu7UXgYHG5/UtvsjWKpU+H8KHWHzNyB4bgDvqQNEfV/XiCKhnSETba+9l7vx3Jj3YKl7GXrihhjObpQ6stvInMiW0PpnQNimHqCrkKBu0o8+N6kaUJrAbeBillpzLBSTuVoWLdHu9Wr715hIfpU3yzJKsDANXhnERr7Eu7J4g4oAQeAx7+DfCePowqYM1BU3zNg7ZEnkweIl1FCfUBNEE5js3HHx2oW6OZl+Djt89c6pKf6IWHMnDsLzzBGfHt1ZONdR3L4b0AlNmD84Efnf root@ubuntu -----END SSH HOST KEY KEYS----- Cloud-init v. 0.7 finished at Wed, 14 Aug 2013 11:58:04 +0000. Datasource DataSourceNone. Up 372.38 seconds 2013-08-14 11:58:04,384 - cc_final_message.py[WARNING]: Used fallback datasource ]

Nova Console-log for worker node ---- >

//////////////////////////////////////////////////////////////////

[.............................

0.545048] vgaarb: bridge control possible 0000:00:02.0 [ 0.548420] SCSI subsystem initialized [ 0.550117] ACPI: bus type usb registered [ 0.552103] usbcore: registered new interface driver usbfs [ 0.553832] usbcore: registered new interface driver hub [ 0.556189] usbcore: registered new device driver usb [ 0.558023] PCI: Using ACPI for IRQ routing [ 0.562213] NetLabel: Initializing [ 0.564050] NetLabel: domain hash size = 128 [ 0.565599] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.568075] NetLabel: unlabeled traffic allowed by default [ 0.569992] HPET: 3 timers in total, 0 timers will be used for per-cpu timer [ 0.572083] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.574110] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.580138] Switching to clocksource kvm-clock [ 0.629017] AppArmor: AppArmor Filesystem Enabled [ 0.631122] pnp: PnP ACPI init [ 0.632339] ACPI: bus type pnp registered [ 0.634654] pnp: PnP ACPI: found 8 devices [ 0.636084] ACPI: ACPI bus type pnp unregistered [ 0.644401] NET: Registered protocol family 2 [ 0.645926] IP route cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.649893] TCP established hash table entries: 262144 (order: 10, 4194304 bytes) [ 0.662305] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 1.063239] TCP: Hash tables configured (established 262144 bind 65536) [ 1.065271] TCP: reno registered [ 1.066461] UDP hash table entries: 1024 (order: 3, 32768 bytes) [ 1.069345] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes) [ 1.071410] NET: Registered protocol family 1 [ 1.072883] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.074699] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.077155] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.079699] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 1.122762] audit: initializing netlink socket (disabled) [ 1.124533] type=2000 audit(1376481113.124:1): initialized [ 1.146679] Trying to unpack rootfs image as initramfs... [ 1.260863] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 1.266361] VFS: Disk quotas dquot_6.5.2 [ 1.267772] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.270515] fuse init (API version 7.19) [ 1.271938] msgmni has been set to 3963 [ 1.274120] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 1.276654] io scheduler noop registered [ 1.277990] io scheduler deadline registered (default) [ 1.279628] io scheduler cfq registered [ 1.281231] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 1.283001] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 1.286093] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 1.288583] ACPI: Power Button [PWRF] [ 1.291751] GHES: HEST is not enabled! [ 1.293799] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 1.295978] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 1.325144] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled [ 1.349465] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.394806] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 1.456339] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.535754] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 1.571399] Linux agpgart interface v0.103 [ 1.576246] brd: module loaded [ 1.578395] loop: module loaded [ 1.669599] vda: vda1 [ 2.234816] Refined TSC clocksource calibration: 2394.003 MHz. [ 2.322451] vdb: unknown partition table [ 2.526950] scsi0 : ata_piix [ 2.571421] scsi1 : ata_piix [ 2.572682] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc1c0 irq 14 [ 2.574749] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc1c8 irq 15 [ 2.587358] Fixed MDIO Bus: probed [ 2.588692] tun: Universal TUN/TAP device driver, 1.6 [ 2.590301] tun: (C) 1999-2004 Max Krasnyansky maxk@qualcomm.com [ 2.592256] PPP generic driver version 2.4.2 [ 2.593714] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 2.595953] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 2.597894] uhci_hcd: USB Universal Host Controller Interface driver [ 2.599911] uhci_hcd 0000:00:01.2: UHCI Host Controller [ 2.602058] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 [ 2.825596] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c180 [ 2.876567] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 [ 2.878949] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 2.882633] usb usb1: Product: UHCI Host Controller [ 2.884555] usb usb1: Manufacturer: Linux 3.5.0-36-generic uhci_hcd [ 2.890140] usb usb1: SerialNumber: 0000:00:01.2 [ 2.899940] hub 1-0:1.0: USB hub found [ 2.913245] hub 1-0:1.0: 2 ports detected [ 2.914970] usbcore: registered new interface driver libusual [ 2.932286] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.174045] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.175811] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.184181] mousedev: PS/2 mouse device common for all mice [ 3.210569] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.213822] rtc_cmos 00:01: RTC can wake from S4 [ 3.220416] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 3.266165] usb 1-1: new full-speed USB device number 2 using uhci_hcd [ 3.527125] hpet1: lost 2 rtc interrupts [ 4.631645] rtc0: alarms up to one day, 114 bytes nvram, hpet irqs [ 4.634386] device-mapper: uevent: version 1.0.3 [ 4.635959] device-mapper: ioctl: 4.22.0-ioctl (2011-10-19) initialised: dm-devel@redhat.com [ 4.638831] cpuidle: using governor ladder [ 4.640246] cpuidle: using governor menu [ 4.641608] EFI Variables Facility v0.08 2004-May-17 [ 4.703225] ashmem: initialized [ 4.704620] TCP: cubic registered [ 4.758452] NET: Registered protocol family 10 [ 5.666175] NET: Registered protocol family 17 [ 5.748298] Key type dns_resolver registered [ 5.817371] usb 1-1: New USB device found, idVendor=0627, idProduct=0001 [ 5.933739] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=5 [ 5.935883] usb 1-1: Product: QEMU USB Tablet [ 5.937377] usb 1-1: Manufacturer: QEMU 1.0 [ 5.938806] usb 1-1: SerialNumber: 42 [ 6.305794] registered taskstats version 1 [ 6.484833] Freeing initrd memory: 18476k freed [ 6.984644] Key type trusted registered [ 6.994136] Key type encrypted registered [ 7.003795] Magic number: 9:568:883 [ 7.028949] rtc_cmos 00:01: setting system clock to 2013-08-14 11:51:58 UTC (1376481118) [ 7.032084] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found [ 7.034207] EDD information not available. [ 7.099575] Freeing unused kernel memory: 932k freed [ 7.101669] Write protecting the kernel read-only data: 12288k [ 7.107900] Freeing unused kernel memory: 1468k freed [ 7.113746] Freeing unused kernel memory: 1132k freed Loading, please wait... [ 9.821507] usbcore: registered new interface driver usbhid [ 11.149419] usbhid: USB HID core driver [ 11.634797] 8139cp: 8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004) [ 20.620193] input: QEMU 1.0 QEMU USB Tablet as /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input2 [ 20.849425] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v0.01 Pointer [QEMU 1.0 QEMU USB Tablet] on usb-0000:00:01.2-1/input0 [ 21.001048] FDC 0 is a S82078B [ 21.075144] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 21.081246] 8139cp 0000:00:03.0: eth0: RTL-8139C+ at 0xffffc900008ac000, fa:16:3e:74:a8:5c, IRQ 11 [ 21.087646] 8139too: 8139too Fast Ethernet driver 0.9.28 error: unexpectedly disconnected from boot status daemon Begin: Loading essential drivers ... done. Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done. Begin: Running /scripts/local-premount ... done. [ 26.380350] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) Begin: Running /scripts/local-bottom ... [ 40.504733] vda: vda1 GROWROOT: CHANGED: partition=1 start=16065 old: size=3052350 end=3068415 new: size=20948760,end=20964825 [ 43.386164] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null) done. done. Begin: Running /scripts/init-bottom ... done. [ 56.234453] EXT4-fs (vda1): re-mounted. Opts: (null) mountall: Plymouth command failed mountall: Disconnected from Plymouth Cloud-init v. 0.7 running 'init-local' at Wed, 14 Aug 2013 11:53:36 +0000. Up 104.59 seconds. Cloud-init v. 0.7 running 'init' at Wed, 14 Aug 2013 11:53:41 +0000. Up 110.26 seconds. ci-info: ++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++ ci-info: +--------+------+-----------+-----------------+-------------------+ ci-info: | Device | Up | Address | Mask | Hw-Address | ci-info: +--------+------+-----------+-----------------+-------------------+ ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | ci-info: | eth0 | True | 10.1.1.7 | 255.255.255.224 | fa:16:3e:74:a8:5c | ci-info: +--------+------+-----------+-----------------+-------------------+ ci-info: +++++++++++++++++++++++++++++++Route info+++++++++++++++++++++++++++++++ ci-info: +-------+-------------+----------+-----------------+-----------+-------+ ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | ci-info: +-------+-------------+----------+-----------------+-----------+-------+ ci-info: | 0 | 0.0.0.0 | 10.1.1.3 | 0.0.0.0 | eth0 | UG | ci-info: | 1 | 10.1.1.0 | 0.0.0.0 | 255.255.255.224 | eth0 | U | ci-info: +-------+-------------+----------+-----------------+-----------+-------+ 2013-08-14 11:54:41,603 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: socket timeout [timed out] 2013-08-14 11:55:58,418 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [126/120s]: socket timeout [timed out] 2013-08-14 11:55:58,474 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 127 seconds Generating public/private rsa key pair. Your identification has been saved in /etc/ssh/ssh_host_rsa_key. Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. The key fingerprint is: 75:1e:9c:5c:a9:4e:81:32:f7:dc:87:b6:a2:9b:55:7f root@ubuntu The key's randomart image is: +--[ RSA 2048]----+ | . .. | | o oo.o. | | +.o+ . | | . o=.+ .| | S o...o | | o... | | ... E| | .o .| | o. | +-----------------+ Generating public/private dsa key pair. Your identification has been saved in /etc/ssh/ssh_host_dsa_key. Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub. The key fingerprint is: e8:96:ee:e1:97:0a:c3:9d:84:0b:f9:7c:df:03:43:6f root@ubuntu The key's randomart image is: +--[ DSA 1024]----+ | | | | | | | . . .. | | o . o.S. | | = = oo E | | B O = | | B oo.. | | .=o. .. | +-----------------+ Generating public/private ecdsa key pair. Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key. Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub. The key fingerprint is: 94:90:5c:61:ab:2f:80:52:ea:9b:e8:b2:cb:1f:65:1a root@ubuntu The key's randomart image is: +--[ECDSA 256]---+ | ..o+. | | oo o | | . + | | o . o | |o .E.o. S | |.. =. . | | . o . . | |+ o . . | |B.. | +-----------------+ * Starting AppArmor profiles [80G Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd Cloud-init v. 0.7 running 'modules:config' at Wed, 14 Aug 2013 11:56:49 +0000. Up 298.44 seconds. [74G[ OK ] [74G[ OK ]g iSCSI initiator service iscsid [80G * Setting up iSCSI targets [80G iscsiadm: No records found [74G[ OK ] [74G[ OK ]g network filesystems [80G landscape-client is not configured, please run landscape-config. + touch /var/log/first-boot.d.log + chmod 0600 /var/log/first-boot.d.log + run-parts /etc/first-boot.d Generating locales... en_US.UTF-8... + rm -fr /etc/first-boot.d + rm -f /etc/rc.local + '[' -f /etc/rc.local.REAL ']' + mv /etc/rc.local.REAL /etc/rc.local + exit 0 done Generation complete. Cloud-init v. 0.7 running 'modules:final' at Wed, 14 Aug 2013 11:58:04 +0000. Up 372.55 seconds. ci-info: no authorized ssh keys fingerprints found for user ubuntu.ci-info: no authorized ssh keys fingerprints found for user ubuntu.ec2: ec2: ############################################################# ec2: -----BEGIN SSH HOST KEY FINGERPRINTS----- ec2: 1024 e8:96:ee:e1:97:0a:c3:9d:84:0b:f9:7c:df:03:43:6f root@ubuntu (DSA) ec2: 256 94:90:5c:61:ab:2f:80:52:ea:9b:e8:b2:cb:1f:65:1a root@ubuntu (ECDSA) ec2: 2048 75:1e:9c:5c:a9:4e:81:32:f7:dc:87:b6:a2:9b:55:7f root@ubuntu (RSA) ec2: -----END SSH HOST KEY FINGERPRINTS----- ec2: ############################################################# -----BEGIN SSH HOST KEY KEYS----- ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNwf7JMAA7E20Q29j+fjMv0/74cPZhU7NT/S1YfvFnehaMp2jy7+1RC+MNWd8kKwY7KUtpNfhKUjF4QMPkUNg4E= root@ubuntu ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsVGbLjYR8vZdGbtG2AVUXK6ibcUaL0p5bUXHA5vm9DYhWA+w2fu1Yszl+QUAlzmUaQSwEpmj2/JCfyH5aGtJUyRGl4ppgubCyZmfvtl7kUeBDw1rq7E+edTn1BQMZLbN/oODLqfuY/rjboV1Fd+4Tml7rbcby5hfHMRAzzN/8Gt7sJApuwhgTGIlPTaMkQ66xBFpRYM1N1KH+T3mCkN2VL0vi4561a2rLCmddDYw4HttbNxXlbocj2bJSFXn1/+XV+DsMOioNN58N5EsTDTjPzESbFlpf1znaAO07HESTFLm4x0vB4o7DIdkVgU5roKQio6c+Vvn5pEk/c177+jtT root@ubuntu -----END SSH HOST KEY KEYS----- Cloud-init v. 0.7 finished at Wed, 14 Aug 2013 11:58:04 +0000. Datasource DataSourceNone. Up 372.97 seconds 2013-08-14 11:58:04,457 - cc_final_message.py[WARNING]: Used fallback datasource ]

The cluster-instances are connecting via their internal IP using ssh.

The debug log file of savanna ---------------> ///////////////////////////////////////////////////////////////////////////////////////////////

(31614) accepted ('10.208.36.50', 35577) 10.208.36.50 - - [14/Aug/2013 17:30:08] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/cluster-templates HTTP/1.1" 200 2649 0.015877 (31614) accepted ('10.208.36.50', 35578) 10.208.36.50 - - [14/Aug/2013 17:30:09] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/images HTTP/1.1" 200 585 0.203048 (31614) accepted ('10.208.36.50', 35589) 10.208.36.50 - - [14/Aug/2013 17:30:09] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/plugins/vanilla/1.1.2 HTTP/1.1" 200 92340 0.030381 (31614) accepted ('10.208.36.50', 35783) 10.208.36.50 - - [14/Aug/2013 17:30:25] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/cluster-templates HTTP/1.1" 200 2649 0.018988 (31614) accepted ('10.208.36.50', 35784) 10.208.36.50 - - [14/Aug/2013 17:30:25] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/images HTTP/1.1" 200 585 0.113088 (31614) accepted ('10.208.36.50', 35793) 10.208.36.50 - - [14/Aug/2013 17:30:25] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/plugins/vanilla/1.1.2 HTTP/1.1" 200 92340 0.015437 (31614) accepted ('10.208.36.50', 35800) 10.208.36.50 - - [14/Aug/2013 17:30:26] "POST /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters HTTP/1.1" 202 1288 0.884477 (31614) accepted ('10.208.36.50', 35829) 10.208.36.50 - - [14/Aug/2013 17:30:27] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters HTTP/1.1" 200 4387 0.044614 (31614) accepted ('10.208.36.50', 35855) 10.208.36.50 - - [14/Aug/2013 17:30:27] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters/5a515a4c-2e51-467c-aa32-964223c58ab1 HTTP/1.1" 200 1280 0.154778 2013-08-14 17:30:27.686 31614 WARNING savanna.service.instances [-] Can't start cluster 'tt' (reason: Second simultaneous read on fileno 11 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)) Traceback (most recent call last): File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 97, in wait readers.get(fileno, noop).cb(fileno) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main result = function(args, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/context.py", line 132, in wrapper func(args, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 111, in _provision_cluster i.create_cluster(cluster) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 52, in create_cluster _rollback_cluster_creation(cluster, ex) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 35, in create_cluster _create_instances(cluster) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 106, in _create_instances _run_instance(cluster, node_group, idx, aa_groups, userdata) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 165, in _run_instance key_name=cluster.user_keypair_id) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 638, in create *boot_kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/base.py", line 369, in _boot return_raw=return_raw, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/base.py", line 144, in _create _resp, body = self.api.client.post(url, body=body) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 232, in post return self._cs_request(url, 'POST', *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 213, in _cs_request *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 195, in _time_request resp, body = self.request(url, method, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 166, in request *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/requests/sessions.py", line 335, in request resp = self.send(prep, *send_kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/requests/sessions.py", line 438, in send r = adapter.send(request, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/requests/adapters.py", line 292, in send timeout=timeout File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 428, in urlopen body=body, headers=headers) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 288, in _make_request httplib_response = conn.getresponse(buffering=True) File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 407, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 365, in _read_status line = self.fp.readline() File "/usr/lib/python2.7/socket.py", line 447, in readline data = self._sock.recv(self._rbufsize) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/greenio.py", line 262, in recv timeout_exc=socket.timeout("timed out")) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 151, in trampoline listener = hub.add(hub.READ, fileno, current.switch) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 48, in add listener = BaseHub.add(self, evtype, fileno, cb) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 126, in add evtype, fileno, evtype)) RuntimeError: Second simultaneous read on fileno 11 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False) Removing descriptor: 20 (31614) accepted ('10.208.36.50', 35909) 10.208.36.50 - - [14/Aug/2013 17:30:30] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters/5a515a4c-2e51-467c-aa32-964223c58ab1 HTTP/1.1" 200 1612 0.023379 (31614) accepted ('10.208.36.50', 36048) 10.208.36.50 - - [14/Aug/2013 17:30:40] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters HTTP/1.1" 200 4719 0.032693 (31614) accepted ('10.208.36.50', 36407) 10.208.36.50 - - [14/Aug/2013 17:31:09] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters HTTP/1.1" 200 4719 0.027751 2013-08-15 11:04:47.607 31614 WARNING savanna.service.instances [-] Can't start cluster 'cluster-1' (reason: Unauthorized (HTTP 401)) 2013-08-15 11:04:48.039 31614 ERROR root [-] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 39, in create_cluster\n _await_instances(cluster)\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 208, in _await_instances\n if not _check_if_up(instance):\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 217, in _check_if_up\n server = nova.get_instance_info(instance)\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/utils/openstack/nova.py", line 69, in get_instance_info\n return client().servers.get(instance.instance_id)\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 365, in get\n return self._get("/servers/%s" % base.getid(server), "server")\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/base.py", line 139, in _get\n _resp, body = self.api.client.get(url)\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 229, in get\n return self._cs_request(url, \'GET\', *kwargs)\n', ' File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 226, in _cs_request\n raise e\n', 'Unauthorized: Unauthorized (HTTP 401)\n'] Traceback (most recent call last): File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 97, in wait readers.get(fileno, noop).cb(fileno) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main result = function(args, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/context.py", line 132, in wrapper func(args, *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/api.py", line 111, in _provision_cluster i.create_cluster(cluster) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 52, in create_cluster _rollback_cluster_creation(cluster, ex) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 292, in _rollback_cluster_creation _shutdown_instances(cluster, True) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 321, in _shutdown_instances _shutdown_instance(instance) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/savanna/service/instances.py", line 327, in _shutdown_instance nova.client().servers.delete(instance.instance_id) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/v1_1/servers.py", line 668, in delete self._delete("/servers/%s" % base.getid(server)) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/base.py", line 153, in _delete _resp, _body = self.api.client.delete(url) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 238, in delete return self._cs_request(url, 'DELETE', *kwargs) File "/etc/init.d/savanna-venv/local/lib/python2.7/site-packages/novaclient/client.py", line 226, in _cs_request raise e Unauthorized: Unauthorized (HTTP 401) Removing descriptor: 48 (31614) accepted ('10.208.36.50', 50014) 10.208.36.50 - - [16/Aug/2013 08:39:33] "GET /v1.0/11f94438f0f1466d814f2cf5a6f2839a/clusters HTTP/1.1" 200 4888 0.569258