Undercloud upgrade failed

asked 2019-08-29 03:50:09 -0500

ClaudiuPID gravatar image

updated 2019-08-29 05:06:04 -0500

Hi,

After updating openstack-tripleo-heat-templates from 10.5 to 10.6 (using centos) and other tripleo packages (minor versions) undercloud fails to upgrade. Ran:

openstack undercloud upgrade

Got as result:

Exception: Post Upgrade failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1262, in _standalone_deploy
    raise exceptions.DeploymentError('Post Upgrade failed')
DeploymentError: Post Upgrade failed
None
** Found ansible errors for undercloud deployment! **
[
 [
  "Clean podman images",
  {
   "stderr_lines": [
    "Error: failed to prune image: Image used by 9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5: image is in use by a container"
   ],
   "changed": true,
   "end": "2019-08-29 11:42:30.877390",
   "_ansible_no_log": false,
   "stdout": "",
   "cmd": "podman image prune -a",
   "start": "2019-08-29 11:42:30.682528",
   "delta": "0:00:00.194862",
   "stderr": "Error: failed to prune image: Image used by 9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5: image is in use by a container",
   "rc": 125,
   "invocation": {
    "module_args": {
     "creates": null,
     "executable": null,
     "_uses_shell": true,
     "strip_empty_ends": true,
     "_raw_params": "podman image prune -a",
     "removes": null,
     "argv": null,
     "warn": true,
     "chdir": null,
     "stdin_add_newline": true,
     "stdin": null
    }
   },
   "stdout_lines": [],
   "msg": "non-zero return code"
  }
 ]
]
Install artifact is located at /home/stack/undercloud-install-20190829084230.tar.bzip2
The heat stack undercloud virtual state/action is reset to CREATE. Use "--force-stack-update" to set it forcefully to UPDATE
Removing the stack virtual update mark file /var/lib/tripleo-heat-installer/update_mark_undercloud
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-EofYwP

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

reset failed: reset: standard error: Inappropriate ioctl for device

Forgot to mention, process running for that image ID:

/usr/libexec/podman/conmon -s -c 9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5 -u 9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5 -n ironic_api -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5/userdata -p /var/run/containers/storage/overlay-containers/9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5/userdata/pidfile --exit-dir /var/run/libpod/exits --conmon-pidfile /var/run/ironic_api.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 9c7b8d36f95751d86f9c0287a9552c5c3dae4e511e0fcb3f89890620c0ac31f5 --socket-dir-path /var/run/libpod/socket -l k8s-file:/var/log/containers/stdouts/ironic_api.log --log-level error

Any ideas on what might be wrong?

edit retag flag offensive close merge delete

Comments

I managed to get it reinstalled by stopping all containers reported by "podman image prune -a" and removing them. From what I can see my modes are ok. Am not sure about: Deployment successful! AND reset failed: reset: standard error: Inappropriate ioctl for device (say success but with an error)

ClaudiuPID gravatar imageClaudiuPID ( 2019-08-29 14:17:56 -0500 )edit