At first the system will attempt to use swap if available before eventually killing processes due to Out of Memory (OOM), so the compute node itself will not crash if it reaches a full memory load but individual processes including the qemu-kvm processes that represent the guests can and will.
Techniques like memory ballooning can help alleviate such issues but they are not currently utilized by OpenStack Compute (Nova). Kernel Samepage Merging (KSM) is another technique that can help, albeit at the expense of some CPU usage, and can be used with Nova. Refer to https://www.kernel.org/doc/Documentat... for more information about KSM.
Neither of these techniques can completely eliminate the OOM killer if you overcommit too heavily, it is important to set the overcommit ratios after careful consideration of the real characteristics of your workload and environment.
You can find some more information about hypervisor tuning regarding memory at https://wiki.openstack.org/wiki/Docum...
