Yea the documentation on that is sparse to non existant. The spicehtml5proxy.py:L77-L79 is where the ssl stuff would be read.

You should be able to set the following options in the nova.conf file:

[DEFAULT]
....
cert=/path/to/certfile
key=/path/to/keyfile
ssl_only=true

Alternatively, if you are using some sort of load balance I would highly recommend offload ssl at the load balancer then you don't have to config individual services for ssl. haproxy >=1.5 supports ssl offload