You are correct that a slow "controller" would generally impact the user experience through slower performance of the web GUI and potentially slower delivery of images to compute nodes. It would not impact the performance of instances once they have booted no your compute nodes.

It's not really possible for us to comment on your selection of hardware because we don't know what your requirements are in terms of number of instances, size of instances, and so forth. I will say that system memory is often the resource in greatest demand, although this is not necessarily true in your environment.