Yes, you can. You need one instance of libvirtd for each ESXi and then you can add each ESXi host as a compute node. Keep in mind though, that most of VMware's interessting functios (HA, DRS, dvSwitch, I believe even migration) is provided through and/or by vCenter and will not be available without vCenter (duh!).