I have a growing ESX environment. We started with about beefy 3 ESX hosts, 150 VMs and 10 volumes. Each ESX host sees all 10 volumes. No big deal, makes it easy to move vms and not be concerned about any visibility issues.
Now we are 9 ESX hosts, 600 VMs and 150 volumes. Now the "mesh" scenario becomes more impractical.
We are adding 6 more ESX servers and upgrading to ESX 5, the environment is built but we don't have any VM's in there yet.
We typically keep only one or two "heavy I/O" VM's on one volume, and for "light I/O" VM's we stack 10-15 without issues. There are several pools of spindles and storage arrays to spread out the load. As of today, we are performing well. I want to keep it that way.
To be clear, we are NOT having any issues at the moment, but I do notice scanning volumes takes longer, and there is also increased risk of a single volume issue taking down a lot of ESX hosts if one volume should unexpectedly die or become invisible. Performance issues have occurred across the board when we had a couple of storage related hardware issues in the past and I see no reason if a host does not have any VM's on a volume that it should really need to see that volume in every single case. However, there is convenience and practicality when you work with clusters and HA and vmotion if you have all hosts visible to all volumes.
So, I want to know if anyone has had to deal with large numbers of volumes and hosts. What strategies do you employ and at what rules of thumb do you go by. I am working on a plan to create more than one cluster, divide LUNs among them and use a few volumes that are visible by all to handle vmotions between clusters. How big should a cluster be, how many LUN's should it service and what are some other factors I should be considering.
Thanks in advance for helpful comments!