Connecting a NAS or SAN to a VMWare host



  • OK I'm having a discussion with a colleague and I guess I'm missing something.

    How much bandwidth is needed between a NAS or SAN to run say an SQL server on a VMWare host over ISCSI?

    I realize you need a whole host of information, like how many IOPs are needed, and much data is moving - I can't give that to you.. so let's just start with a starting point.. you would never do less than what?

    Now let's ramp that up. 3 blades (in an IBM Blade Center) connected to that same NAS - running 26 windows VMs - now how much bandwidth should you have? (i.e. how many 1 gb connections?


  • Service Provider

    Generally... relatively little. What you care most about is having a low latency. The bandwidth is rarely an issue. You can never know for sure without measuring (DPACK or whatever) but you can run a lot of IOPS over a single GigE iSCSI channel. Throughput is less of a concern, especially for databases. IOPS are the bigger concern.


  • Service Provider

    You would never do less than: GigE

    GigE is definitely the lowest speed channel that I would give for a database's storage backbone. That could be iSCSI, NFS, FC, whatever.


  • Service Provider

    @Dashrender said:

    Now let's ramp that up. 3 blades (in an IBM Blade Center) connected to that same NAS - running 26 windows VMs - now how much bandwidth should you have? (i.e. how many 1 gb connections?

    Remember that this is purely hypothetical what you are asking. Never, ever would I have a NAS or a SAN backing VMs in anything less than four nodes, ever (in production.) Slow and fragile while being expensive. That's a horrible combination.

    Really the slowest you should ever have is 6Gb/s because anything else, even things that are slower like GigE, is more expensive without benefit.


  • Service Provider

    Number of VMs and number of nodes and all of that might give us a tiny insight into IOPS, but only the tiniest. But bandwidth, that's really, really hard. IOPS tend to stick within a 100x variation or so (100 IOPS on the low end, 10,000 IOPS on the high end) but bandwidth would be much bigger with less than 100Mb/s on the low end and 12Gb/s on the high end. The fluctuations are bigger and the predictability is worse.



  • My friend's current setup in a 'hosted' solution is currently 26 physical servers.

    I say 'hosted' because it sounds like the hosting company is a home grown extension of what used to the company my friend works for.

    Anyhow - He's looking at another real hosted solution where they are providing 3 IBM blades in a bladecenter. Each blade has 2 attached disks (for the OS I'm guessing) but not for the VMs. His proposal is to install a NAS (IBM V7000 - I think) and connect those to the blades.

    Should he just walk away from that solution?



  • @scottalanmiller said:

    Number of VMs and number of nodes and all of that might give us a tiny insight into IOPS, but only the tiniest. But bandwidth, that's really, really hard. IOPS tend to stick within a 100x variation or so (100 IOPS on the low end, 10,000 IOPS on the high end) but bandwidth would be much bigger with less than 100Mb/s on the low end and 12Gb/s on the high end. The fluctuations are bigger and the predictability is worse.

    What caps IOPs? Is there a max you can pipe through a 1 GbE connection?


  • Service Provider

    @Dashrender said:

    Anyhow - He's looking at another real hosted solution where they are providing 3 IBM blades in a bladecenter. Each blade has 2 attached disks (for the OS I'm guessing) but not for the VMs.

    This is partially why I hate blades. They not only have problems on their own (more fragile, too expensive) but they then lead to second and third order bad decision making. The blades on their own might not be a horrible decision (but they probably are) but because of that one bad decision, then a second bad decision of needing high cost, fragile, complex shared storage is suddenly needed.

    Debunking the Blade Server Myth


  • Service Provider

    @Dashrender said:

    Should he just walk away from that solution?

    Not, not out of hand. But he needs to be wary. It's a silly setup top to bottom and probably way too costly for what he is getting.


  • Service Provider

    @Dashrender said:

    Each blade has 2 attached disks (for the OS I'm guessing) but not for the VMs.

    That is the intention of the design, correct.


  • Service Provider

    @Dashrender said:

    What caps IOPs? Is there a max you can pipe through a 1 GbE connection?

    There is probably a theoretical maximum but it would be really, really high. Your needs need to be determined by a combination of your IOPS and your throughput needs. Your IOPS are "almost" purely determined by your disk array. Your throughput is primarily determined by your connection medium. So to make any kind of determination you really need a complete picture, which is not easy to get. That's why a lot of people just go with 8Gb/s fiber channel, 10GigE NFS or iSCSI or 6-12Gb/s SAS. Just overbuy and not worry about it.


  • Service Provider

    IOPS are throughput are only kind of tied together. But in practice, the higher the IOPS the higher the throughput. Each IOP, by definition, requires some amount of bandwidth, but it might be a pretty small amount.

    Some workloads, like databases, typically are huge IOPS with very small throughput.

    On the opposite side are fileservers which are typically tiny IOPS with huge throughput.


  • Service Provider

    Sorry to sidetrack for a second, but I may be misunderstanding IOPS a little bit. Does the bit-size of an IO function vary, or is it 1-bit-per-I/O operation?


  • Service Provider

    @art_of_shred said:

    Sorry to sidetrack for a second, but I may be misunderstanding IOPS a little bit. Does the bit-size of an IO function vary, or is it 1-bit-per-I/O operation?

    Correct, the "payload" of an IOP can be all over the place. An IOP would be a single SCSI or ATA command (operation) which might be really simple and tiny or it might be rather sizable. So while throughput could be roughly calculated as IOP x IOP Size, the IOP Size will vary per IOP so you would need to know the average size for a specific workload to have a good idea of the throughput.


  • Service Provider

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)


  • Service Provider

    @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    LOL, that's correct obviously. It's pronounced that way, though, because people are thinking of "Input / Output Operation", the "Op" of operation makes the letters feel that way in your head. So people say IOP all of the time.

    Here is the calculation as Wikipedia writes it...

    IOPS * TransferSizeInBytes = BytesPerSec



  • @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    Lol Yup, it's either IO or IOPS. One is the actual thing, the other is a measurement.


  • Service Provider

    @ajstringham said:

    @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    Lol Yup, it's either IO or IOPS. One is the actual thing, the other is a measurement.

    Sort of. IOPS is actually sort for "Input / Output Operations Per Second." So which letters stand for which words?


  • Service Provider

    IOOP... Aye-Oop!


  • Service Provider

    @scottalanmiller said:

    @ajstringham said:

    @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    Lol Yup, it's either IO or IOPS. One is the actual thing, the other is a measurement.

    Sort of. IOPS is actually sort for "Input / Output Operations Per Second." So which letters stand for which words?

    Never saw it with "operations" in there. And if so, where is the other "O"?


  • Service Provider

    Something to think about....

    Traditional hard drives have a transfer cap of ~6Gb/s from the media perspective. In reality no drive can deliver that. But they might push over 100MB/s which is not far off from 1Gb/s. But they rarely top 150 IOPS.

    New SSDs are still capped at ~6Gb/s and while they will generally push a big more than 100MB/s, they can't push all that much more. However they routinely top 25,000 IOPS.


  • Service Provider

    @art_of_shred said:

    @scottalanmiller said:

    @ajstringham said:

    @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    Lol Yup, it's either IO or IOPS. One is the actual thing, the other is a measurement.

    Sort of. IOPS is actually sort for "Input / Output Operations Per Second." So which letters stand for which words?

    Never saw it with "operations" in there. And if so, where is the other "O"?

    No idea, but it was always stated that way and I looked it up after I said it to make sure that it wasn't one of those things that I made up in my head and just assumed was true but it wasn't, it really does have "operations" in there.



  • @scottalanmiller said:

    New SSDs are still capped at ~6Gb/s and while they will generally push a big more than 100MB/s, they can't push all that much more. However they routinely top 25,000 IOPS.

    But that aren't good for DB storage even though you need fast speeds for SQL as you have a lot of transactional writes. and SSDs have limited writes.... hmm.


  • Service Provider

    @thecreativeone91 said:

    @scottalanmiller said:

    New SSDs are still capped at ~6Gb/s and while they will generally push a big more than 100MB/s, they can't push all that much more. However they routinely top 25,000 IOPS.

    But that aren't good for DB storage even though you need fast speeds for SQL as you have a lot of transactional writes. and SSDs have limited writes.... hmm.

    Actually SSDs are ideal for databases. That limit write thing is a silly concept from a different era. Spinning rust have more limited lifespans than SSDs do. They just have different ways to measure and predict failure. Good SSDs have so many writes that their limitation is a positive, not a negative. SSDs + databases is the sweet spot. Nothing is better for a database. There is a reason that for the last five years nearly every high end enterprise database has been deployed to nothing except SSD.


  • Service Provider

    @scottalanmiller said:

    @art_of_shred said:

    @scottalanmiller said:

    @ajstringham said:

    @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    Lol Yup, it's either IO or IOPS. One is the actual thing, the other is a measurement.

    Sort of. IOPS is actually sort for "Input / Output Operations Per Second." So which letters stand for which words?

    Never saw it with "operations" in there. And if so, where is the other "O"?

    No idea, but it was always stated that way and I looked it up after I said it to make sure that it wasn't one of those things that I made up in my head and just assumed was true but it wasn't, it really does have "operations" in there.

    I think you're making that up.



  • @scottalanmiller said:

    @art_of_shred said:

    @scottalanmiller said:

    @ajstringham said:

    @art_of_shred said:

    OK, got it... but stop calling it an IOP. It's an Input/Output, not an Input/output Per. (sorry :P)

    Lol Yup, it's either IO or IOPS. One is the actual thing, the other is a measurement.

    Sort of. IOPS is actually sort for "Input / Output Operations Per Second." So which letters stand for which words?

    Never saw it with "operations" in there. And if so, where is the other "O"?

    No idea, but it was always stated that way and I looked it up after I said it to make sure that it wasn't one of those things that I made up in my head and just assumed was true but it wasn't, it really does have "operations" in there.

    Probably because pronouncing IOOPS is weird, where IOPs (aye-ops) is easy to say 😉



  • You're also going to need to know how read / write your environment is in terms of hitting the database, no?


  • Service Provider

    @NetworkNerd yes, you need to know your read / write mix to know what IOPS you need and where you need them.



  • Would NIC teaming / link aggregation help at all from a VMWare standpoint (i.e. multiple uplinks from SAN / NAS to switch to host)?



  • @NetworkNerd said:

    Would NIC teaming / link aggregation help at all from a VMWare standpoint (i.e. multiple uplinks from SAN / NAS to switch to host)?

    Yes, but You usually use 10GB ethernet or fiber for it not your normal 1GB switch.
    You also need switches designed for Iscsi/sans for the best performance.

    Granted it's been about 2.5 years since I've done a major SAN rollout so it could have change some since then..