Just spit-balling here....

Dashrender

The HP Proliant DL585 G2, the first machine that McAlvin and I designed to take on NetApp in a 10,000 node compute cluster for NFS performance in 2007. Used RHEL 5 and NFS 3. Crushed a half million dollar NetApp tuned by the NetApp team directly for the test. This is the SAM-SD 0, it wasn't called a SAM-SD until years later.

what about your setup do you think allowed this box to crush them?

scottalanmiller

@Dashrender said:

what about your setup do you think allowed this box to crush them?

We know very well what it was - vastly higher threading performance provided by the combination of the Opterons and the RHEL OS. NetApp is built of FreeBSD and does not thread as well as Linux and is not tuned as well for storage performance (FreeBSD shines at network performance over Linux and the reverse for storage.) And the NetApp devices are disk heavy but did not have particularly powerful CPUs. The combination meant that the NFS layer would literally demand more from the CPUs than the platform could provide. The $20K Proliant with Linux was able to crush the $500K NetApp with the NetApp actually crashing and dying while the Proliant was able to complete all of the tests.

It was all predicted ahead of time and the DL585 chosen because of its ability to vastly "outthread" the NetApp.

scottalanmiller

Sun's ZFS team had helped us to identify why the NetApps that we had were unable to supply NFS reliably or quickly to our server cluster (10,000 HP Proliant DL145 G2s) and the design was based around addressing that issue cost effectively. We knew when the NetApps that we had would fail and invited NetApp to provide a solution, which they failed to do. The test harness that we used to gauge the NetApp vs. the SAM-SD was 3,000 compute nodes pulled out of production. The NetApp was unable to complete the test, the SAM-SD completed it with ease showing that it could handle, when used in a storage cluster, the entire compute cluster without a problem.

PSX_Defector

@Dashrender said:

@scottalanmiller said:

The HP Proliant DL585 G2, the first machine that McAlvin and I designed to take on NetApp in a 10,000 node compute cluster for NFS performance in 2007. Used RHEL 5 and NFS 3. Crushed a half million dollar NetApp tuned by the NetApp team directly for the test. This is the SAM-SD 0, it wasn't called a SAM-SD until years later.

what about your setup do you think allowed this box to crush them?

NetApp was pretty weak in performance back then. They were designed around massive scale out, which broke into the performance.

As someone who works in various "cloud" providers, using this kind of method would not really be worthwhile for us. I've used 3par and NetApp, and now Pure and Compellant for storage. But for SMB, this is absolutely perfect. I used a Dell PE2950 stacked with a bunch of SATA drives for a SAM-SD once. I needed file server space and backup destinations, not SQL IOPS. Thats the beauty of it, stack it with SSDs, you got something close to what Pure can supply. Stack it with SATA, you have a "NAS" that rivals anything out there. It's flexible and customize-able. You just have to decide what is most important for your organization.

Dashrender

@scottalanmiller said:

@Dashrender said:

what about your setup do you think allowed this box to crush them?

We know very well what it was - vastly higher threading performance provided by the combination of the Opterons and the RHEL OS. NetApp is built of FreeBSD and does not thread as well as Linux and is not tuned as well for storage performance (FreeBSD shines at network performance over Linux and the reverse for storage.) And the NetApp devices are disk heavy but did not have particularly powerful CPUs. The combination meant that the NFS layer would literally demand more from the CPUs than the platform could provide. The $20K Proliant with Linux was able to crush the $500K NetApp with the NetApp actually crashing and dying while the Proliant was able to complete all of the tests.

It was all predicted ahead of time and the DL585 chosen because of its ability to vastly "outthread" the NetApp.

With solutions like this, why isn't someone crushing the NAS/SAN market with these types of much lower cost solutions than the competition? Seems like these would fly off the shelf at 1/10 or less the cost of the competition.

scottalanmiller

@Dashrender said:

@scottalanmiller said:

@Dashrender said:

what about your setup do you think allowed this box to crush them?

We know very well what it was - vastly higher threading performance provided by the combination of the Opterons and the RHEL OS. NetApp is built of FreeBSD and does not thread as well as Linux and is not tuned as well for storage performance (FreeBSD shines at network performance over Linux and the reverse for storage.) And the NetApp devices are disk heavy but did not have particularly powerful CPUs. The combination meant that the NFS layer would literally demand more from the CPUs than the platform could provide. The $20K Proliant with Linux was able to crush the $500K NetApp with the NetApp actually crashing and dying while the Proliant was able to complete all of the tests.

It was all predicted ahead of time and the DL585 chosen because of its ability to vastly "outthread" the NetApp.

With solutions like this, why isn't someone crushing the NAS/SAN market with these types of much lower cost solutions than the competition? Seems like these would fly off the shelf at 1/10 or less the cost of the competition.

We've been providing these solutions for years. No one cares, people buying SAN nearly always are buying it for the purpose of saying that they have X brand SAN, not to actually get performance. You can call @ntg today and get this kind of storage from us. The SAM-SD is a concept, but you can get SAM-SD branded machines and support through xByte today too.

scottalanmiller

They do sell, but it isn't the massive market that you would imagine. The big money of SAN and NAS solutions is in their sales teams. If you aren't adding 100% markup so that you can take people to dinner and give away golf club memberships, people aren't buying your storage. Just because we CAN sell things at a fraction of the price doesn't mean that we can put a salesforce out there wining and dining the decision makers. That's how nearly all storage is sold.

DustinB3403

@scottalanmiller who is selling the "home-brew" NAS units that are mentioned?

scottalanmiller

@DustinB3403 said:

@scottalanmiller who is selling the "home-brew" NAS units that are mentioned?

I would never associate terms like "home brew" or "DIY" to a SAM-SD approach. Remember this is the identical process to how all enterprise servers are handled. Would you call your domain controller "home brew" or your database server a "DIY"?

DustinB3403

Sadly... with where I am employed at the moment I would call the DC and file servers Home Brew and DIY and DWTH (Disaster waiting to happen)

scottalanmiller

@DustinB3403 said:

Sadly... with where I am employed at the moment I would call the DC and file servers Home Brew and DIY and DWTH (Disaster waiting to happen)

But what about them makes them home brew? Did you write the OS in your space time? What makes them more or less "home brew" than any server in any enterprise? What makes them different than say a million dollar IBM mainframe? Expense, of course, and the quality and size of the parts, but the process of buying a server, installing an OS and setting it up is always the same. One is not more "home brew" than the other. This is a critical mindset that needs to be fixed as it is often used to sell low quality, overpriced appliances to management who get confused about how servers are deployed.

DustinB3403

The OS is all Microsoft Server 2### so an Enterprise operating system. The equipment that this is installed onto is my concern its not much more than a consumer desktop.

Hopefully I can get this to change for the better in the immediate future.

scottalanmiller

@DustinB3403 said:

The OS is all Microsoft Server 2### so an Enterprise operating system. The equipment that this is installed onto is my concern its not much more than a consumer desktop.

Hopefully I can get this to change for the better in the immediate future.

But that isn't homebrew in the least, it's just junk. Unrelated concepts. You can say a BMW is an enterprise car and a Toyota is junk. But no matter how crappy your Toyota is, it doesn't become homemade until you start getting parts and putting your own car together. The amount that it is "manufactured" in the same style process as a BMW is 100%. Quality is not related to the concept of being made at home as a hobby project.

JaredBusch

I disagree. Buying all of the components and putting it together is homebrew.

scottalanmiller

@JaredBusch said:

I disagree. Buying all of the components and putting it together is homebrew.

I would agree to some degree IF we are talking about whiteboxing. I'm not clear that that is what they have done.

Although, do you feel the same way if you get parts for a Proliant or a PowerEdge? What about self-assembly of a SuperMicro? It's a grey area as we almost always do some amount of assembly manually and almost always some is done for us. At what point, how many or which components, that we have to do ourselves does it switch from a "pre-made server" to a "homebrew"? Whitebox we know is when we assemble disassociated parts, but just assembling stuff at home doesn't imply that we are doing anything more "at home" than putting in a few screws.

The problem is, it implies so much more than is there. Even assembling a server from scratch (I've done this on a Proliant recently) I would never call that home brew. I think the term doesn't apply to servers. We have OEM and whitebox systems that cover the describable aspects. Homebrew makes people think of the IT guy "making" some aspect of the system himself (the OS, the application, the hardware... something) that goes above and beyond the existing concept of a whitebox.

At least that is how I hear it used - and people use the term to refer to Proliant and PowerEdge servers normally, not in the way that Dustin might be using it here. They actually use it on Spiceworks to mean "installing an OS yourself" being homebrew. Meaning every cloud system, every VM, every normal server is homebrew to those people.

scottalanmiller

I get that taking a bunch of spare parts laying around your house and assembling them into something that kind of works seems pretty home-brewish and I'd be okay if the term was being used when that happened and not just when someone has to install a server OS. But it shouldn't be linked to quality specifically, perhaps to support or something.