Best posts made by StorageNinja

@jaredbusch said in Ubiquiti Security Gateway:

@john-nicholson said in Ubiquiti Security Gateway:

@stacksofplates said in Ubiquiti Security Gateway:

I never saw the value in the USG. Maybe if you were doing full network automation and copying configs, and then that defeats the purpose of the USG anyway. I have an ERL and an EdgeSwitch Lite at home, along with 2 APs.

The rest of my gear is manageable by Unifi. It's a bit weird my one device that does DPI and edge services isn't.

From that point of view, if that is all you need from the router, the USG is a perfectly valid choice for you.

Really I'm going to keep it as my base layer 4 edge device and likely put an inline IPS system here (Palo Alto currently winning).

@jaredbusch said in Looking for virtualization advice:

@scottalanmiller said in Looking for virtualization advice:

@garyp said in Looking for virtualization advice:

We have started looking at HCI solutions, including Scale. StarWind and HPE SimpliVity as we do not the expertise in managing a hypervisor nor the time to manage it.

That's the appropriate short list. Of those, @Scale is the one that is going to offload the most from your plate. Starwind provides HC but you are still managing the hypervisor on your own, separately. It's architecturally all together, but the management console is not.

He doesn't need hyperconvergence. Don't sell him something he does not need. There is no way to intelligently get 15TB of storage on a Scale box for a reasonable price compared to local storage.

Other HCI platforms have no problem getting 15TB in a box (I had a serious discussion with an SE about a 300TB+ per host vSAN design and all the considerations around it). I think someone was taking socket based licensing to a ridiculous logical conclusion but with 100Gbps networking becoming affordable this stuff isn't that far off.

I regularly see 90TB configurations. Starwind and other mirror based SDS systems with direct connect can scale pretty deep.

The other thing to note is if most of this data is ice cold, it may be better to put it into an archive system. CloudArray or Choehsity virutal appliances and other systems like it allow you to dump the cold data into an ingestion point (NAS share, or iSCSI) where it is dedupe and compressed and then cold data is pushed out to the cloud and tiered to an object store of your choice. This way you can run HCI with Asymmetric storage growth. For engineering shops who have to retain 10 years of stuff this is a good way to make it efficient, archive it, tell it to mirror to two different AZ's in glacier and then ignore it. By doing data reduction on the ingestion you can cut down on your cloud storage quite a bit.

He can likely get a decent solution for a lot closer to 50K than 100K. (Go single socket, and use the HCI acceleration kits).

While a single server solution is fine, being able to do non-disruptive maintenance is really damn nice, especially with systems like HVAC and security that may have compliance or safety requirements on staying online.

For telecom, pay attention to what hypervisor your platform supports. A lot are picky because of timing concerns, or how they do clocking for trans-coding. If your not trans-coding it's not normally a huge deal but trans-coding is where stuff can get weird.

Curious why DR to Azure or AWS. While there are solutions that can do it, there are cheaper/better IaaS players who offer DRaaS (Look at Veeam's partner network, there's a lot of good players there).

If your hell bent on DR to Azure and AWS, choose Hyper-V for Azure, and ESXi for AWS (as VMware on AWS will be coming out of beta soon) would be my picks. Ideally though if your doing DR to either you'd be doing it at app/PaaS abstraction layer using something like Pivitol but you are too small for this.

@jaredbusch said in Looking for virtualization advice:

This is way to generic. A large part of this will depend on the hypervisor and also on retention needs.

Use an agent-based backup software and upload anythig to s3.

Also will the backup software restore things like Active Directory, GPO's, SQL Schema's. Does it allow you to search for files (Good luck finding that PDF if you don't have an index!). Does it let you restore pieces of Mail from exchange etc.

Also you say DR to Azure or AWS. Copying 16TB of data there isn't DR. That BC. IF you need to re-hydrate 16TB of data at a file level over the WAN from S3 (which isn't fast to re-hydrate cold data over the WAN) it can get... fun....

@jaredbusch said in Looking for virtualization advice:

I don't thing you need any sort of HA technology. You'll thank me later for the removed complexity.

Hypervisor HA (at least on ESXi) isn't that complex anymore. It's well known, there are hundreds of thousands of people certified on it, and you can get remote install support from major HCI players for a few thousand bucks to make sure it's setup right if your really paranoid. The era of LUNs, and tuning APD/PDL timers, and figuring out SCSI queues, and Fibre Channel, and DCB is over. Even internal stuff is more safe as you have push button updates of Firmware and Controllers for the system that will roll through the cluster and take care of stuff.

Modern Hypervisor HA is actually really damn smart. IT can even detect that host MIGHT fail (failing hardware sensors) and pro-actively quarantine a host. It can fence through multiple levels (File system, host heartbeats, isolation address's) with and deliver incredibly consistent outcomes on different failure odes.

The book on HA was never that long, and Duncan's simplified it when he now gives it away for free.

Now App HA is still often expensive, and compex (relatively BAG and AD are not that bad) but for stuff like HVAC systems it's rarely an option.

@jaredbusch said in Looking for virtualization advice:

VMware-based appliances will probably stay with a vCenter + plug-in, but Hyper-V (and KVM soon, very soon) are getting own HTML5 GUI.

Starwind supports vVols, so you can manage storage without a plugin using that (well VASA provider is required, but no weird GUI plugin nonsense).

@jaredbusch said in Looking for virtualization advice:

For your setup, I would go with a single large server with local storage. Get 8 x 4 TB SATA or NL SAS disks in a RAID 10 for 16TB of usable space. The Dell R720 would work fine here (or whatever the current model is).

The R720 isn't being certified for new OS and hypervisor releases I'm pretty sure no 6.5 for it. I wouldn't get one of those.

R730 is current with R740 phasing in to replace it. I'd get the R640/740.

Don't deploy Magnetic SATA drives. I don't care if they say enterprise. Spend the extra $30 and get NL-SAS. Also worth noting is that NL-SAS have awful performance characteristics. Make sure your workload has incredibly low IO load or get ready for the database server when you run a report on the PBX to implode.

@jaredbusch said in Looking for virtualization advice:

@scottalanmiller said in Looking for virtualization advice:

I had to go searching for it.

Not my job. It is their job to not turn me off. Which they did. fuck that.

Everyone has public pricing if you know how to use Government contract price lists that contractually must be posted

Citrix, Microsoft, Dell, Cisco and others have public pricing, but if you just go with that you can easily miss out on a special 30% off bundle or deal (or in Cisco's case likely just overpay 70%). I get this weird attitude in IT where you don't want to deal with sales people but do you want to know the real reason why vendors don't give out price lists?

One is volume. If Bank of America agree's to buy my software I might be willing to discount to 85% for 100 million dollars in sales, because my net cost to produce licensing keys is low, and the cost to support someone that large and well developed who's going to have a homogeneous design is actually rather simple. Joe's gravel pit with 10 VM's and 2 IT guys with no formal training meanwhile could easily be a endless pit of support calls because they cook their unit that's in a trailer, and they are abusing your support organization for basic helpdesk because they don't understand basic things like NTP/DNS and routing. That's ony part of it. The other issue is how SMB's buy complicated products.
They go and buy it, and then buy the wrong damn thing. They will buy 1/2 as many nodes and not realize they need mirroring overhead (Scale shows RAW on this chart, they don't show overheads, reserves or protection. They will buy zero slack space on the boxes and crash them or not be aware of the file system formatting. They will then complain on Spiceworks "Storage vendor B sucks because their box ran out of space and caught fire and gave me measles because I didn't talk to an SE who could have helped me get the right box or realize that I shouldn't buy the product and get something else that was in budget!"

As someone who worked for a VAR for years and saw customers try to design their own stuff and submit it, we ended up blocking 90% of those deals and getting a SE to get them the right stuff. Some of it was hilarious (RAID 0 ALL THE THINGS) some of it was easy to overlook stuff (Product required 3 phase power, or a rack depth they didn't have), some of it is just annoying details (Product had SFP+ ports, and they needed 10Gbase-T). Minor paperwork mistakes (Shipping address being different from install) on a customers part could cripple support responses (Dispatch part to the wrong location). Also for systems with support contracts array vendors and HCI vendors will require a diagram of how it's installed, cabled, connected so their global support teams know what you have when they call in. If you let customers size, purchase, and install none of the paperwork required for a proper support experience gets completed. A VxRAIL or Scale or Synology could be setup by a 4th grader with the check list. The reason one of them requires a trained staff member is VCE/Scale is on the hook for aggressive support and management of patching and things for the next 3-5 years, while Synology will say "meh" if you call them with an outage.

As a vendor if you shut up and take their money you risk tanking your NPS, and having them tell their friends you suck and making it harder to get deals from people who actually have budget and care about a solution actually working.

Catering to the IT know it all who thinks he can correctly purchase systems that with sub-variations often produce 1500 SKU's (RAM, CPU, Disk, NIC interfaces, Power connectors, Fan direction etc) isn't worth it to these vendors as they are more worried about other customers hearing about your negative experience than loosing a sale that was never going to happen anyways. More importantly if you correctly know the Sub-SKU's for the flex midplane option so you can run 2 HBA's in split mode on a DL380 you REALLY shouldn't be working at a customer with that skill (it's something you rarely use) and instead should be working for a VAR or distributor. You'll make a ton more money, and you'll actually get to use that skill more than once every 3-5 years.

@dashrender said in Looking for virtualization advice:

@scottalanmiller said in Looking for virtualization advice:

@dashrender said in Looking for virtualization advice:

@scottalanmiller said in Looking for virtualization advice:

@garyp said in Looking for virtualization advice:

We have equipment in 5 NA locations and over 500 phones, so management is not looking to move to anything else at this time.

Why would management be involved? IT should be like "we can save money, improve systems" and that's the end of it. Why would management have any say other than verifying cost savings and such? The bigger the network, the more money there is to be saved, right?

He's in the same situation as me. The hardware is already in place. Moving to FreePBX would likely require purchasing all new phones, or moving users to softphones on their computers, which would require purchasing headsets most likely.

In either case, there would be a substantial hardware outlay likely if they changed.

But they are looking at significant outlay to keep using what they have. They have to invest specifically in a VMware solution instead of what meets the needs of the business, they have to pay to keep the Avaya running and they have to take on the risks of using a solution from a non-viable or marginally viable vendor. That's all real costs that they are facing to NOT switch.

I'm not saying you're wrong - but 500 phones, even Yealink aren't cheap, not to mention the training to the staff, the IT time, etc.

If it really boils down to it, they can just leave it on the server it's currently on, and change nothing else about that one server. We assume there is already a backup solution in place - so that shouldn't be that bad to maintain.

Then the business can plan for this change over down the road.

Consulting across mid-market and enterprise really exposed to me how much money it is to replace some systems no matter how old and shitty they are just not being worth the capital. I was helping one customer with Millienium PBX"s (yes that ancient) that were 20 years out of service. Did they need to upgrade? Sure. Did they have over 10K phones they would need to replace when they moved to a new system. yup. Sadly this system was designed with 100% American military spec capacitors and DSP's and was used on coast guard ships so it was hardened to hell and back against the elements so it wouldn't die.

Did it support VOIP trunks? No. Did we cludge in VOIP to PRI Adtrans to make up for this? Yup!

40-50K for a new VMware HA cluster vs. Millions for some nicer features, but a huge balance sheet hit for the quarter that will spook investors on the ROIC ratio? Thanks but no thanks.

add in risks of outages, disruption on retraining staff and that 40-50K to kick that can down the road looks REALLY damn atractive.

Remember why people ran AS400's and mainframes forever? Beacuse migrations were trivial and the incremital trade in cost was often low. IBM kept the "total value" vs. a full code re-write always SLIGHTLY better looking in the short to middle term and that's all most people care about.

That $20 hr was for a contractor (yah, that's insanely low labor considering someone's skimming something off it). Note I lice in Houston, a fairly cheap city and that rate is what you'd pay a supervisor at a gas Station. It's unskilled cannon fodder here at that rate. Sometimes you get lucky (I hired a guy who moved furniture at IKEA for that for helpdesk, he ended up being smart and had to give him a raise to 56K at the end of the year though to keep him around.

In this case It was an unskilled typist, with no formal equation we taught to do the copy pasta cleanups from some scripts we wrote to try to accelerate it.

How I got out of making $20 an hour (hell slightly less than that when I started in this field) was identifying the lowest skilled things I did and then finding the cheapest resource who could do it for me. If your the god of Oracle RAC but still changing printer toner management is going to pay you like a printer serf. Stop doing cheap labor and you'll get paid more (assuming there are more valuable things to do in the day, if not find a new job).

I don't judge the value of a human being based on how much they are paid (I just spent July mostly in countries where a lot are on less than $2 a day). I do judge the value of labor (I was a hiring manager and had to know fair market rates). If your un-happy getting paid $20 an hour that's honestly your issue. If your unhappy that I say it's a cheap rate for labor in the US in a metro area for someone handling work on an IT department that's just disagreeing with a fact. I hear that's popular these days, but I've never understood it as a concept.

@psx_defector said in Looking for virtualization advice:

@storageninja said in Looking for virtualization advice:

Something I learned from being a group admin for Spiceworks for years is that enterprise types lurk heavily in SMB forums.

Hey, shush your mouth. I don't lurk, I troll.

WAIT, PSX IS STILL A THING?!?

@dashrender said in Is this server strategy reckless and/or insane?:

UREs are probably pretty low on these SSDs, but not zero, so something else to consider, what are the chances of a URE killing your RAID 0? (now Scott will educate me that these don't matter - seriously don't know if do or not)

The failure mode that you should be afraid of isn't a URE, but data loss on power loss that is OUT of order with acknowledged writes. This breaks standard data loss recovery you get from a Journal Log on MySQL and other database apps.

If it would just cleanly loose the last write that would be fantastic, sadly it's how the Samsung consumer drives tend to recover lost data. Normally they are used in laptops that have a giant battery attached, are not running RAID (Which will see this out of order recovery as a failed drive when it fails their ECC check). This can/will catastrophically fail with multiple drives dropping on something as simple as a controller or host failure or hard reboot.

@creayt said in Is this server strategy reckless and/or insane?:

@storageninja said in Is this server strategy reckless and/or insane?:

@creayt said in Is this server strategy reckless and/or insane?:

Ideally more than that, but it'll be a gradual climb. Right now it's in private alpha w/ ~ 100 users and they post stuff all the time. Once I make it public I imagine the content volume will skyrocket.

Why not use Cloud/PaaS? There are some systems where you pay by the transaction so you're not out capital for hardware that will not scale where you need to go a long time, and you will not waste money on hardware if this project goes nowhere.

Pricing out equivalent horsepower on Amazon I think came to something like $50k a month, this whole set up cost me under $10k I believe. By the time I exhaust the capabilities of this hardware/investment, I hope, I'll be at the venture capital phase and and can redeploy into a fully cloud strategy, grinning shit-eatingly at how well that original $10k investment served me.

Will also mention that colocation where I live is a dirt-effing-cheap $55-per-U/month.

There are far cheaper IaaS providers than Amazon (I assume you are looking at EC2, when you should be looking at RDS if you're doing AWS). I'm partial to Softlayer these days, but to each their own.

Deploying and managing your own infrastructure for a startup is a nightmare as if/when your product "Blows up" and goes from 100, to 100K users it will implode and crash on the weekend before you can get new hardware in and scale it, or refactor for a platform with real scalability. If your worried about cloud lock-in use abstraction systems that allow for multi-cloud strategies (although honestly in the early phase I'd just accept the lockin as that's easier to refactor than trying to refactor the platform AND scale at the same time).

If you can't maintain growth and have large hiccups in engineering VC gets spooked easily.

Also If you're really looking to scale one thing is trying to limit your dependency on RDMS in general. 9/10 times I see a startup using one, they should have used object storage or a No-SQL system.

Then again, I'm just a Palo Alto Serf working for "the man" and not feeling the wind in my hair of founding the next big thing in the garage.

@matteo-nunziati said in Is this server strategy reckless and/or insane?:

This. I asked the reseller about this feature. They anwer: disable ssd cache anyway and use controller cache.
The latter former is a safer choice while the latter is too new/untested feature...

To be blunt, the reseller doesn't know what they are talking about. Every enterprise SSD in the modern era (using some sort of FTL) uses this design and has for years. They are configured this way even in big enterprise storage arrays with the unique exception of Pure Storage who re-writes their firmware to basically use drives as dumb NAND devices (and then has MASSIVE NVRAM buffers fronting the drives that do the same damn thing at a global level).

Some SDS systems want you to explicitly disable the front cache as it will coalesce data and prevent data proximity optimizations in the actual raw data placement. It also exists as yet another place that data can be lost or corrupted and for systems that want to "own" IO integrity end to end they want to know where stuff is.

Then again, what do I know...

@travisdh1 My job is to fly drink and talk primarily

@kooler HTML5 GUI's you say?

@travisdh1 said in Is this server strategy reckless and/or insane?:

You're job description includes talking to people on web forums now doesn't it? Also, when do you stop drinking?

Fly, Drink, Talk. There you go.

No, hanging out on web forums is not my job.
I actually didn't drink that much this weekend (was too hot, working on the beach house).

My day job involves...

1. Flying to conferences and speaking. I have 11 conference presentations in the next 4 weeks. Crowd size is 200-800.

2. Flying to fun places and meeting with people. I'll be in India soon meeting with Customers, Partners, and SE's training them and taking questions, and collecting feedback for engineering.

3. Breaking things. I technically am classified as a R&D employee and have full access to our nightly builds, our BAT private cloud, and a dozen "Fully loaded" servers for a lab. I test the new stuff, send feedback through my customer [0] Team, and meet with engineers to capture the subtitles of what's coming out. I don't write the technical publications (core documentation), but I do draft thousands upon thousands of words for design and sizing and usage guides, blogs.

4. I host a podcast for the lols.

@scottalanmiller said in Is Tintri Heading for Pure and Nutanix Territory Financially?:

Depends. Ones like Nutanix with zero plan, that makes no sense and that's why we are concerned here. Others, like Amazon, that are obviously and publicly trading profits today for market growth, but could be profitable at any moment if they wanted to be make total sense.

But...... they are a cloud software company who makes an HCI appliance that can BACKUP DATA TO THE PUBLIC CLOUD! OMGZ IT'S SO FORWARD CLOUD THINKING! CLOUD CLOUD CLOUD CLOUD! OHHH IT CAN CREATE A CONTAINER ON GOOGLE! OMGZ CLOUDS!

On a serious note, the cloud washing in this industry is awful when someone who makes a hypervisor that doesn't run in/on ANY public clouds can be considered a cloud company.

At large scale people tend to use host profiles or some sort of state management so they can just restore configurations.

Also note: ESXi has 2 boot banks so if one corrupts or doesn't install cleanly (to where it can't boot) it will roll back to the alternative one.

@scottalanmiller said in Malicious Logins To Zimbra Mail Server:

Correct, MTA is always on 25 unless you have an agreement with someone. Then it could be anything.

I'm a bigger fan of having an external service or device (that can mailbag) do your filtering, and then you only accept SMTP with TLS from that service (So your firewall rules don't allow port 25 from the world to the actually mail back end).