Never Let the Vendor Set Up a Server
-
@MattSpeller said:
Set RAID level
Install OS
Update OS
Install driversThese are all tasks that you must be competent in and must verify to reliably run servers. Even if a reseller can do this reliably and does it for free, I don't feel that you should ever allow them to do so in order to find out if they can or not. These are standard operational tasks. That the server is being freshly delivered is a very odd break in standard process to have that one special case where a third party does a one time, highly critical engineering setup on your behalf.
-
@scottalanmiller it's common to have that basic stuff included for cheaper than I could do it, or often free. Why would I use a vendor/reseller who wouldn't offer such a basic service? I mean it's not a make or break thing by any means, but certainly saves me a few hours of poke & wait.
-
@MattSpeller said:
@scottalanmiller it's common to have that basic stuff included for cheaper than I could do it, or often free. Why would I use a vendor/reseller who wouldn't offer such a basic service? I mean it's not a make or break thing by any means, but certainly saves me a few hours of poke & wait.
Because it is absolutely critical and needs to be double checked which is just as costly and time consuming as doing it yourself and does not provide the additional benefit of double checking that you have all necessary skills, documentation and resources in order to repeatably do this over and over again.
If you read the original article, I cover why you need to do this yourself and why it isn't connected to the vendor's ability to do this but to having them do it not being a good idea.
This is from first hand watching many companies get burned because they had servers delivered and just enough work done on them and then later, once the workloads are in production, find out that they don't know how to maintain them and are stuck with unsupportable systems (beyond the knowledge of their IT, lacking tools, etc.) in production.
-
@MattSpeller said:
@scottalanmiller it's common to have that basic stuff included for cheaper than I could do it, or often free.
Since it costs just as much (or more) to double check and is a less reliable process. Even free is too costly, right?
-
@MattSpeller said:
I mean it's not a make or break thing by any means, but certainly saves me a few hours of poke & wait.
How is this taking a few hours? A new server build in my experience is normally just a few minutes. If it takes any longer than that, what is happening and what's happening that you would want someone outside of IT doing it without IT insight?
-
At a larger scale I can see how you'd have a solid argument but for basic stuff, sure I check it but it's pretty trivial.
When was the last time you installed a bare metal OS? The updates alone, good grief. Granted you can do other stuff while it grinds, but.... ughh wasted time.
-
@MattSpeller said:
At a larger scale I can see how you'd have a solid argument but for basic stuff, sure I check it but it's pretty trivial.
But how trivial would it have been to do it yourself rather than checking? What are you gaining?
I think a normal server setup that I see is about fifteen minutes and saves time looking things up since you are setting it up in situ rather than elsewhere and reconfiguring the network once it is in place.
-
@scottalanmiller said:
But how trivial would it have been to do it yourself rather than checking? What are you gaining?
Several hours of company time that could be better used for surfing the web and chatting on ML
-
@MattSpeller said:
When was the last time you installed a bare metal OS? The updates alone, good grief. Granted you can do other stuff while it grinds, but.... ughh wasted time.
Long time, since that's a no no. Why are you installing a bare metal OS? I think this is one of those "one lack of best practice cascades to another" situations. Fix the bare metal install, suddenly the best practice of never let someone outside IT set up the server makes sense. Best practices in a vacuum rarely make sense.
-
@MattSpeller said:
Several hours of company time that could be better used for surfing the web and chatting on ML
But if it taking over fifteen minutes, maybe fixing that process would be best
Even a bare metal OS deploy should be pretty fast. I do OS deploys to VMs all of the time. Install itself takes seconds and the setup of it is a single script. I can do my entire portion from soup to nuts in about two to three minutes. Including setting up authentication, all patches, updates, security configuration, adding the system to monitoring and a final reboot.
Is your vendor doing that kind of stuff? If not, it seems like you are optimizing processes in different places. Put that all together and you might find that whole portions of your process need not exist at all.
-
For those who let vendors do basic setups....
- How and when is RAID selection made?
- Is the BIOS being updated?
- Is the firmware of the RAID controller being patched?
- Is the OOB being setup so that it just works when you plug it in?
- Are RAID cache balances being set?
- Is someone considering block and stripe sizes?
- For vendors like LSI (PERC, Dell, etc.) how do you deal with the RAID 100 configurations as they don't allow you to specify this?
- How are OS partitions, filesystems, and other install time settings being determined?
- Is networking set up for the OS, OOB, HV, etc. before it arrives?
- Who is dealing with making copies of physical boot media like SDs and USB?
- Is someone setting simple stuff in the BIOS like ID fields?
- Who is setting up things like authentication?
- How are custom packages being applied (GPO, script, etc.?)
-
@scottalanmiller said:
Even a bare metal OS deploy should be pretty fast.
Should be, but isn't - go grab yourself a DVD and have at it lol
Whaaaaaaaaaaaaaat?? I don't have ready to go OS images? No, no I don't because managing a small server fleet where I believe we have only 2? of the same model makes that slightly impractical
Yeah, maybe I could spend some time and make hardware agnostic sexy little deploys pre-loaded with updates and software and all the good stuff. Doubt it'd be a good use of the company's time, though I'd enjoy learning more about it.Again, for your scale, I completely agree with you - it'd be pretty stupid not to do your own from the ground up. For SMB (emphasis on S I suppose) it makes a ton of sense and saves quite a bit of time to have them do the brain-dead basics for me. It's been a while since I did one but I recall it taking more than half a day between raw install, drivers & downloading / installing updates.
-
@MattSpeller said:
No, no I don't because managing a small server fleet where I believe we have only 2?
Again, only because you are installing physically. Once you install to a hypervisor, the abstraction layer means that you install the same OS, repeatably, every time. Best practices are best practices for a reason. Often the big benefits are in areas we never talk about because they are such clear wins that no one talks about those key benefits anymore.
-
@MattSpeller said:
Yeah, maybe I could spend some time and make hardware agnostic sexy little deploys pre-loaded with updates and software and all the good stuff. Doubt it'd be a good use of the company's time, though I'd enjoy learning more about it.
How could it be a waste of time? It must save time by the second deployment at the kind of time that it is taking for you to do deploys. It might actually save time on the very first deployment, although that is unlikely.
-
@MattSpeller said:
Again, for your scale, I completely agree with you....
Again, I do all sizes. I would do this 100% of the time for a company of a single machine or even for home use. I don't see scale as being a factor. Good practices cost less to implement in small environments. The impact of little mistakes is often magnified, though. It's trivial at scale to move workloads around to fix hardware configuration errors. Not so much in the SMB. And if you are putting OSes on bare metal, magnified many times again.
-
@MattSpeller said:
For SMB (emphasis on S I suppose) it makes a ton of sense and saves quite a bit of time to have them do the brain-dead basics for me. It's been a while since I did one but I recall it taking more than half a day between raw install, drivers & downloading / installing updates.
But if you don't create the need for time savings, you can save time and get the protection.
-
Why the hell would he waste timeless bing how to script a server install he is going to do twice? The next time he installs a server it will be a different OS and he would have to take all the time to update the script for negligible benefit. There is also the poor here that as you need to know your hardware is configure right, you also need to know how to setup ye IS correctly manually before you can correctly write a script. That cuts into the savings even more.
You are completely missing the point on that Scott.
It has nothing to do with being bare metal or virtual. It ha to do with quantity.
I do happen to agree that you should always configure the bare metal yourself though. That is a different subject than scripting the server installs.
-
@JaredBusch said:
Why the hell would he waste timeless bing how to script a server install he is going to do twice?
He shouldn't, but how is he getting the vendor to do this if he isn't providing that script (human instruction list) to them? If there is time to be saved, you can save it in either place.
And my scripting was definitely easy enough that it was worth it for two boxes. Any additional ones or rebuilds.... those are freebies.
-
@JaredBusch said:
There is also the poor here that as you need to know your hardware is configure right, you also need to know how to setup ye IS correctly manually before you can correctly write a script. That cuts into the savings even more.
That's fine, but you need that before telling the vendor what to do, right? And you need to know that stuff for documentation, decision making, being prepared, etc.
My point here is not that everything should be scripted, I'm offering that as a way to fix issues that he seems to be having that he feels is making him have to have someone else do manual work that only takes a few minutes of manual effort (even updates take no human time, just lots of download time and mostly only if things like WSUS are not set up) but that having the vendor do these things is not reliably safe or effective. The scripting is ancillary.
-
@JaredBusch said:
It has nothing to do with being bare metal or virtual. It ha to do with quantity.
It should not. If I'm doing an install of one server or a million, I still want virtualization for reliability and safety, portability, stability and speed of installs, changes, etc. And the more that I rely completely on a single piece of hardware, the more that I need to be concerned that the hardware and HV/OS are set up correctly, right? Because the smaller that your environment is, the bigger the impact of a problem like that.
