Never Let the Vendor Set Up a Server


  • Service Provider

    When I order a car, I don't expect the dealer to have my mirrors adjusted or the seat in the right place. When I get a new laptop or desktop, my first task is to install a fresh copy of the OS. You don't want your vendors making decisions for you about these things, you need them to make a good product and deliver it to you and provide support when things fail or provide updates and patches, but actually setting up and operating the devices are things that you do yourself and require a certain about of knowledge and effort to do well.

    When it comes to servers, making sure that all of the parts are understood and ensuring that the setup process is correct and repeatable is very important, far more important than with something like a desktop.

    Because we need it to be repeatable. Should we need to rebuild the server, build another to match it, repurpose it or whatever - knowing that the process of setting up the server can be done, reliably, is very important. If you always have your vendor set up the server for you, you have no means of ensuring that you have access to the necessary tools, installation media, drivers, knowledge of what to do, an understanding of what has been done and have produced appropriate documentation.

    Because it needs to be right. Even if the vendor does the work for you, which is not very much work anyway, you need to double check everything. It is literally less work, in nearly all cases, to do the work yourself rather than to attempt to double check everything that someone else has done.

    Because it requires less faith. Some big vendors have already been caught using "pre build" processes to put spying software, bloatware and malware onto customer systems. Skipping the "vendor do my work for me" step reduces the risk that this can happen and increases your opportunities to catch it if it does.

    Because vendors don't offer every option. Commonly only limited RAID options are presented for pre-configuration and more details selections may be needed for the real world setup.

    Because it creates complicated licensing situations. Your vendor does not have knowledge of or access to your licenses so having the vendor do anything that involved licensed products can make things more complicated.

    Because vendors are not experts. Server vendors are manufacturers, not IT companies. They lack the skills, knowledge, mandate or business reason to understand what is needed by the customer. This is not their wheelhouse and there is no reason to think that they should know what configuration is good for a customer.

    Because vendors are not knowledgeable of your business. Server vendors simply don't know your needs and factors and cannot possibly know what would be right for you without you providing them so much detail that there was no value to having them do this work anyway.

    A server is a big investment and an important piece of infrastructure for any company. Hoping to skip a few minutes of very simple, basic work like setting BIOS settings, setting up the RAID and installing the hypervisor buys you essentially nothing but adds a lot of risk that is very unlikely to rear its ugly head until months or years down the road.

    Take the small effort to do things right at the beginning and protect yourself and your business from surprises down the road.



  • What risk?

    My time and knowledge is limited and the more I can safely outsource the better. Anything involving a screwdriver I'm not interested in and I have no problem getting someone to setup the RAID and install the hypervisor. As for BIOS settings, I wouldn't even know what the settings are supposed to be.

    Comparing installing a hypervisor to adjusting the mirrors on a car is quite frankly ridiculous :)
    .


  • Service Provider

    @Carnival-Boy said:

    What risk?

    As an example that comes up over and over again - the vendor does not properly set up RAID and gives the companies a system that has no RAID, or RAID 0 or just RAID different than what was expected. Or an OS that is improperly installed. The RAID issue is the most common. Wrote this article, which I've been meaning to write for a while, after talking to someone who discovered after years of running a system with no RAID and their only reason for believing that they might have had some was that they looked at a paper invoice that said that that is was how it was purchased. Had they looked at the RAID controller, they would have known instantly that things were very wrong.

    Additional risk is that when things do go wrong, the staff is not familiar with the hardware that they have in production and do not know how to work on it.



  • @Carnival-Boy said:

    Anything involving a screwdriver I'm not interested in and I have no problem getting someone to setup the RAID and install the hypervisor. As for BIOS settings, I wouldn't even know what the settings are supposed to be.

    What does setting up a RAID involve a screw driver. Sure I get someone else to do stuff with screwdrivers. Someone else can rack it and any repair would be outsourced but, I'm not sure how raid setup would fall under that.

    I'm not sure how IT could properly administer a server if they didn't understand how the underlying hypervisor and server should be configured.



  • @Carnival-Boy said:

    Comparing installing a hypervisor to adjusting the mirrors on a car is quite frankly ridiculous

    There are many settings to be tweaked for the "tastes" of each company and environment.


  • Service Provider

    @Carnival-Boy said:

    My time and knowledge is limited and the more I can safely outsource the better. Anything involving a screwdriver I'm not interested in and I have no problem getting someone to setup the RAID and install the hypervisor.

    There is a difference between having your IT outsourced (IT department is external) and having a vendor ship gear as a one time setup. In the one case it is still your IT department who are knowledgeable as to the needs of your business, double checking things before systems go into production are correct, etc. Having a factory do the IT work and hope for the best is dangerous in a different way. There is no double check, there is no goal level understanding so they can't tell when things are wrong. And when you need to change the setup and do other things, patch and whatever you need more information and skills than the initial setup.


  • Service Provider

    @thecreativeone91 said:

    What does setting up a RAID involve a screw driver. Sure I get someone else to do stuff with screwdrivers. Someone else can rack it and any repair would be outsourced but, I'm not sure how raid setup would fall under that.

    I've seen it go both ways where the "screwdriver guys" get assigned doing the RAID setup. But it is such a critical IT task that it is very hard to believe that anyone would let non-IT people be responsible for storage decisions that even many seasoned IT pros are not very familiar with. Unless you have a stock, repeatable process and good processes for checking up (which mostly means the effort of having someone else do it is wasted) it would seem pretty dangerous.


  • Service Provider

    @Carnival-Boy said:

    Comparing installing a hypervisor to adjusting the mirrors on a car is quite frankly ridiculous :)

    The point is that personal customizations that are part of ongoing operations need to be done by the operator, not the factory. To run a server, you need to understand the hypervisor and maintain it. If you drive a car, you need to operate and tweak the mirrors. Both are trivial tasks but ones that need to be done.


  • Service Provider

    @Carnival-Boy said:

    Anything involving a screwdriver I'm not interested in ...

    I don't know many full time IT people who do bench work. Bench work does not involve critical business decision making and certainly can be outsourced. Replacing a processor on your server or a hard drive is a purely repeatable, scriptable, mechanical operation. It does not even require knowing what a computer is to do well (although that presumably helps.) That's completely different than what we are discussing here. Obviously the vendor will build the server, no one is suggesting that you assemble your own (not that whiteboxing can't be done, it's just not being suggested.) It's the IT tasks of operating the computer that need to be handled by IT (in house or out sourced) rather than manufacturing vendors.



  • You mean like getting HP to setup the server and ship it from their factory pre-configured? I don't do that. I buy from a small IT company, HP deliver it direct to my site and the small IT company send someone to rack it up, install any additional components (the screwdriver bit), setup the RAID and install the hypervisor.


  • Service Provider

    @Carnival-Boy said:

    You mean like getting HP to setup the server and ship it from their factory pre-configured?

    Correct. HP does not push this as much but you can do it. Dell pushes this hard. They force you to "set it up" on the web GUI when ordering the machine. Tons of people do this and ignore all of this basic setup when they receive the box and just "use it" however it arrives. This often leaves them not just with a lack of knowing how it was set up (as they often seem to come wrong) but also not knowing how to check it, change it or operate it.


  • Service Provider

    @Carnival-Boy said:

    I don't do that. I buy from a small IT company, HP deliver it direct to my site and the small IT company send someone to rack it up, install any additional components (the screwdriver bit), setup the RAID and install the hypervisor.

    Why does the person who racks and stacks it (entry level bench work) also do the RAID and hypervisor work (IT?) Doesn't this make the cost of the racker too high? Setting up the OOB I could see just because it's trivial and is what grants the IT team access. But who operates the RAID and hypervisor, handles patching and monitoring and that stuff once the server is in place? Is it that same small company (and you are just paying a premium for rack and stack) or is it another group?



  • All my vendors charge the same day rate regardless of who is doing the work or what the work is.

    I generally handle everything once the server is in place, but I have a support contract in place for third-line support (ie anything that is beyond my fairly limited knowledge).

    Isn't the issue here that Dell set the servers up wrong? Or that someone has failed to check that the RAID config they ordered hadn't actually been setup? These seem like pretty basic mistakes that shouldn't have been allowed to happen. A better thread title might be "Never Let Dell Set Up a Server"



  • @Carnival-Boy said:

    All my vendors charge the same day rate regardless of who is doing the work or what the work is.

    So they charge the same for racking a server as they would to setup exchange do network configs? Seems like they are ripping you off.



  • @Carnival-Boy said:

    Isn't the issue here that Dell set the servers up wrong?

    It isn't just dell it's most manufacturers. Its all manual, they look at what you request on the paper and set it up. Lots of mistakes will be made. There's no automated imaging and checking process like there would be for a desktop to check it has the correct version of windows and even the automated process to check that the CTO order was done right. (it's easy to tell a automated process to look for hardware, the custom settings and configs not so much.
    )



  • @thecreativeone91 said:

    @Carnival-Boy said:

    All my vendors charge the same day rate regardless of who is doing the work or what the work is.

    So they charge the same for racking a server as they would to setup exchange do network configs? Seems like they are ripping you off.

    Or under-charging me for Exchange ;)



  • @scottalanmiller said:

    @Carnival-Boy said:

    Comparing installing a hypervisor to adjusting the mirrors on a car is quite frankly ridiculous :)

    The point is that personal customizations that are part of ongoing operations need to be done by the operator, not the factory. To run a server, you need to understand the hypervisor and maintain it. If you drive a car, you need to operate and tweak the mirrors. Both are trivial tasks but ones that need to be done.

    True, if you're not the one who understands the hypervisor for your company, then the vendor you bring in to install/configure and manage the hypervisor should be the one setting it up as well.



  • Ideally I want a vendor that I can trust to set it up right. I don't know of such a vendor. Or rather I don't know of a vendor that I trust, rightly or wrongly. So I end up doing loads myself and getting stressed by long hours and lack of progress :(



  • @Carnival-Boy said:

    Ideally I want a vendor that I can trust to set it up right. I don't know of such a vendor. Or rather I don't know of a vendor that I trust, rightly or wrongly. So I end up doing loads myself and getting stressed by long hours and lack of progress :(

    It really sounds like you use vendors to do a lot of the IT work for you. Why not partner with a GOOD MSP?



  • I don't know a good MSP.



  • @Carnival-Boy said:

    I don't know a good MSP.

    You might have to find someone who isn't local for that I guess - though non local won't be of much help when it comes to turning screws or mounting servers.



  • I'm recruiting at the moment, so that should help. I don't trust outsiders that much. My old boss said I was a control freak, but I think that is slightly unfair.



  • @Carnival-Boy said:

    I don't trust outsiders that much.

    It is borderline paranoia that you don't trust outsiders much... But I find borderline paranoia a relatively safe place to be some times...



  • @dafyre said:

    @Carnival-Boy said:

    I don't trust outsiders that much.

    It is borderline paranoia that you don't trust outsiders much... But I find borderline paranoia a relatively safe place to be some times...

    Really? Why would you trust people you don't know? Granted you do at some point have to move from the non-trust to the trust zone, and that normally starts, in situations like this, are with a contract in place.


  • Service Provider

    @Carnival-Boy said:

    Isn't the issue here that Dell set the servers up wrong? Or that someone has failed to check that the RAID config they ordered hadn't actually been setup? These seem like pretty basic mistakes that shouldn't have been allowed to happen. A better thread title might be "Never Let Dell Set Up a Server"

    There are many issues here. Including that Dell (or whoever) doesn't offer the needed configurations, that they then get it wrong, that someone doesn't check, that there is no process for repeatability, there is no verification of knowledge, documentation and supplies, etc.

    There is nothing here specific to Dell. The mistake is in basically skipping the step of setting up a new machine upon delivery and just using it "as it arrives." You wouldn't do that with a car or a laptop, why with an enterprise server that lots of things depend on?


  • Service Provider

    @thecreativeone91 said:

    @Carnival-Boy said:

    All my vendors charge the same day rate regardless of who is doing the work or what the work is.

    So they charge the same for racking a server as they would to setup exchange do network configs? Seems like they are ripping you off.

    I don't see how else that would work. If you only need a minute of racking and hundreds of hours of other stuff sure, it's not worth the effort to bring in other people. But if you have any amount of this happening there is no way to do this without being ripped off. You are stuck paying engineering rates for physical bench work.


  • Service Provider

    @Carnival-Boy said:

    Or under-charging me for Exchange ;)

    Possible, of course, but that just means that either they are not financially viable and while you are getting a deal, it won't last and you will have churn. More likely it means you are getting unskilled screwdriver guys that you are paying for Exchange work.


  • Service Provider

    @Dashrender said:

    True, if you're not the one who understands the hypervisor for your company, then the vendor you bring in to install/configure and manage the hypervisor should be the one setting it up as well.

    Right, it's still IT that needs to do it. Just your IT or that portion of your IT is outsourced. Nothing wrong there. That's how I advise it for most SMBs. When I say "your IT" I mean to imply that this likely means "your MSP / outsourcer."


  • Service Provider

    @Carnival-Boy said:

    I don't know a good MSP.

    Two secrets to that that we've discussed a lot...

    1. Never look locally. Don't avoid local, but don't choose based on it. The chances that the good vendor is local to you is near zero. Places like Manhattan and London have better chances than most places, but locality is not a valuable discriminating factor for IT services.

    2. Don't use resellers for IT support or decision making because their interests do not align with yours.

    There are tons of bad MSPs out there. There are tons of good ones too. There is no magic to finding a good one. But there are easy ways that people often practically force themselves into bad ones.


  • Service Provider

    @Dashrender said:

    You might have to find someone who isn't local for that I guess - though non local won't be of much help when it comes to turning screws or mounting servers.

    Sure they are. As that part is not IT work but bench work, it is absolutely trivial for distant MSPs to bring on low cost remote hands to do that work. There is no IT skills needed there to do rack and stack and only typical home user skills needed to set up things like ILO and iDRAC. That's all that that role needs (or should) be doing.

    Not only does this work, it often actually helps. Having too much skill level in your data center rackers can lead to them attempting to do work that really needs to be done by someone with clear IT oversight.


Log in to reply
 

Looks like your connection to MangoLassi was lost, please wait while we try to reconnect.