@bigbear said in How to Grow from a One Man Operation to Two:

@scottalanmiller said in How to Grow from a One Man Operation to Two:

@bigbear said in How to Grow from a One Man Operation to Two:

A small "MSP" has the chances a rock band has of both sustainability let alone large-scale success.

Its got me thinking, and maybe a good topic for another thread, what is the "hot business" now that is accessible for a small startup. Something where customers are actually seeking out solutions the way small offices needed a server, email and network help in the late 90's and early 2000's?

If we only knew.

In the context of starting from scratch and ascertaining new accounts in 2017. Incumbent businesses with existing revenues and clients have a 20x advantage.

If your looking for "hot jobs"
SRE, Machine Learning, and general development are all doing quite well.
In infastructure NSX people are hard to find and pay decent (~200K+).
Operations people who know enough automation programming and scripting to do Devops (DevOps is about ops people becoming devleopers not the other way around).

On the topic of advantages of large shops vs. small it's pretty huge.

You are also competing against guys with DEEP skill benches. Not only can they bill more per hour, but they have guys who can do what you do in 1/4 (or less) the time.

New site deployed? I would quote the labor at flat fee 2 hours $400 knowing it would take the engineer maybe 15 minutes to copy paste an existing DMVPN config he had done.

Larger shops also get significantly better pricing on MSP tools (RMM, and PSA pricing models skew towards heavy discounts once you have 10,50,100 employees or thousands of sites being monitored). I've seen 80-90% off list for software.

Large shops have benefits that just can't get matched.

Going from a small employer who's health insurance was so bad it was cheaper out of pocket to a medium sized shop where it was all free was huge. There are guys with chronic conditions (Type 1 Diabetes, Cancer survivors) who historically couldn't work for a small business because of the out of pocket insurance costs.

Going from a medium to a larger shop can double your compensation. Even if you pay them the same in cash you are likely not to offer:

ESPP - Variable income between ~3K and 30K depending on how the stock is doing. Can pay employees at long-term capital gains with this to cut their tax bill in half.

RSUs - Variable income, that has special tax treatment if held. Can pay employees at long-term capital gains with this to cut their tax bill in half. It's a golden handcuff but the employee can leverage this into a larger signing bonus at their next hop.

Bonus - 5 figure performance bonuses and bonuses paid out more than once a year are nice.

401K with match, and low carry costs. Note the last one can be a legal liability if you screw up.

Education (Cover full costs of an Masters/MBA, any certification or training they want and send them to 1-2 conferences a year.

Expense/travel reimbursement policy - Small shops will balk at someone spending $30 on dinner. Large shops will not care about a $100 lunch. Small shops will force lowest cost fair rules on travel, and force discount carriers, and tickets. Larger shops will allow business class on long flights.

Equipment/other misc - Small shops will give you a 10-year-old compaq laptop. Larger shops will happily give you a $3K XPS or Macbook pro. Cell phone reimbursements and other fun things also add up. Small shops have to count pennies....

@irj said in How to Grow from a One Man Operation to Two:

Zero experience means you have to ask about the home lab. If they really have an interest in something they will probably be doing it in their free time. If they have zero professional experience and no lab experience, I wouldn't be interested either.

So if they have a home lab and professional experience , you're gonna still pay minimum wage?

I paid a guy who was doing part-time desktop support at IKEA, and had a 5 router home lab, with a VMware cluster ~38K to hire. At 90 day I moved him up to 56K. He got brisk (Sometimes twice a year raises) and was over 100K in under 3 years. I could also bill his time at 200-250 an hr (and had work backed up for weeks sometimes for him. It was clients requesting him specifically for him). I got lucky and found someone who had a curiosity and capability but that is useless without the work for him to grow into. I was constantly throwing him into projects that were above what he had done. Do you have that kind of work available? I hired away a lot of guys from Small MSP's who had guys with VCP/CCNA's skill ranges fixing printers, and desktops. The initial cost to hire them was never that bad (45-70K) but it was the upside. of growth to 6 figures (and the skills to demand even more) that brought them on. I hired one guy from a low end MSP for 44K OTE, and ended up having to give him a raise to 70K in under 2 years. He got bored quickly and left (he's running NSX/vRA automation deployments for an airline for ~180K OTE now). If you can't keep the work to keep someone interested it can be more disruptive onboarding, and offboarding them than it's worth. I tried to shoot for ~2 years tenure as my value target.

@mike-davis said in How to Grow from a One Man Operation to Two:

@storageninja said in How to Grow from a One Man Operation to Two:

I never hired guys with zero experience, because they cost me more than they made me vs. the premium to pay someone for 40 hours and benefits who was... useful and I could bill at $120-140 an hour.

Zero experience means you have to ask about the home lab. If they really have an interest in something they will probably be doing it in their free time. If they have zero professional experience and no lab experience, I wouldn't be interested either.

Home lab worked for networking (WAN networking, and MPLS, and OSPF scale well from labs to enterprise shops). There's a lot more clearly defined best practices in networking, than in the systems world.

On the systems side, it's not the same. My shop didn't do desktop support, so the traditional low-end windows stuff didn't exist to work people up. Our lowest tier was doing VDI support (clustered storage, hypervisors, GPO in environments with 8000 users). There are people who run VDI at their house and run NSX for security services but they are few and far between and they certainly aren't cheap.

@bigbear said in How to Grow from a One Man Operation to Two:

I would think first hire to be low end, entry level looking to make his way. 30 hours per week and someone who is flexible to work less hours or more.

The problem is it's a VERY narrow spectrum between "So low cost, and so low skilled that he generates more clean up work, and bill credits than he is worth" and "valuable enough that he takes clients and runs, or get someone to pay him 40 hours and a proper salary" You are danging on a knife edge, and given you lack proper hiring process's, back ground checks, and other procedures the odds are you'll screw up somewhere...

I never hired guys with zero experience, because they cost me more than they made me vs. the premium to pay someone for 40 hours and benefits who was... useful and I could bill at $120-140 an hour.

@scottalanmiller said in How to Grow from a One Man Operation to Two:

There is no organic path from one to two people in the MSP game

I would never do it. 50/50 ownership structures or 49/51% tend to implode poorly.

@scottalanmiller said in What are the highest paying IT careers?:

Because they are not IT careers

They are careers that require IT skills (despite not being directly in the IO path), and are attainable by IT staff.

@mike-davis said in What are the highest paying IT careers?:

@storageninja said in What are the highest paying IT careers?:

Get to know employees of the company you want to work for. I was referred by the chief technologist in the BU. That likely went a ways towards making sure I got a call back.
I wrote some blogs that he liked, and met him at a conference. Go to conferences.

good advice here.

Here's his (the guy who referred me) advice on how to advance your career.
http://www.yellow-bricks.com/2015/02/24/how-do-i-get-to-the-next-level/
http://www.yellow-bricks.com/2016/05/26/get-next-level-part-2/

@networknerd said in What are the highest paying IT careers?:

I was wondering why no one mentioned Pre-Sales or Technical Marketing until now. And there's also Product Management too, but I'm not sure if many sysadmins can make that leap.

Pre-Sales is easy (most are former Sysadmins), Technical Marketing (there's just not a lot of positions in the industry open, but starting pay (See link for a lower level position) isn't that bad. Note vendor to vendor the "technical" and corresponding pay can vary quite a bit. For a lot of companies, this is just a "mildly more tech-focused" marketing while for others they want high levels of technical capability.

Product marketing is something that some can make the leap (Allen Renouf is a good example) but many are MBA, or ex-engineers.

@travisdh1 said in What are the highest paying IT careers?:

@scottalanmiller said in What are the highest paying IT careers?:

If you'd like a local address, I have a house for sale, though!

Yeah, and with no income I doubt I'd be able to buy a house, even a dirt cheap house.

Sigh, NiNJa (No income No Job) Loans are not a thing anymore...

@travisdh1 said in What are the highest paying IT careers?:

@scottalanmiller said in What are the highest paying IT careers?:

@jmoore said in What are the highest paying IT careers?:

@tim_g said in What are the highest paying IT careers?:

If one were to search those types of jobs on indeed for example,

It could just be me but I am thinking the jobs on Indeed are on the verge of fraudulent. There are so many in the tech fields that seem unrealistic. Again it could just be me and when I was looking, but there were many IT jobs that wanted at least 5 years experience in a networking, system admins, sql expert and so on with 5 years experience in each category.

Each job was different but it was some combination of skills that just seemed unrealistic such as jobs with 15 years experince in both Linux and Windows admins. When I would try to contact about these jobs to see if they would entertain a person with less experience I could never get in touch with anyone, ever. I'll never use Indeed again because I think many or most of those IT jobs are just fraudulent but maybe thats just my experience.

Remember that I said 85% of job listings anywhere are fake .... and that was back when people had to pay to post them!

Do you know of any good headhunters accepting new people? Some of us could really use good help.

We don’t use headhunters at our company and everything is posted on our jobs site. (VMware.jobs)
I’ve posted my referral link before for gigs. Something like 2/3 of our hires come internal references.

General advise is:

Get to know employees of the company you want to work for. I was referred by the chief technologist in the BU. That likely went a ways towards making sure I got a call back.

I wrote some blogs that he liked, and met him at a conference. Go to conferences.

We hire all over the place for Field SEs. We have to staff weird regions or accounts (Like Kansas City) all the time.

We hire for PSO all the time. Be willing to fly anywhere and travel 80% and you can get a job.

@scottalanmiller said in What are the highest paying IT careers?:

@tim_g said in What are the highest paying IT careers?:

@scottalanmiller said in What are the highest paying IT careers?:

@tim_g said in What are the highest paying IT careers?:

@wrx7m said in What are the highest paying IT careers?:

@scottalanmiller said in What are the highest paying IT careers?:

@wrx7m said in What are the highest paying IT careers?:

@scottalanmiller said in What are the highest paying IT careers?:

SMB is all the same two jobs.... generalist or generalist manager. All comes down to seniority, experience, market, etc.

This is where I am, currently. What do you think national average is?

No idea about national average, nor would it likely be meaningful in any way. Pay is so dependent on the job, job role, company, level, and location that an average would be kind of meaningless. Imagine asking "what's the average pay of a business person" or "how much can you expect to earn as a business owner?" An average in anything like those would be totally meaningless.

Hmm... Oh well. I hit 6 figures this year.

That doesn't seem like an easy thing to do in the the typical SMB as a regular generalist employee.

Actually SMB is one of the highest paying categories. But it is misleading. It's larger and very focused SMBs that might be small in people, but big in profits. They tend to behave like enterprises, even with relatively few people. Nearly all $300K+ jobs I know of are in the SMB range. Whereas the enterprise tends to dominate the $200K+ range.

These must be the F100 SMBs then, because I've never seen anything anywhere close to that in real life... not personally and not advertised.

If one were to search those types of jobs on indeed for example, you will not fine that role and that pay, nowhere close. Maybe you do, because you know someone who knows someone.... but I'd never find an offer like that, even if I'm worth a billion dollar salary.

That's because high end roles aren't advertised You work through headhunters. People who place people at that level, have them vetted, and such.

Many of them are SaaS providers for niche industries. You can have 20 employees running a platform with 50K customers. Tech heavy jobs. As Scott mentioned if you lack the skills (and don’t run in the right networks) you’ll never find out about them.

I work in storage which while a relative large field, still feels small when you get to the 200K+ tier.

@dashrender said in The argument for official support vs third party support:

Same problem for the vendors. If you are dealing with unpatched spares, so are they. Having worked for some of the big ones, I know that their supply chains struggle to get parts, too. Heck, IBM couldn't deliver a server internally in more than six months, imagine how hard it is to get support parts!

Shit like this just blows my mind.

Parts Bins, internal supplies for labs, and customer supply chains are all completely different (well IBM may have been a gong show). Dell and HPE staff can't just go grab something off the line, with Mfg you have to account for the costs and someone gets to pay (and often at a premium to prevent abuse) for those internal servers.

Parts Bins and stocking those are different, and supply chain for a OEM might actually be different in the us than EMEA.

@scottalanmiller said in The argument for official support vs third party support:

Same problem for the vendors. If you are dealing with unpatched spares, so are they. Having worked for some of the big ones, I know that their supply chains struggle to get parts, too. Heck, IBM couldn't deliver a server internally in more than six months, imagine how hard it is to get support parts!

Unpatched spares are supposed to be handled by the support staff, but that gets overlooked way too often.

We've actually been building life cycle tools into the hypervisor to mitigate this (because expecting SMB's to deploy HP One View etc, is a non-starter. ESXi can patch RAID controllers, and SAS HBA's today with more coming in the future.

@scottalanmiller said in The argument for official support vs third party support:

Commodity parts, get one that works and move on. Paying for unique support and code changes to support a single SMB use case rarely works. As Dustin points out, I can swap the part in an hour. I can't get support to understand what the issue is in that amount of time. Vendor support, in that example, represents high cost and high risk. I know that I can swap parts and that I can swap them now. I don't know if the vendor can fix the issue or if they will in any reasonable amount of time.

This only works if you keep non-matching spares. So For every LSI based controller you keep a similar Adaptec (and are ready to do a swing migration). For every Intel NIC have a QLogic, for every Intel Flash drive have a HGST one.

I watched (larger server OEM) techs replace a back plane twice and following that we got routed to the right team who got us a hotpatch for a SAS expander.

What's more fun with server OEM's is while the back end vendor may have identified the issue (say Broadcom), they run different code trains, and they may not check out that fix from the main OEM's trunk (for fear it causes other issues) until a customer reports an issue. Because of the hell of properly regression testing hardware this is how a lot of stuff works.

@scottalanmiller said in The argument for official support vs third party support:

Same with a hard drive. In a lot of cases, having parts available is far more reliable, faster, and cheaper than a warranty from a vendor.

The quality of firmware and drivers on flash devices is a lot more all over the place than magnetic drives. Having 14 drives that implode under a burst of writes isn't that helpful.

@scottalanmiller said in The argument for official support vs third party support:

@wrx7m said in The argument for official support vs third party support:

@scottalanmiller said in The argument for official support vs third party support:

@wrx7m said in The argument for official support vs third party support:

@scottalanmiller It is the value of time spent troubleshooting vs time spent replacing. Sometimes it makes more sense to replace something than try and figure out what it is. If I spend 2 hours trying to figure something out on a $100 printer, I should have just replaced it.

Yup, which is often the case with operating systems, too!

Exactly. A re-image solves almost everything.

Yup, the DevOps philosphy. Rebuilt to know good rather than trying to troubleshoot the unknown.

This is fun until it involves hardware. Ever try to downgrade a firmware on a NIC or SSD? For some vendors this is technically impossible without shipping it back....

@olivier said in Xenserver and Storage:

And it's second time I heard about the "intelligent split brain" management on StarWind but didn't see any paper nor a start of explanation about how it works (nor even a simple link). Can you elaborate please? If it's the witness node, it's the classical thing, but I'm curious about the split brain protection without using a witness node.

My understanding is they can do multiple links, multiple heartbeats,

Or a discrete and Stateful witness service on a 3rd system that will completely solve the problem.

VMware vSAN prevents this on 2 node and stretched clsutering by keeping witness components with sequence numbers on the witness system. In a vote is called the one has a updated sequence number matching the winner that side wins. In the event a stretched cluster partitions and both have matching sequence numbers the "Primary" side wins.

I'd argue isolation behavior goes beyond the storage heartbeat to how isolation is handled at the VM and Hypervisor level. STNITH is kind of a barbaric way ot handle this in 2017.

Other fencing systems that exist in VMware are for HA. Pings between hosts (Default on management network, moved to vSAN network if in use) Isolation address's (can have multiple) and heartbeats through datastore heartbeats (a file that is updated) for non-vSAN datatsore's. Based on this you can configure different VM and host isolation responses (maintain power, power off, shut down etc).

@olivier said in Xenserver and Storage:

That's why I asked if you have better knowledge of community on this solution because I really don't. So if it's the case, that it's not stable (darn, it's here since a long time!), that's indeed not an option.

Issue with HA lizzard is that it doesn't have a stateful quorum system (just pinging a single IP address). You can split brain it.

@scottalanmiller said in The argument for official support vs third party support:

That was my point, they offer it AFTER you find a bug and they can decline it AFTER you are stuck with them

Not all companies subscribe to the first rule of acquisition. Gint was wise, but software vendors tend to not be that bad. I have seen this happen, but it tends to be with vendors who are more concerned about their next customer than their existing ones for revenue. Your large enterprise vendors (Redhat, Juniper, HDS, Oracle, VMware) tend to not do this. Startups and hardware vendors tend to be the worst offenders (Hardware because it can cost them a lot for a recall like the Intel Clock issue that impacted the Atoms), and startups because they may realize that their first 100 customers are not going to be the kind of customers or industry that gets them to 10K customers and they will "Pivot" and abandon a field.

The reality is depending on the platform the labor to migrate (or to account for missing capabilities) isn't free. I've seen people replatform their monitoring system every 3 months. This just isn't a good idea as the value of the monitoring solution tends to come from the time invested in customization and training etc.

What's your reaction if you hit a buggy driver from Intel. Buy a different NIC, or call the vendor to see if you can get a fix or a workaround for it? That I guess fundamentally determines if you're the kind of person who buys vendors support.

@scottalanmiller said in The argument for official support vs third party support:

Microsoft doesn't even offer support in the SMB

They offer pay per ticket support. It's questionable the quality, but it does exist and if you find a bug they give you your money back (We rarely paid for tickets, but sadly it took weeks to get resolution sometimes).

The issue of paid vendor support isn't that a bug that's critical will not get fixed, it's just it may not get fixed in the timetable you need. Paying for expediency shouldn't be that shocking. We had a Brocade switch that we found a nasty RSTP bug in. We had a hotfix the same day. I found similar bugs in a Linksys switch. I never saw a resolution on it (They just didn't care). A Red Escalation (Priority 1 bug) from a paying customer will trigger engineers to work weekends and get overtime pay. A report from someone in a forum having an issue will get a "send me the logs and I'll look at it when I can" from most vendors.