Best posts made by StorageNinja

@scottalanmiller It's OK I got this!

Gentoo - The distro for people who like to beef up 15 year old Honda Civics with fancy lights because blue makes them go faster.

Ubuntu - The distro for people who like artisanal hand crafted coffee and cool backgrounds with each release.

Redhat/CentOS - For Grown ups who just want shit to work.
SuSE - Same as Redhat but for German's and Austrians for some reason.

FreeBSD - For the paranoid
NetBSD - For your weird tin foil hat neighbor who wants to run something on his toaster.

Windows 2000-2008R2 - For someone who likes to click next a lot
Windows 2012R2-2016 - For the people who like writing the longest possible CLI commands (Seriously Powershell!)

MacOSX 1985 - 2009 - HIppies. Damn Dirty Hippies.

Mac OS X 2010 - current - Network and Unix administrators who hated putty, and wanted something as stable as their server to run as their desktop OS.

Linux on Desktop - For masochists who somehow lack my burning hatred for SystemD.

@DustinB3403 said in At Least 30,000 U.S. Organizations Newly Hacked Via Holes in Microsoft’s Exchange:

At a prior position they went full tilt "O365/SSO everything" and while it all worked with a LOT of effort the monthly cost was insane per user, something like $42/U/Month for just our 1 location of 160 people.
Globally they had over 9000, that's a huge burden.

Except it's not.

1. It's opex not capex, so it's not dragging down RIOC ratio's for wall street. (big in Mfg and some industries).
2. It's just dumped into the fully burdened cost of an employee. If your average employee is paid 50K they probably cost another 20K in benefits, training, taxes, office space, utilities and other overhead a year. Paying $42 a user per month at that scale gets you out of:
3. "owning" versions of Office Suite is great until you end up with 4 different versions of office in the office. Then it becomes a nightmare
4. Managing Exchange and Sharepoint etc at scale is a full-time job. paying someone else to manage it wins vs. hiring people to do that.
5. Again it's $42 per user per month. We were spending more than that per employee on drinks and snacks before COVID hit. stocking 14 flavors of le croix, and the thousands of pounds of M&M's and "the good nuts" adds up. For a company with 9000 users, something that people are spending hours a day in, that's just cheap.

Assuming he hasn't patched it in a while...

http://[RouterIP]/cgi-bin/;REBOOT

Couple thoughts...

1. Explain the width of the field. Like medicine we have 100 directions you can go with sub specialization to the point that two professionals in the same room might not understand a single thing the other does. (IE Developer vs. Sysadmin, and within Sysadmin a AIX admin vs. a Windows admin will still have a lot of different concepts and tools).

2. Explain how NOT to get in the field. AVOID all for profit scam shops. Note that a large number of people work in IT without holding degree's, or holding non-traditional degree's. Explain how you can learn many concepts at home with a small lab.

3. Explain the role IT plays (Operations, Finance etc). In a lot of ways IT is becoming less of a stand alone field, and discuss how understanding business and operations are critical to the job (It isn't for people who just want to hide in a basement).

@scottalanmiller said in Handling Downvotes:

Votes should be public now.

On the condition that meta bitching about who downvoted you should be a ban

@IRJ said in Was It the Last IT Guys Fault:

@Carnival-Boy said in Was It the Last IT Guys Fault:

Probably the main thing that puts me off moving jobs is that it means moving in to someone else's shit, which you then have to spend months, or even years, sorting out.

That is definitely true, but generally when moving jobs, the pay increase is significant. I haven't changed jobs for less than $10k and sometimes closer to $20k.

There are jobs that are "net new roles" (Maybe a DBA for a new project) so you get to avoid some technical debt. Personally I didn't mind cleaning up crazy messes as long as I had the budget to do something about it (Joy of working for a MSP/Consulting company is you can tell people what it costs to fix, and if they balk you just go find someone else with money).

I"ve changed jobs for as little as 4K (but ended up being 8K after 90 day bump).
and I've changed jobs for 80K.

The thing I've seen with changes is that they would advance my career and give me skills I needed to move up and on. I never took a pay raise for a job that would hold me back.

@dashrender said in Looking for virtualization advice:

@tim_g said in Looking for virtualization advice:

@jaredbusch said in Looking for virtualization advice:

@tim_g said in Looking for virtualization advice:

@john-nicholson said in Looking for virtualization advice:

That $20 hr was for a contractor (yah, that's insanely low labor considering someone's skimming something off it). Note I lice in Houston, a fairly cheap city and that rate is what you'd pay a supervisor at a gas Station. It's unskilled cannon fodder here at that rate. Sometimes you get lucky (I hired a guy who moved furniture at IKEA for that for helpdesk, he ended up being smart and had to give him a raise to 56K at the end of the year though to keep him around.

In this case It was an unskilled typist, with no formal equation we taught to do the copy pasta cleanups from some scripts we wrote to try to accelerate it.

How I got out of making $20 an hour (hell slightly less than that when I started in this field) was identifying the lowest skilled things I did and then finding the cheapest resource who could do it for me. If your the god of Oracle RAC but still changing printer toner management is going to pay you like a printer serf. Stop doing cheap labor and you'll get paid more (assuming there are more valuable things to do in the day, if not find a new job).

I don't judge the value of a human being based on how much they are paid (I just spent July mostly in countries where a lot are on less than $2 a day). I do judge the value of labor (I was a hiring manager and had to know fair market rates). If your un-happy getting paid $20 an hour that's honestly your issue. If your unhappy that I say it's a cheap rate for labor in the US in a metro area for someone handling work on an IT department that's just disagreeing with a fact. I hear that's popular these days, but I've never understood it as a concept.

I completely understand. I'm not saying I make that little. My issue lies with how you say things. You seem to always say it in a degrading manner when referencing someone making less than you or boasting how many millions you make. It's not about the 20/hr, or any specific number, or any specific job or task.

Since it does not seem that way to me, then I can only assume that you are the one with the hangup on things here.

You know what they say about assumption.

I do tend to agree that @John-Nicholson posts often mention money and how much he's getting. This particular post isn't one such post. And the benefits post was really asking about compensation, so it's kind of expected there.

But the reality is that this is an SMB forum for the most part. Sure there is a small handful of people in the Enterprise, or have been in the enterprise making hundreds of thousands or millions, but those dollars just aren't the norm around these parts.

Something I learned from being a group admin for Spiceworks for years is that enterprise types lurk heavily in SMB forums. They PM people, and when they post it's often under pseudonyms as they are a bit more easily spooked about revealing who they are, largely for fear of vendors deal registering things or stalking them once they learn they have the budget or need. When you control even something small like 100K in annual capital spend the vendors can get kinda crazy. While they may not purposely come to places like this, Google and odd questions will lead them here. For the longest time "HDS vs. Netapp" led you to a thread on Spiceworks I started. HDS and Netapp both have MASSIVE forums, but they are either largely ghost towns, or hidden from public view (or just have Awful SEO). Also just because a place is a SMB don't assume they have no budget or pay people peanuts. I consulted across plenty of SMB's who paid individual contributors six figures and spent millions a year on IT purchases. I actually kind of hate using the term "SMB" (and some vendors don't even use it internally in account classifications) because it is often associated with a pejorative image of rock farmers, in a 6 man office with 1 IT guy who removes virus's all day. This just isn't reality. Those rock farmers have real time bidding systems on their rocks, and drones that map their piles of rocks several times a day to update their inventory of their rock piles (I'm not joking, an actual company here locally does this). Inversely I consulted in enterprises where they clung to NT4, paper processes, and 10-year-old servers. I've seen companies with 10,000 employees require the CFO's signature to buy a brand new $2000 MacBook Pro they were so cheap on capex!

As Scott's mentioned - If you are thigh well paid and are in IT, it's likely you would never visit a forum like ML as part of your day job because we have little to nothing to offer you. You area already probably as knowledgeable as most support staff at the vendor for whatever thing you're supporting, so it's likely there would be little to gain here.

Even if your not in SMB IT, there's reasons to be at places like ML and SPiceworks.

1. The lolz. There is some funny content here.

2. SMB overlaps with home IT on vendors sometimes (Ubiquit was a great choice for my home router, and wifi and you will not find tips on how to configure them in more enterprise places)

3. It's a forum. Most of the more enterprise conversations I have online are on Twitter, or Telegram or Slack. Forums have some old school charm in their permanence.

4. To give back. Some people got their start in the SMB space in forums like this. Personally, If it wasn't for forums like this my career would have been vastly different (It's been something like 7 years since I met Scott at the first Spiceworks user group).

5. An opportunity to argue with Scott. It's fun, you learn things in forming your arguments and it helps you sharpen your rhetoric.

6. There's a lot more consistency of actors here than you'll find on places like Reddit. You can learn to understand who/what people are about etc. There's a fair amount of people on this community I've had a beer with over the years.

I'm curious, is @John-Nicholson even in IT anymore? I can't quite tell from the conversations. It seems that he's more in a leadership role, but that might be a misunderstanding of what he's posting?

He's still around the field. he stopped touching production a year and a half ago. He's clearly an individual contributor. If you go to VeeamOn, Dellworld, VMworld he'll buy you a drink and explain.

@tim_g said in Company Benefits:

@jaredbusch said in Company Benefits:

The owner pays out bonuses to all the employees twice yearly, but it is simply profit sharing.

Basically he keeps cash banked to handle XX months of payroll. Then as long as we have that he pays out the overage as a bonus to us based on full time / part time and how long we been here.

Yeah, and bonuses get taxed like crazy. If you get a 5k bonus, you get less than half of it in your pocket.

I'm not sure if your serious but...

@scottalanmiller said in You can't quit, you're fired!!:

want you at the office, and you've given 2 weeks (at least in NY) you're entitled to that pay.

No. They can totally fire you. Otherwise you could put in long notice to avoid being fired

If they fire you can claim unemployment. This raises/spikes their insurance costs though so smart companies go to GREAT lengths to not fire people.

@JaredBusch said in Domain Controller Down (VM):

@wirestyle22 said in Domain Controller Down (VM):

Thanks to @John-Nicholson for the help! A lot of great information.

The problem is now that we stepped up to help a member of our community and the bosses know nothing of how much this should have cost you to get repaired.

VMware GSS is around 24/7 to help with stuff like this.
Just call them next time. 1 (877) 486-9273
Make sure to add yourself to the authorized list ahead of time for faster service, and write down your customer support. This is why you use enterprise products, so you can get help quickly.

Steps everyone else missed.

vCenter came up after the DC's so we restarted the vCenter Services.

Some of the VM's were running as zombie's (storage had dropped to long, OS had crashed) so the VM's had to be reset once storage came back up (Note APD detection in 5.5 and 6.x is better and this isn't as common).

Storage latency over 100Mbps iSCSI is awful. 500ms max on one VM, and average of 45ms. This is like running your hard drive remote from Houston to Atlanta. Hence my recommendation of some proper enterprise direct FC attached storage to deal with this mess.

This stuff isn't supported (and has NEVER been supported to use vMotion or iSCSI over 100Mbps) in the past 8 years I've worked with VMware.

As we discussed my home lab from 5 years ago was in a better state for availability and performance and supportability.

Full disclosure I work at VMware blah blah blah words are my own...

1. vSphere is an eco system. There are TONs of solutions that it interacts with (and has really tight integration with). Running Horizon for VDI? Your only option is ESXi (although, Azure will be an option at some point here). There are tons of Ecosystem features that you REALLY can't get anywhere else, or are less mature on other platforms. Examples include

Microsegmentation with selective layer 7 service offload (NSX).

Platform residency like Pro-active HA (as well as vSphere HA and VMFS being far more resilient and tunable than other platform's options).

Platform features (vSAN isn't the only local storage replication system, but it's more tightly integrated and offers more data services and maturity than most systems out there).

Granular performance controls (DRS's algorithms are unmatched in the industry, NIOCv2, SIOCv2, highly mature APD/PDL detection at the storage PSP layer that would require a 3rd party software on most platforms). Using all this stuff I typically see 40% denser host utilization than people using purely free solutions (or just ESXi free).

Wholistic monitoring solutions and integrations (vROPS, Hyperic and LogInsight have crazy amounts of out of the box functionality across everything from hosts, to applications, to networking and external devices).

Hybrid cloud. Your replicating to a VMware partner in the vCloud Air Network like OVH or Softlayer (VCAN), or wanting to run Hybrid cloud operations to AWS (coming soon (TM).

Operational reasons. I can throw a rock and hit someone who knows how to manage ESXi and vSphere. There are a bazillion people are trained and know how to do not just basic Install configure manage, but also advanced troubleshooting.

Low cost 24/7 Enterprise support. I can support 3 hosts with 24/7 phone support for ~$1200 on an essentials plus bundle. Microsoft's lowest flat fee support option I've seen is 40K a year as part of an ELA.

Lost cost (Essentials Plus is ~6K. For similar functionality I'd need to buy SCCM VMM which costs more and lacks a 24/7 support option).

Better driver/firmware quality control. VMware VCG doesn't blindly certify anything for a 5 pack of heineken (Realtek was banned for a VERY long time). Do to what customers use the drivers for (Never consumer use cases) they get more attention and there is a higher expectation.

The vDS is incredibly advanced and mature. Beyond NIOCv2 shaping functionalities that are a check box away, Advanced LACP, single click to setup CDP/LLDP send/receive. VERY rich load balancing algorithms hashes (not just a basic IP hash). Other platforms historically require 3rd party NIC vendor dependent tools do things like this.

Security/Compliance. There are platform features that are painful to replicate in other places, or if doable require endless amounts of scripts or 3rd party projects that may or may not be stable. There is a full DISA STIG for classified use case. Due to ESXi's tiny size (few hundred MB for the kernel) it's patch surface is tiny. Compared to platforms that minimum install is 20+GB who require monthly patching to remain compliant.

Mature native backup API's (VADP/CBT), Native Write Splitting API's (VAIO allows near zero RPO) and all kinds of fun platform features that allow for a rich 3rd party ecosystem.

Rich API's, and SDK's that are stable and managed by grown up's (This isn't docker where everything breaks every 3 months).

VMFork plus Photon allows for linked clones VM's that have zero net Memory/CPU/Disk overhead and can be booted in 400ms. This can be controlled by Kuberentes or Docker endpoints and gives you the benefits of containers, with the isolation of VM's (Performance control, microseg, visibility) with vSphere integrated containers.

Free OpenStack deployment that's a simple wizard to setup (vSphere Integrated OpenStack).

Also, ESXi is free and offers a nice HTML5 UI that's quite easy to use.

Anecdotally, it's just easier to deploy and manage, and tends to do weird things a lot less (and when it does you've got support to reach out to). The new VCSA vSAN bootstrap system, and the configuration assist for bulk setup is awesome. With a simple wizard I can have my entire cluster deployed in under an hour.

There's some more obscure things to consider.
On the VMware front...

SMP-FT (what witchcraft is this?)

How are heartbeats done, and is HA managed passively by staged scripts and the hosts, or a central server? (vSphere it's stand alone, works if vCenter down)

Storage HA heartbeats. What are they, and why do they rock? Why can't I use HCI storage for this?

What are fault isolation responses and why do you I need to know this?

Proactive HA. I have a server that is dying, why doesn't stuff move off of it pro-actively?!?

What is Virtual Machine HA, and what are API hooks for application HA?

A good book on HA and DRS by Duncan.
http://ha.yellow-bricks.com/

@matteo-nunziati said in Is this server strategy reckless and/or insane?:

@storageninja said in Is this server strategy reckless and/or insane?

2. There are a lot of Cache's.. There are controller caches, their are DRAM caches inside of drives (You can't disable this on SSD's, and can only sometimes turn they off on SATA magnetic drives and others). Some SDS systems use one tier of NAND as a Write Cache also, some do read/write caches.

I was aware of only 3 levels of cache: os, controller, disk.
Disk cache can be disabled according to my controller. Is The cache you are talking about another level ofthe disk?

It's technically inside the disk controller (you can't put DRAM on a platter!).

Enough Magnetic SATA drives will ignore the command, combined with the performance being completely lousy that VMware stopped certifying them on the vSAN HCL.

As far as SSD's go, they use DRAM to de-amplify writes. If you didn't do this (absorb writes, compress them, compact them) the endurance on TLC would be complete and utter garbage, and the drive vendor would get REALLY annoyed replacing drives that either performed like sludge or were burning out the NAND cells. Some TLC drives will also use a SLC buffer for incoming writes beyond the DRAM buffer (and slide the NAND in an out of SLC mode as it can't take the load anymore and retire it for write cold data). SSD's are basically a mini storage array inside of the drive (which is why you see FPGA's and ASIC's and 4 core ARM processors on the damn things).

There are also hypervisor caches (Hyper-V has some kind of NTFS DRAM cache, ESXi has CBRC a deduped RAM cache commonly used for VDI, Client Cache a data local DRAM cache for VMware vSAN) there are application caches (Simple ones like SQL, more complicated ones like PVS's Write overflow cache that risks with data loss to give you faster writes so must be strategically used).

On top of this there are just other places besides this disk for bottlenecks to occur inside of various IO queues or other bottlenecks.

The vHBA, The Clustered File System locking, Weird SMB redirects use by CSV, LUN queues, Target queues, Server HBA queues (Total, and max LUN queue which are wildly different). Kernel injected latency can cause IO to back up (no CPU cycles to process IO it will cause storage latency) as well as the inverse (high disk latency, CPU cycles get stuck waiting on IO!) which can lead to FUN race conditions. Sub-LUN queues (vVols) and even NFS have per mount queues! IO filters (VAIO!), guest OS filter driver layers can also add cache or impact performance. Throw in quirks of storage arrays (Many don't use more than 1 thread for the array, or a given feature per LUN or RAID group like how FLARE RAN for ages) and you could have a system that's at 10% CPU load, but it being a 10 core system the 1 core that does Cache logic is pegged and causing horrible latency.

You can even have systems that try to dynamically balence these queues to prevent noisy neighbor issues (SIOCv1!)

http://www.yellow-bricks.com/2011/03/04/no-one-likes-queues/

Another trend in benchmarking is using stuff like HCI bench or VM Fleet to test LOTS of workloads. A single worker in a single VM doesn't' show what contention looks like at scale.

You have a front-facing service that has a login prompt. Random automated login attempts are just part of life. What can you do?

1. Setup Fail2Ban. (Smart botnets split the load across lots of IP's).

2. ~Geo Blocking~ useless, as bots are all over the place (many in the US)

3. Double Check your password policy (make sure they can't use easily guessable passwords).

4. If you actually have users with highly valuable data in their email, force MDM agents on their mobile devices, if they want to use mobile access Exchange, can be configured to do this. Alternative use a whitelist for remote/mobile devices (Exchange 2010 on has a ActiveSync device quarantine options where devices even if they can authenticate don't get email till you approve them).

5. I've seen it done with AirWatch so only Boxer as a mail client will work as it has a device-specific VPN.

6. Disable unneeded and insecure protocols. IMAP and POP3 shouldn't be externally facing it's 2017...

Lastly, who still uses Zimbra? We used to own it, but now just use O365 (and have Microsoft's billion dollars of security spending and IDS in front of it).

@scottalanmiller said in I think I am missing something about Hyper-V....?:

As @StorageNinja once said, 99% of Hyper-V installations are done by accident or by mistake.

John White and I used to keep a list of people who were mistaken that Hyper-V had special licensing rights. At one point it was over 75% of why people went Hyper-V.

@EddieJennings said in Where IT Consultants fit between Vendors and Clients:

I don't recall saying consultant vs non-consultant, but the responses in the thread have addressed the question of who should interface with a vendor.

I feel like I should provide some context for how some vendors operate to get a better idea for the level of vendor involvement and who the vendor wants to work with.

Few things....

1. It depends on the vendor, and who the customer is. For instance, some vendors are 100% channel sales (Datto I think fell in here) and a customer outright can't buy them directly.

2. Most larger vendors DO NOT WANT to talk/sell to smaller customers directly (It's too expensive, as they pay too good of benefits, and too high of compensation to their salespeople to scale down to small accounts that because they only sell their products can't form a meaningful relationship). There typically are 4 "buckets" for products.

a. Retail sales for VERY low-value non-complicated sale items that a website can sell. These products don't require sizing assistance or are pretty simple. Think an ethernet patch cable.

b. More complicated items on smaller deals that intend to be 100% channeled in sales (You don't want this stuff sold by Amazon as the customer will likely buy the wrong SKU, or screw upsizing). Note, the vendor may offer a direct model but will often have "cannon fodder" class salespeople in this space, and generally will even charge more for going direct. A VAR is your best bet here. Think someone buying 3 servers, or 20 laptops, or a single palo alto firewall for a SMB. all services are going to be VAR partner led when possible beyond post-sales support escalations. Also in these smaller accounts it's expected that the VAR/MSP is more than likely going to know the needs potentially better than the customer does.

c. larger enterprise deals where the VAR is still involved but the vendor takes some leadership because the account is big enough to matter, or the vendor wants a strategic presence in this account. The paper may shift to being run by the vendor at the higher end of this, with a small revenue share back to the VAR who brought this deal to them for the life of this deal. Think ELA's, 100 site MLPS circuit deals etc. services might be delivered by either the partner or the vendor at this stage.

d. Direct only deals. These are sometimes called "named accounts" and the vendor will 100% run paper directly. A partner of record might get 3% of the deal if they are lucky, or be subcontracted if they are a marque support partner with tons of certifications.

Others can comment but sales teams tend to be organized around these different groups Example:

1. Commercial-1 Smallest accounts and people who haven't bought anything in 5 years from you. These are called "Whitespace accounts" and you basically have people trying to get a meeting with hundreds of these in a territory or verticle and seeing if they can find some gold and get people with a low priced entry solution. ALL sales will be inside teams at this scale with VAR's or MSPs type shops doing any in person meetings.

2. Commercial-2 Slightly larger accounts. Might have spent a few thousand, but there isn't a strategic or lucrative relationship. You might have a field team at this point but they will likely cover hundreds of accounts still.

Midsized Accounts - Still larger. They will likely have some clue who their account team is, but still rely on a VAR for most day to day stuff.

Large Enterprise - Big names you recognize. These accounts will have teams who might have only 5-10 customers. Alignment on this is going to be tied to geograhpy still more than likely.

Globals - Account teams will be in some cases 1:1, or if there is a specific industry (Say automakers, or oil gas) you might have a team in a city (like Houston) whose job is to wrangle these guys. The Cxx levels of the vendor likely have strong relationships with these accounts and for a software vendor these accounts could be spending 9 figures at a time, or for hardware companies 10.

@kelly said in Which skills is more valuable to you? Hard skills or Soft skills?:

@storageninja said in Which skills is more valuable to you? Hard skills or Soft skills?:

@black3dynamite

3. Soft skills are harder to teach.

This. Assuming that all other things are equal (teachability, motivation, etc.) I would much rather have soft skills.

We do one soft skills class (1-2 day) a year. Focusing on speaker and communication training. It's not cheap but our team is really good at presenting idea's and content as a result.

Base, plus growth structure - can it grow? Is there an org chart with clear steps to moving up and getting bumps in pay? Does everyone get 2% and stagnate till they leave?

OTE Bonus. I get paid an extra multiplier based on my base pay based. Mine isn't actually tied to metrics (I prefer this) and is completely based on my boss's assessment of me doing my job. While there is an "On Target Earnings" nothing stops you from getting over 100%. While when accepting the offer I weighted the bonus at 50%, I will say "it's not virtual". The biggest way to see how real this is is go check with GlassDoor and existing employees who've been there 4-5 years.

RSU (Restricted Stock Units)'s - If you keep getting these every year on a standard 2-5 (Depends on company and grant window) year vestment schedule, you eventually end up with a rather nice kicker. This also is really nice if your stock doubles within a given year (Well except for capital gains). The longer you stay the stickier these become, and the more a company likes you the more they will give you to "handcuff" you to the company. The more a company wants you to stay the more you get these (i know people with solid 7 figure piles). A decent 6 figure pile of this is nice and can be used in leverage with a company who wants to poach on you why they better give you a bigger base (or a bigger pile of them!).

Education School, College, Certifications, Classes. I can take a pretty limitless number of certifications and classes. Wishing my wife had this as the never ending amount of certifications she has gets fun..

Sabatacle In our company you can apply for 3-month transfers to wildly different jobs to learn about how that role functions. You can do a 1 week education track (Go take education in something unrelated).

Stock Options - Inversely if you work for a startup you might get stock options. These are a LONG shot gambling game (like 2% pay off) but I know some guys who their stock is trading in the 30's and their options were in the $2 range so assuming they make it to lockout I expect to get a call to hang out on their yacht....

ESPP - Buy stock at a discount (See above comments). Note these are generally bought at a 10-15% discount based on the beginning or ending window (Whichever is lower) so its a game of heads I win, tails you loose against the market and can pay pretty well (or just be a nice couple grand of cash). I've had windows where I made 15%, sometimes I've made 115%. Either way, making 15% on a 6 month time period on the market with 100% certainty can't be beaten.

Paternity leave - 18 weeks full pay, maternity, paternity, and adoption leave. Per kid and against my salary if you take 18 weeks of my pay that's the equivalent of 2 years of tuition at the local state school in town. My sisters is 6 months (she's taken twice now!).

vacation Unlimited. Just got done with 4 weeks traveling Asia, spent a week in Mexico, and have another week or two in India later this year on top of some 3 day weekends.

Work from home/anywhere Sometimes I just leave town on Wen/Thursday and go to a beach house to finish working out the week.

Travel Points and status - Traveling for work a lot adds up. Note this is a NON-taxable (Weird exclusion). So when traveling I can get hotel points and airline points. With SoutWest I have a companion pass (My wife flies free with me), and with Marriot, I get free cocktails and appetizers in the afternoon and breakfast in the morning in the executive lounge. I get free upgrades with Marriot when traveling so that $150 small room can turn into a 40th-floor suite sometimes. I just stayed a week in bali at a 5 star hotel without having to pay for the rooms or $30 breakfasts.

Expense
Do they let you do your own booking, do they require a corporate credit card (no points can be brutal, to the point of $20-30K easily for some people in compensation) Can you expense travel lounges (Sooo nice). With customers, I can pay for fairly nice meals/drinks etc without issue. I have Uber and Lyft integrated into my expense account so I've managed to cut my travel in my own car to ~600-700 miles in 6 months.

Travel
Travel Policy - Do they make you fly 18 hours, 5 hops to save $100?
Do they put you in first class if the flight is over 4 hours?
Do you stay in the Motel 8 and have to share a room (or PAY for your spouse's 1/2 of the room if they happen to travel with you!).
Do they make you fly in the morning you are presenting when it's 12 times zones away, or do they put you up in the hotel for the weekend to adjust to the time zone, and be a tourist for the weekend?
When you're at a conference in Vegas can you boss write off $150 tickets to see Billy Idol.

My wife's got some other weirder stuff

non-profit/public retirement options.

401A - Like a 401K match but you don't have to put money in, they just put x% of your salary.

457(b) - Can withdraw from it without early penalty if you no longer work for said employer.

403B - A lower overhead 401K plan with no match.

allowance for continuing education.

Equipment allowance. She can spend money on books of stethoscopes.

By strategically maxing out withholding on all this, she can massively reduce her taxable salary.