Best posts made by NetworkNerd

I remember attending the VMUG UserCon in Dallas / Fort Worth for the first time back in 2013 and hearing Damian Karlson speak about vBrownBag. At the time I had never heard of vBrownBag, but one thing that really hit me was his mention of community. When he said “the community” he meant the virtualization community. He encouraged folks to get on Twitter to participate. To me, that was completely preposterous because Spiceworks was “the community.” That might sound a little closed minded considering I was at a VMUG event, so let’s start with my community background up to this point.

It Started with One Installation

For me, Spiceworks was where it all began and will always feel like home. I joined the Spiceworks community in December 2010, which is when I first installed the application and began using it for ticketing. A friend from church (now my current boss) told me about it. I mostly used the app and didn’t pay much attention to the community, not really knowing much about it or how to use it.

But because I had installed the app and registered a community account (where the NetworkNerd name was born), I got notified about a local group of IT Professionals called SpiceCorps. It was free to attend, so I decided to go. At this first meeting in particular, I got to meet David Babbitt, one of the developers who worked at Spiceworks. He was in the area to get feedback on some new product features and to take any questions attendees had. I remember picking his brain about using Spiceworks for our Maintenance department in addition to just for IT. I thought it was so exciting to get to meet someone who worked at Spiceworks.

You can read the rest of the story here: http://blog.thenetworknerd.com/2017/11/18/the-power-of-finding-your-community/

@scottalanmiller said:

@NetworkNerd said:

Yep - that's about what they told me when I called a few minutes ago. And I have no clue what to expect a procedure like draining the puss / fluid from a kid's toe to cost. They couldn't tell me either (or just plain wouldn't).

Yup, wouldn't. They can't make their money if they agree to a price.

The good news is I was able to get here worked in at her pediatrician's office. The visit is only $84 (not nearly as hefty), and they will bill us for the rest. I'm not sure what that will be, but at least I don't have to pay an unknown amount up front.

@scottalanmiller said:

@Lakshmana said:

How to block the SSH port in the IP table.I used the following command but it does not work?
My aim is that when I try to access the machine in Putty,the Centos machine should not be taken in the Putty

/sbin/iptables -A INPUT -p tcp --destination-port {PORT-NUMBER-HERE} -j DROP

You don't block port by port, you simply need to stop allowing it. You need to block everything. Show us your entire config file. IPtables can't be discussed without seeing the file you are asking about.

And don't forget that the traffic will follow the first matching rule in iptables. If you wanted to allow SSH for only certain ips, for example, you could put ACCEPT rules above the general DROP rule for SSH in the iptables config. That way those specific ips could get in as expected, but all others get blocked.

Excitement was in the air for the technology team of Beast Mode, Inc. After months of planning, some shiny new vSAN ReadyNodes finally arrived at the datacenter. The implementation team gathered to rack and cable the new cluster and all other necessary equipment. Even though ESXi was installed on the hosts with basic network settings configured, there was no datastore on which to install vCenter. They had yet to configure the vSAN cluster.

A bit of worry set in, but John, the team lead, remembered something. As part of the vCenter Server Appliance installer in vSphere 6.5d and later (vSAN 6.6 and later), they could leverage Easy Install to create a vSAN datastore on some of the hosts and use it to deploy vCenter. With vCenter online and hosts updated to the latest patch of ESXi 6.5, they configured the rest of the hosts in the vSAN cluster with all networking needed for vSAN traffic, management, and vMotion. Now that the cluster was fully functional, the team was ready to deploy some virtual machines…or so they thought.

John also reminded the team to apply the VMware licenses before going any further. The cluster in question contained 4 hosts, each with a single physical processor, fully licensed for vSphere Enterprise Plus and vSAN Standard. Each host would be managed by vCenter Standard. The team knew how to apply the vSphere licenses as well as the vCenter license. But the mystery began when they tried to apply their licenses to use vSAN.

You can read the rest of the story here - http://blog.thenetworknerd.com/2017/12/11/captain-vsan-and-the-case-of-the-cluster-license-mystery/

My wife and I are going on a cruise next week, and I will be completely unavailable to my team during that time. I'm not paying for wifi on the ship and will likely not check e-mail while in port at some of our cruise stops.

So after taking vacations where I had constant access to e-mail and phone calls / texts if needed, I am going cold turkey. For those who have done the same, did you feel a little anxiety in doing that? After working several side jobs in the past couple of years, it will be very different to not check e-mail every day. I'm sure it will be enjoyable, but I'm actually getting anxious about being disconnected. As long as my team can handle matters while I am away, it should be fine, right?

This will be the first time in over 7 years of IT where I will be completely disconnected.

Regardless of your field, if you aren’t a member of a professional networking group / community, I hope 2018 will be the year you choose to get involved. There are powerful benefits to your career for participating.

I’m the kind of guy who participates in IT communities and happen to be the co-leader of the SpiceCorps of Dallas / Fort Worth. What is SpiceCorps, you ask? At the core, SpiceCorps are local meetings of IT professionals to discuss technology, career, and support one another as we navigate the fast and furious world of IT. These groups were formed by members of the Spiceworks community after the first Spiceworld in 2008, originally meeting to discuss the Spiceworks application. Over time, the conversations evolved into general IT discussion and became what you see today.

In case you weren’t able to make it, here’s an overview of 2017 in our SpiceCorps - http://blog.thenetworknerd.com/2018/01/09/dfw-spicecorps-a-2017-year-in-review.

If you have done much with VMWare troubleshooting using esxtop, this is a handy addition to your toolkit - VisualESXTop.
https://labs.vmware.com/flings/visualesxtop

Here's a blog post that gives some background on how to use it: http://www.yellow-bricks.com/esxtop/. What I like most about it is the ability to make charts so you can see the performance data in real time. It works on ESXi 5.5u1a at least, but I have not tested it on higher versions.

If you have not used it, check it out!

Like many nerds out there, I enjoy tech podcasts. Datanauts is one of my favorites and a must listen for the technology professional. A recent episode on masters and mentorship was one of their absolute best, and the career discussions on it can be applied to any field. In this episode, guest Don Jones makes a thought provoking comment: “you can be master and apprentice all at the same time.”

Think about that for a second. Do you think of yourself as a master? Or do you normally identify more with the apprentice, a mere padawain seeking to learn and master a trade?

A master is not just a skilled tradesperson but someone skilled and able to teach. The master’s job is to train an apprentice and to help him / her one day become a master. It’s a role taken up to protect and preserve the craft for future generations. Once an apprentice achieves the level of master, there is room for another apprentice to take up the trade. In honor of teacher appreciation week, the following advice for the master encourages a look at the process through the eyes of an apprentice.

You can read the rest of the story here - http://blog.thenetworknerd.com/2018/05/24/through-the-eyes-of-the-apprentice-mentoring-advice-for-the-master.

Before I started here a couple of months ago, my boss purchased a couple of Dell R630s and a PowerVault MD3820i (20 drive bays) to be our new infrastructure at HQ. We have dual 10Gb PowerConnect switches and two UPS devices, each connected to a different circuit. The plan is to rebuild the infrastructure on vSphere Standard (licenses already purchased) and have a similar setup in a datacenter somewhere (replicate the SANs, etc.). We're using AppAssure for backups (again, already purchased).

The PowerVault has 16 SAS drives that are 1.8 TB 7200 RPM SED drives and 4 SAS drives that are 400 GB SSD for caching. Well, we made disk groups and virtual disks using the SEDs (letting the SAN manage the keys), but it turns out we cannot use the SSDs they sent us for caching. In fact, they don't have SED SSDs for this model SAN.

At the time the sale was made, Dell ensured my boss everything would work as he requested (being able to use the SSDs for caching with the 7200 RPM SED drives). Now that we know this isn't going to be the case, we have some options.

First, they recommended we trade in the PowerVault for a Compellent and Equalogic. The boss did not want that because he was saying you are forced to do RAID 6 on those devices and cannot go with RAID 10 in your disk groups. As another option, Dell recommended we put the SSDs in our two hosts and use Infinio so we can do caching with the drives we have. In this case we would make Dell pay for the Infinio licenses and possibly more RAM since they made the mistake.

But I'm wondering if perhaps there is another option. Each server has 6 drive bays. So we have 20 drives total. Couldn't we have Dell take the SAN back, give us another R630, and pay for licenses of VMware vSAN for all 3 hosts? Each server has four 10 Gb NICs and two 1 Gb NICs. That might require we get additional NICs. But in this case, I'm not sure drive encryption is an option or if we can utilize the SEDs at all.

I've not double-checked the vSAN HCL or anything for the gear in our servers as this is just me spit balling. Is there some other option we have not considered? We're looking to get the 14 TB or so of usable space that RAID 10 will provide, but the self-encrypting drives were deemed a necessity by the boss. And without some type of caching, we will not hit our IOPs requirements.

Any advice is much appreciated.

It's podcast Tuesday, ladies and gentlemen. This week we talk about the affect of company culture on career and take an initial pass at resume writing advice. Get the full episode here.

On this Podcast Tuesday, are you searching for a new job? Are you using the shotgun approach, or have you learned to be a job-hunting sniper? Tune in to this week's Nerd Journey episode as Thomas Delicati shares his experience landing a new job after relocating and the strategy that set him up for success.

Get the full episode here.

I heard about this opening recently from a member of my professional network and wanted to pass it along to others. I'm happy to vouch for a fellow community member that has the right skills. Feel free to send me a chat message.

Pro tip - watch the video on the job posting.

https://jobs.entrustdatacard.com/jobs/sr-systems-engineer-2019

It's nice to be wanted, no? I wish you all the best, and maybe someday you will learn to love Dell.

@art_of_shred said:

If I saw numbers like that, I'd be looking for what went wrong. No way that's real!

If only I had a Dropbox account big enough to try and upload a full backup of all my servers I would be able to tell you.

It was mid-day on a normal Wednesday when I started getting alerts that devices have fallen offline at one of our locations. Seconds later the alerts about the firewall at this location falling offline were hitting my Inbox. Something was awry, and I needed to determine exactly what happened. My team and I are physically stationed at the company headquarters but support several remote sites in the area (and some out of state).

I called a user at the location in question (which is just over 30 miles from my office) to see if the facility still had power. He said there were no power issues and even went into our telco room at this facility to check equipment lights. Everything seemed normal on our UPS (not running on battery), firewall, switches, and the ISP's modem (all lights for US/DS and online solid blue).

This location has about 20 users and uses Time Warner coax as their internet connection. For the most part the connection has been reliable, but it is still coax and not as reliable as fiber. Since we couldn't ping the ISP's device at that location, I had the user power cycle the modem to see if that would fix the problem. Once the modem was online again I could ping it for about 15 seconds and then never again. Our firewall never came online, and folks at the location could not connect to the internet.

My mind immediately jumped to possible ISP issues, and I called Time Warner support to see if they might have an outage in the area. After giving the technician all the account information and describing the problem, I hear him pause and then say, "oh, there's the problem. Your account has been issued a soft disconnect because your account is overdue." Honestly that's the last thing I thought could have been the problem in this situation.

The technician transferred me to someone in billing who told me the account was 3 months overdue, and we owed over $1000 to them. I check with Finance, look through all of the billing statements, and find we issued a check on 8/18 for over $1000, which Time Warner had not received and had not yet processed. Even if they had received it, the processing of the check and applying it to the account takes time. Waiting on the check to process was not an option.

Luckily, I was able to pay the minimum amount to get us back online again (a little shy of $700). After the person in billing processed the payment, they had our internet circuit turned on again within 5-10 minutes. Once they receive our check and process it, we'll have some credit on our account for a couple of months.

After all of this I went and spoke with Finance about the situation one more time. It looked like we had received the invoices but that they had been bouncing them around to different people to approve before issuing the check, which caused a bit of a delay. In any case, they are going to look to pay this specific bill using an auto-draft so we won't run into this problem again.

At your company, have you ever had this happen? Do you have a role in approving the bills for recurring charges like internet and phone? What's your process?

Thanks! She surprised me with a cruise for a gift. It's tough to compete with that.

Let me just say that when I upgraded from 5.0 to 5.1, some things went semi-haywire. I've not gone to 5.5 yet on any ESXi hosts, but if it were me, I would go straight to 5.5 with a clean install on a USB drive as you mentioned.

All I did was upgrade vCenter 6.0U3 to 6.5a, and things went horribly wrong. Before getting into the details, let me take you back to where our story begins.

Where We Began
It all started when we began setting up new hardware to completely gut and rebuild our infrastructure. We started at site 1 of 3.

We had two Dell PowerEdge R620 servers, some PowerConnect N4032 switches, and a MD3820i SAN with plenty of storage. The R620s came from the factory with ESXi 6.0U2 installed. We had license to both vSphere Standard for all hosts as well as vCenter Standard. The end goal was to put 1 vCenter instance at each site and use Enhanced Linked Mode between the different vCenter instances. For Enhanced Linked Mode to work as expected, we had to have one or more external Platform Services Controllers.

Since we had planned to have only two hosts inside the cluster at site 1 using the same back end storage, I used a different server outside the cluster running an evaluation version of ESXi 6.0U2 and local storage to get started with vCenter.

This was early February 2017, so there was no 6.0U3 yet (not released until 2/24/2017). Even though 6.5a had just been released on 2/2/2017, our backup software vendor was not yet officially supporting vSphere 6.5. Also, the vCenter plugins for management of the MD3820i were only supported in vCenter 6.0. At this point we could not yet upgrade to vSphere 6.5 but agreed that we would once all other environmental components were officially supported.

Since the hosts were at 6.0U2 and I was chomping at the bit to get started, I grabbed the vCenter Server 6.0U2a installer from VMware's site and went for it. I deployed the Platform Services Controller (PSC) appliance and a Windows vCenter Server (just in case we needed Update Manager) on the ESXi host outside the cluster. We used vCenter to configure our cluster without any trouble, and vCenter 6.0U2a was working like a champ.

The Plot Thickens
As described in this post, we were not able to use the SSDs we had purchased as cache for the SAN, so we began testing Infinio as a caching mechanism. At the time, Infinio recommended we update everything to vSphere 6.0U3. I went ahead and upgraded the PSC and vCenter to 6.0U3 and used Update Manager to update the hosts to 6.0U3. This was early March 2017.

At this point we had vCenter 6.0U3 working fine. As you can see from the linked post above, we ended up sending the R620s and the MD3820i back to Dell in exchange for 4 R730s that we would use as a vSAN cluster at site 1. We would eventually follow a similar pattern at other sites.

Since we already had a vCenter and PSC outside the cluster, I decided to upgrade them both to 6.5a while we were waiting on the new R730s to arrive after confirming this would be fully supported by our backup vendor. I used the vCenter Server Appliance installer and the built-in migration assistant to migrate the PSC to 6.5a and migrate the Windows vCenter instance to the vCenter Server Appliance 6.5a.

I documented the entire process. The migration assistant worked like a champ to create a new PSC and VCSA running 6.5a with all of my configurations and history saved.

At this point we were at version 6.5a with vCenter and the PSC.

When Things Went Haywire
The new R730s arrived in mid-April 2017, and each one had 6.0U3 factory installed. Around that same time, the newest version of vSAN was released as part of vSphere 6.5d, so the first thing I did was upgrade the PSC and VCSA to 6.5d. I have outlined how simple that process is to do in this post.

We had a PSC and VCSA running the latest and greatest version (6.5d) on a host outside of the cluster we were about to build. I was able to use Update Manager to upgrade all of the new R730 servers to ESXi 6.5 (using the Dell ISO for ESXi 6.5) and then patch them (with a host patch baseline) to 6.5d.

Before adding the new R730 servers to vCenter and starting the vSAN build, I realized I still had an empty cluster object sitting in vCenter. To start fresh, I went ahead and deleted the cluster object from the vSphere Web Client, and that is when vCenter went completely nuts on me. After right-clicking the cluster and selecting the option to delete it, I would see it disappear from the object tree on the left but would also see a task in the task pane that would never complete.

Then, everything seemingly disappeared from the host and clusters view, the vms and templates view, and essentially everywhere else inside the vSphere Web Client.

Hosts and Clusters View

VMs and Templates View

Then, eventually, this error showed near the top of the vSphere Web Client.

Trying the vSphere Client instead showed a 503 error.

At this point, vCenter was essentially hosed. I rebooted it from the VAMI. After the reboot, everything was fine again. I tried deleting the cluster again only to have the same thing happen. The only way to bring vCenter back was to reboot the VCSA. This time I tried creating a new cluster object and then deleting it. The exact same thing happened. I tried rebooting the PSC first and then the VCSA, but that did nothing for me either. It seemed like I could do anything in vCenter except delete a cluster. It may seem small, but what else may have gone wrong if I had not found the problem before building the vSAN cluster?

I had no choice at this point but to contact VMware Support. The support technician from VMware showed me that the vxpd process would crash any time we deleted a cluster object in vCenter. The technician recommended I backup the vPostgres database and restore onto a VCSA with the same name and ip to see if that resolved the issue so as to retain all vCenter data.

Lessons Learned
After getting some additional help on Twitter, I eventually found my upgrade from 6.0U3 to 6.5a back in March is not officially supported by VMware.

Since we had very little historical data in vCenter, I ended up deploying a fresh PSC and VCSA using the vCenter 6.5d installer and utilizing the Easy Install for vSAN 6.6. That allowed me to setup all of the host networking appropriately, claim disks on all hosts, and use the configuration assistant to provision the vSAN cluster with no issues.

Why wasn't my upgrade supported? It turns out vSphere 6.0U3 was released on 2/24/2017, which was actually after 6.5a was released (2/2/2017). And as is clearly stated in the release notes for vCenter 6.0U3, you can see that an upgrade from here to vCenter 6.5 is not supported.

Always, always, always check to see if moving from your version to a higher version of the product before performing upgrades. In my case, I wish the installer for vCenter 6.5a had stopped me from upgrading altogether. Even though I goofed up the upgrade the first time, I learned the product so much better by having to re-create from scratch.

I'd also like to give a special thanks to Adam Eckerle of the VMware Technical Marketing Team for his help and advice in figuring out the root problem and helping me get it resolved.

I happen to know an individual who recently invested in a pretty nice Dell server for his business. He decided to spend a few thousand extra to get all enterprise SSDs rather than 10K SAS disks to hit an IOPs home run with his ERP system. It's OBR10 and all local storage (very nice gear). I did encourage him to get the Perc card with more cache as well. This will be a VMWare setup with the Essentials kit and a single VMFS5 datastore (nothing more than that was needed). For purposes of this thread, I am not concerned with backups.

If this individual had, for example, eight 320 GB SSDs in OBR10 on local storage with a nice Perc card to create the array, I wonder...

If you're running heavy IOP loads like SQL or Exchange (some environments are much heavier than others, of course), how much performance gain would you really get by doing thick eager provisioning for SQL / Exchange / insert application here over thin provisioning when everything is running on SSDs?

Is it best to go thick in this case if you are heavier write than read?

Do the guidelines of when to use thin vs. thick eager change because of the speed of the storage and possibly allow you to save some storage space by sticking with thin provisioned disks?

I've been thinking about it and don't have a good answer as to some guidelines for using thin vs. thick when talking about SSDs. I wanted to hear from some experts out there. Any help is much appreciated.

Taken from https://blogs.vmware.com/vsphere

It’s no surprise that VMware vSphere 6.7 was just released. Hot on the heels of this release was vSphere 6.5U2. If you’re like me, sometimes it is difficult to stay on top of the latest blogs when new releases hit unless you’re polling Twitter daily. To help us all stay in the know, this post will serve as an aggregated list of blogs related to the release of vSphere / vSAN / vRealize Operations 6.7 and vSphere 6.5U2. You’ll also find other product releases have dropped in under the radar such as vRealize Automation 7.4, SRM and vSphere Replication 8.1, vRealize Business for Cloud 7.4, and Log Insight 4.6.

Make yourself a fresh cup of java, and get ready to learn about what’s new and exciting in the VMware world. And if you’re a vExpert who has blogged about a feature of one of these new products, please reach out so I can add your blog post to the list.

Read the full list here - http://blog.thenetworknerd.com/2018/05/05/an-aggregation-of-vsphere-6-7-and-other-product-release-blogs.