So, after ten years working for the same company (SMB, ~60 employees), I'm a bit lost.

This was my first IT job, and I have been growing by experience, but without a clear guidance.

My boss is focused on the development side (.NET development, SQL queries, integration with 3rd party apps and services, etc), so I always have been on my own.

I'm the "sysadmin" and also technical support for our users, so a jack-of-all-trades: virtualization, networking, storage management, OS deployment, software/apps deployment... You get the idea.

When I joined the company, there was a very basic IT infrastructure. I tried to add value to the business. I know I made lots of errors despite my best intentions, but I could see that I was improving the company, and I was happy for it.

But as the complexity growed, also did it our technnical debt. I simply don't have the time or the capacity to read and learn about all the technologies and keep on top of them. I have a never ending "to do" list of ideas, best practices to apply, problems to solve, documentation to write... but very little time/oportunity to take real action.

Also, I have a bad time trying to identify the recommended IT solutions for a small company like ours, and I'm sure I'm oversizing and overcomplicating our infrastructure "just in case". As we don't have metrics, we are shooting in the dark, following vendor recomendations instead of buying what we really should.

I really care about this company. It's a great team, and I really enjoy working here. They trust out department, the projects are aproved without almost any question and management is ussually on our side.

Since the last year, I feel really overwhelmed, without a clear direction, often blocked and not knowing where should I start or what should I do.

This post really nails it: http://www.smbitjournal.com/2013/02/the-smallest-it-department/

I'm trying to get in touch with IT pros in similar conditions to exchange knowledge, experiences and ideas, because I think that we are facing very similar challenges.

Other than that, could you suggest me how can I improve my situation?

Thank you for your time reading this wall of text.

Hi!

I'm Manu, a jack-of-all-trades working for a small manufacturing company.

Learning config management tools and scripting to automate our small environment (VMware virtualization, Windows and Linux servers, about 50 Windows clients...)

Based in Spain, enjoying the nice weather and delicious food.

Nice to meet you all!

@dbeato said in Seeking how to improve:

Well to start, you have begun the process which is an awesome sign, and you are here asking for help too. So we can help you and believe me you will learn here a lot out of your comfort zone.

You should be able to learn using LABs and even online testing. But if you would like to evolve your systems you should look at what you have been using for a while now, like what Network, Hardware and Software do you have deployed now? What should be changed, what can you improve? How can I grow my skills and it what areas?

So share with us that and we can help.

Thank you!

I will start a really big technology refresh in a few weeks. Please, take a look to my @Dashrender reply...

I think that time management and priorization are my biggest problems at this moment. Too much to do and learn. Not enougt time for everything.

The lab is a great idea. On my todo list for sure (Hyper-V on Windows 10 rocks for this)

But is really great having a community like this. Glad to have found you, guys!

@scottalanmiller said in Seeking how to improve:

@manheraz said in Seeking how to improve:

Our biggest project at this time is a total technology "refresh":

• Migrate from VMware vSphere Essentials to Hyper-V Server 2016
• Prepare to pass out the SAN (we have two ESXi diskless servers connected to an EMC array, going to Hyper-V with local storage)
• New network configuration (adding VLANs for a DMZ, a testlab, management interfaces... nothing fancy)
• New Active Directory domain/forest (we have a very old inherited domain that doesn't follow best practices)
• Upgrade from Windows Server 2008 R2 to Windows Server 2016
• Deploy standarized Windows (client and server) images using Microsoft Deployment Toolkit + WDS
• Manage software in client computers with Chocolatey
• Document everything (going with Dokuwiki)
• Try to automate everything as much as possible with Puppet (using The Foreman) and PowerShell/PowerShell DSC

That's a lot, but sounds like a good, solid list of stuff to be doing.

Thanks, Scott.

Also, thank you for your posts on smbitjournal.com. They helped me a lot!

@dashrender said in Seeking how to improve:

How about starting by sharing a list of your to dos, etc.

Well, that's one of my problems, trying to bite more than I can reallistically chew.

I have hundreds of items in my to do list, 90% "self imposed". As I keep learning, I keep adding ideas to improve the current infrastructure.

Then, I have a tought time trying to prioritize my work (too much items to process leads to something like analysis paralysis)

Also, trying to learn new technologies requires time, and that makes me question: should I improve my knowledge in the technologies in use at the moment or should I focus in learning more useful things for a near future?

Example: Should I improve my VMware skills (currently in production) or should I invest learning Hyper-V, PowerShell, PowerShell DSC, Puppet to be able to use that in a few months?

Our biggest project at this time is a total technology "refresh":

• Migrate from VMware vSphere Essentials to Hyper-V Server 2016
• Prepare to pass out the SAN (we have two ESXi diskless servers connected to an EMC array, going to Hyper-V with local storage)
• New network configuration (adding VLANs for a DMZ, a testlab, management interfaces... nothing fancy)
• New Active Directory domain/forest (we have a very old inherited domain that doesn't follow best practices)
• Upgrade from Windows Server 2008 R2 to Windows Server 2016
• Deploy standarized Windows (client and server) images using Microsoft Deployment Toolkit + WDS
• Manage software in client computers with Chocolatey
• Document everything (going with Dokuwiki)
• Try to automate everything as much as possible with Puppet (using The Foreman) and PowerShell/PowerShell DSC

So, I will be really busy for months...

@dashrender said in Seeking how to improve:

@manheraz said in Seeking how to improve:

@dashrender said in Seeking how to improve:

How about starting by sharing a list of your to dos, etc.

Well, that's one of my problems, trying to bite more than I can reallistically chew.

I have hundreds of items in my to do list, 90% "self imposed". As I keep learning, I keep adding ideas to improve the current infrastructure.

Then, I have a tought time trying to prioritize my work (too much items to process leads to something like analysis paralysis)

Also, trying to learn new technologies requires time, and that makes me question: should I improve my knowledge in the technologies in use at the moment or should I focus in learning more useful things for a near future?

Example: Should I improve my VMware skills (currently in production) or should I invest learning Hyper-V, PowerShell, PowerShell DSC, Puppet to be able to use that in a few months?

Our biggest project at this time is a total technology "refresh":

• Migrate from VMware vSphere Essentials to Hyper-V Server 2016
• Prepare to pass out the SAN (we have two ESXi diskless servers connected to an EMC array, going to Hyper-V with local storage)
• New network configuration (adding VLANs for a DMZ, a testlab, management interfaces... nothing fancy)
• New Active Directory domain/forest (we have a very old inherited domain that doesn't follow best practices)
• Upgrade from Windows Server 2008 R2 to Windows Server 2016
• Deploy standarized Windows (client and server) images using Microsoft Deployment Toolkit + WDS
• Manage software in client computers with Chocolatey
• Document everything (going with Dokuwiki)
• Try to automate everything as much as possible with Puppet (using The Foreman) and PowerShell/PowerShell DSC

So, I will be really busy for months...

As already mentioned by @dafyre, this is a pretty good list. I'd say because of this list, you've already dictated the path you need to take.

Should you bother with VMWare? Nope, you're company is moving away from it, it's useless knowledge at this point.

Your project list looks good. For each part of it, start a thread. It might be a good idea to start a thread first as a project over view - list the current environment, where you plan to go and why you choose that path. I'm sure this group will weigh in and perhaps help you tweak the path.

Thanks. Starting a thread for every project is a great idea.

I'm going to use Dokuwiki to plan and document my projects. I usually document how I did something, but I always forget to document why I plan it/did it that way.

That kind of details could be valuable when posting a new thread.

@scottalanmiller said in Seeking how to improve:

@manheraz said in Seeking how to improve:

@scottalanmiller said in Seeking how to improve:

@manheraz said in Seeking how to improve:

Our biggest project at this time is a total technology "refresh":

• Migrate from VMware vSphere Essentials to Hyper-V Server 2016
• Prepare to pass out the SAN (we have two ESXi diskless servers connected to an EMC array, going to Hyper-V with local storage)
• New network configuration (adding VLANs for a DMZ, a testlab, management interfaces... nothing fancy)
• New Active Directory domain/forest (we have a very old inherited domain that doesn't follow best practices)
• Upgrade from Windows Server 2008 R2 to Windows Server 2016
• Deploy standarized Windows (client and server) images using Microsoft Deployment Toolkit + WDS
• Manage software in client computers with Chocolatey
• Document everything (going with Dokuwiki)
• Try to automate everything as much as possible with Puppet (using The Foreman) and PowerShell/PowerShell DSC

That's a lot, but sounds like a good, solid list of stuff to be doing.

Thanks, Scott.

Also, thank you for your posts on smbitjournal.com. They helped me a lot!

Have you started watching the SAMIT videos, too? It's like SMBITJournal, but on youTube

Will do! Thanks...!

@dashrender said in Seeking how to improve:

hmmm.. well, if you get to choose - I guess the first thing I'd ask is - are there any problems currently that need to be fixed, and will doing one of these items fix it?

If the answer is no - then I would likely start with VM conversion to Hyper-V. How will you do this? new hardware?

Great advice!

Well, I think the most urgent need is to automate OS client deployment. I have lots of computers to reinstall, and it simply takes too much time at the moment. Also, we are using Windows 10 Pro 1511, and I have to upgrade as soon as posible (we are out of support, Microsoft extended the support for 1511 to Education and Enterprise licenses)

But I want every reinstalled computer to be a member of the new domain, so this is the order I was thinking about:

1. Deploy Hyper-V server 2016 in one of the ESXi servers
2. Deploy new domain controller and forest in the Hyper-V server and join the Hyper-V server and a computer to the new domain (that should allow me to manage Hyper-V using the Microsoft tooling)
3. Migrate all the VMware virtual machines to Hyper-V (from the EMC array to a slow local storage - SATA)
4. Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)
5. Install Hyper-V Server 2016 in the last ESXi server
6. Deploy Microsoft Deployment Toolkit and Windows Deployment Services and integrate them in the new domain so I can automate the Windows client and server deployment
7. Deploy a CentOS 7 server with The Foreman so I can automate Linux deployment and start configuring all the clients and servers with Puppet + PowerShell/PowerShell DSC + Chocolatey
8. Deploy Dokuwiki so I can document everything (using Dokuwiki on a Stick at the moment, so steps 1 to 7 will be documented on this local Dokuwiki and later migrated to the production Dokuwiki)
9. Deploy Veeam Backup to protect the new virtual infrastructure and critical endpoints (not sure if virtual machine or physical server, I want to use our old LTO 3 tapes as a last resort offsite offline write protected backups) - Sorry, this wasn't in the original list. Forgot about it.

As you can see, I will be reusing my ESXi servers. I have a third server (old Dell R515), but it doesn't have enought resources to host all the virtual machines during the migration.

Sorry, I think this should be in the new planning post. I can move it if you want to take the discussion there...

@ramblingbiped said in Seeking how to improve:

Before you take on any new work I would recommend reading "Time Management for System Administrators" by Tom Limoncelli.

https://www.goodreads.com/book/show/376236.Time_Management_for_System_Administrators

Put into practice the principles discussed in the book and start organizing your time.

A favorite quote of mine as of late is: "How you do anything is how you do everything". Don't take on a project or task if you can not afford to give it the time, attention, and resources that it deserves. Failing is okay, but setting yourself up for failure is a waste of time and potential. Always keep a focus on attention to detail and maximize your time and efforts.

Hey! Thank you very much for the recommendation.

Tom Limoncelli has really nice books. I'm a fan of "The practice of system and network administration, 3rd ed".

Some projects are ideas from that book.

Will take a look for sure.

@dashrender said in Seeking how to improve:

@manheraz said in Seeking how to improve:

@coliver said in Seeking how to improve:

@manheraz said in Seeking how to improve:

@dashrender said in Seeking how to improve:

@manheraz said in Seeking how to improve

4. Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)

How are you going to do that? Are you buying another SAN? If not, you don’t have HA.

I have no idea if SMB 3 is worth us My or not... other possibilities ISCSI.

Well, I was going to use this:

SMB3 allows deduplication and compression (not available if using iSCSI, and the array doesn't have enought storage capacity for our needs)

That's from the "EMC VNXe3200 HIGH AVAILABILITY" white paper.

Burn that thing in a fire. Low end crap Dell bundled with their servers to promote fake-HA.

What about using it as Veeam backup target?

I got the idea from here: https://www.storagecraft.com/blog/making-the-best-of-your-inverted-pyramid-of-doom/

That could be fine, but I'd probably still want to have another backup besides this - i.e. replicated somewhere else...
Also, when you repurpose it.. RAID 6 that guy for max storage followed by reasonable safety.

Yeah. That would be just one backup target.

I was thinking about two backup targets and one replication target in a DR site.

Also, trying to use our old LTO 3 tapes as a last resort backup.

Thanks for the RAID 6 suggestion. Good idea.

@dashrender said in Seeking how to improve:

Sign up for DPACK. Install it and get a baseline on what you're current workloads are doing. Run it for a week, the longer the better. For example, you might have a process that only runs once a month, and running for a week could miss that process and it's requirements.

This will help you know what you need to shoot for at a minimum.

Thank you very much!

Just what I needed.

@coliver said in Seeking how to improve:

@manheraz said in Seeking how to improve:

@dashrender said in Seeking how to improve:

@manheraz said in Seeking how to improve

4. Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)

How are you going to do that? Are you buying another SAN? If not, you don’t have HA.

I have no idea if SMB 3 is worth us My or not... other possibilities ISCSI.

Well, I was going to use this:

SMB3 allows deduplication and compression (not available if using iSCSI, and the array doesn't have enought storage capacity for our needs)

That's from the "EMC VNXe3200 HIGH AVAILABILITY" white paper.

Burn that thing in a fire. Low end crap Dell bundled with their servers to promote fake-HA.

What about using it as Veeam backup target?

I got the idea from here: https://www.storagecraft.com/blog/making-the-best-of-your-inverted-pyramid-of-doom/

I don't have ANY experience with virtual SANs, but what about StarWind Virtual SAN Free?

I have four Dell SSD 120 GB (server class) available. Our R530 have eight HDD slots each.

Maybe we can go with 2 x 120 GB SSD for caching and 6 x NL-SAS disks for storage on each server. What do you think?

Sorry, I never used tiered storage or virtual SAN products, just started looking at this.

Also, I have no idea about how many IOPS do we need. We have this storage array because it was the recommended solution by our service provider. I know, very bad move. They want to sell services and equipments.

@dashrender said in Seeking how to improve:

@manheraz said in Seeking how to improve:

@scottalanmiller said in Seeking how to improve:

@dashrender said in Seeking how to improve:

@manheraz said in Seeking how to improve

4. Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)

How are you going to do that? Are you buying another SAN? If not, you don’t have HA.

I don't see the model, but if that is a high end EMC, it can do HA. It's not cost effective for it most likely, but it can do it. It's not that single point of failures can't do HA, it's that they can't do it optimally.

Sorry, forgot to mention the model. We have an EMC VNXe3200 with 12 HD SAS 10K configured as two RAID 5 and two RAID 10 LUNs.

OMG - did you said spinning rust in RAID 5? What is the size of those arrays? Remember, at 12 TB for consumer, and 120 TB at business class drives, these arrays reach a near 100% failure rate due to URE during a resilver.

Yes, I was really concerned about that. That was not my design, it was configured by a the same company we bought the array. They are certified by EMC, and it was the first time I was looking at their products, so I kept my mouth shut.

Also, the array has 7.2 TB raw.

@coliver said in Seeking how to improve:

@scottalanmiller said in Seeking how to improve:

@coliver said in Seeking how to improve:

@manheraz said in Seeking how to improve:

Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)

I'm curious as to what EMC array you have? Some of the lower end models are junk and shouldn't be considered.

I'm surprised that it offers SMB3.

Pretty much. The VNXe line was big on "Unified Storage" but it blew up in my face one too many times for it to be considered anything but a toy system.

Well, bad news for me...

We had no problems with the array in two years, but we can't afford a total failure of the array at this moment, so time to look for better options.

@ramblingbiped said in Seeking how to improve:

Before you take on any new work I would recommend reading "Time Management for System Administrators" by Tom Limoncelli.

https://www.goodreads.com/book/show/376236.Time_Management_for_System_Administrators

Put into practice the principles discussed in the book and start organizing your time.

A favorite quote of mine as of late is: "How you do anything is how you do everything". Don't take on a project or task if you can not afford to give it the time, attention, and resources that it deserves. Failing is okay, but setting yourself up for failure is a waste of time and potential. Always keep a focus on attention to detail and maximize your time and efforts.

Hey! Thank you very much for the recommendation.

Tom Limoncelli has really nice books. I'm a fan of "The practice of system and network administration, 3rd ed".

Some projects are ideas from that book.

Will take a look for sure.

@scottalanmiller said in Seeking how to improve:

@coliver said in Seeking how to improve:

@manheraz said in Seeking how to improve:

Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)

I'm curious as to what EMC array you have? Some of the lower end models are junk and shouldn't be considered.

I'm surprised that it offers SMB3.

Yep, it does. I was a bit surprised too when I discovered it...

@coliver said in Seeking how to improve:

@manheraz said in Seeking how to improve:

Reconfigure the EMC array as a high availability SMB 3 file server and move the Hyper-V virtual machines to the array (I have to use the EMC array until it's out of suport - 2020)

I'm curious as to what EMC array you have? Some of the lower end models are junk and shouldn't be considered.

It's an EMC VNXe3200, so yeah, low end model

@scottalanmiller said in Seeking how to improve:

@manheraz said in Seeking how to improve:

(I have to use the EMC array until it's out of suport - 2020)

That's sunk cost / technical debt. You should run the numbers. It should be an IT decision whether it makes sense to use or not, that there is or isn't support would be an "in the weeds" factor that would lead to the final decision, but is just part of the equation.

Yeah, I discovered we have an IPOD when we already had the array.

When we bought the current hypervisors (2 Dell R530, single E5 CPU, 128 GB RAM, 4 GbE NICs + 2 10 GbE NICs for the SAN) and the array, management told us that the new equipment should last five years at least.

That's the reason we had to use the array until it's out of maintenance.

The array is half empty, so we can add disks if needed, but that is really expensive. It's a lot cheaper going with local storage (and maybe more performant), so that's a reason I can use to change the situation in addition to the risk of having a critical SPOF.