I’m looking to my fellow users for help here, and I’ve always been impressed with the breadth and depth of the knowledge that you’ve shared in this forum. My background’s in general educational/ corporate/small business IT support, desktop deployment with MDT, and basic Windows server setups with Hyper-V. I’ve also helped manage a vSphere 5.5 installation with around 30 hypervisors before, so I’m comfortable with that as well, but have more experience with Hyper-V. My new job was initially focused on the same types of tasks, but now management has chosen to discontinue our IT support services and concentrate all of our resources on our core SaaS applications.

The problem is that while our core web applications have tremendous functionality, they have terrible performance and even worse security. The servers were basically set up by the developers years ago as a test environment when initially developing the core business, but then they quickly became the production servers once everything was working and able to generate a profit.

Again, I’m no application architect, but I know that SQL server shouldn’t be installed on the system drive, that the application server probably needs more RAM than the database servers, that field client applications shouldn’t bypass the application server and communicate directly with database servers, and that everything should be behind a firewall and only accessible via VPN connection or federated identity, not RDP. There’s no Active Directory infrastructure for these machines, so there are individual local accounts on each machine with random insecure credentials, and the list just goes on. All of this was set up before I worked here, and I’ve been lobbying to get things fixed ever since. We’d hired an employee that was experienced in infrastructure architecture but he had to move back overseas for family reasons, and the task of setting up an entirely new, performant application infrastructure has been passed to me. I like doing the research, but we’re a small company, and I’m only afforded so much time to come up with a solution. At previous jobs I had network and system engineers that dealt with the big infrastructure issues. It’d be one thing if I was just trying to get things running better at our current load, but within the next 2-3 months we’ve got a large client coming on that will make usage of our system increase by a factor of at least 10.

As it stands, I need help with everything from hardware sizing to SQL licensing. I don’t know the best way to license SQL 2014 for our purposes, and I know that our lead dev would prefer the Enterprise version for some of the extra features, but I think the costs will be too prohibitive.

Right now we’re running IIS 8 on Server 2012, with a SQL 2012 backend. The application’s in .NET 4.6.1, and we’ll be moving it to IIS 8.5 soon. Unfortunately, our entire setup is dependent upon IIS, .NET and SQL, and I don’t think that will be changing anytime soon, if ever.
Our servers interact with several types of clients:

Mobile Devices – Mobile view of web application, plus native Android & iOS apps that tie into a subset of the web application’s functionality. There are generally 30-40 connections of this type at any given time.

• Site Servers – These machines ingest information from SCADA systems & issue automation commands. Each machine runs a local .NET app that has a SQL 2008R2 backend. Almost all of the site servers utilize cellular connections, and traffic volume has been an issue. There are around 80 sites like this that send and receive data to/from our servers every 30 seconds to a minute.
• Web Users – The web application is used by another 30-40 users at any given time.
  Response times on the web application are often painfully slow. Some queries can take over 100 seconds. I’ve run a lot of SQL health scripts from Brent Ozar’s site and those have helped a bit, but I don’t think the speed issue lies with SQL. IIS seems to be the culprit.

CURRENT SERVER SPECIFICATIONS:
We utilize 3 physical servers at this time, along with a Redis Instance in Azure.

Application Server:
Dual Xeon E5645 (6-core) @ 2.4GHz
64GB RAM
System Drive – Intel DC3500 – 600GB
SSD Storage – Intel DC S3500 SSD–800GB
SATA Storage -- Seagate 7200 rpm – 3TB
Network – 1Gbps up/down (usually at 1-3% utilization)
Redis on Windows – Caching App server requests

Database Server – Master:
Dual Xeon E5-2630 (6-core) @ 2.3 GHz
128GB RAM
System Drive – Intel DC S3500 – 600GB
SATA Storage -- Seagate 7200 rpm – 3TB
Network – 1Gbps up/down (usually at 1% utilization)
Redis on Windows – Pub/Sub relationship with Azure Redis – High-volume ASP requests
Transactional Replication Publisher

Database Server – Slave:
Dual Xeon E5-2630 (6-core) @ 2.3 GHz
128GB RAM
System Drive – Micron M510DC SSD – 960GB
Network – 1Gbps up/down (usually at 1% utilization)
Transactional Replication Subscriber
Azure Redis – Standard Tier -13GB (I have no idea why we have a Redis instance in Azure, but I imagine that it’s a speed bottleneck as well. I don’t know how to measure the response time between our servers and the Azure Redis instance, though.)

My proposed hardware is along these lines:

3 Hypervisors with the following specs:
2 x 2.4GHz Octa-Core E5-2630 v3 Haswell Xeon
256GB RAM
Boot Drive - 160 GB SSDs in RAID1
VM drives – Intel DC S3500 or NVMe drives of around 900GB in RAID1
Storage drives – 4 x 6TB SATA in RAID 10 w/ BBU
Server 2012R2 / SQL 2014 Std or Ent

1U Quad-core servers for pfSense & HAProxy.

Here are some basic diagrams of our current setup along with my proposed setup, which is all based heavily on Stack Exchange's setup:

These are all little more than guesses, as I really don’t know the best way to set up a fast and secure IIS / .NET / SQL infrastructure. Is virtualization a bad idea for this type of setup? My thoughts were that the advantage to having 3 or so high-performance hypervisors would be that we could more easily migrate things to better hardware as the need arises, and that it should run nearly as fast as a bare-metal server as long as we’re not putting both databases/app servers/redis instances on just one box, causing resource contention.

Any help you can give would be greatly appreciated.

@scottalanmiller

You've got me! They've been leasing servers this whole time on a monthly basis, so over the last 3-4 years the company has probably paid $30k each for machines that might have been $7k-8k new. SQL Standard's been bundled into that monthly price at around ~$900/mo/SQL server (dual-proc hexacore machines), so full licenses could have easily been bought for that by now. Plus, the SQL servers are generally at maybe 10-15% CPU usage for the "master" server, and maybe 5% at most for the "slave" server, the latter of which is where the app server and clients pull most of their data. It's just been a lack of good long-term planning, really. I'm trying to help now that I'm here, but it would have been nice to have been here before everything was coded and put into production.

I do wonder what, from a technical standpoint, keeps us from using something like Postgresql, as we do industrial automation, and all of the data acquisition devices we utilize have Linux drivers available. Not that we'd ever have time to rewrite things to switch to it, but I wonder nonetheless.

Hello everyone,

I’m working on a project to migrate our hosted IIS/.NET/SQL 2012 environment into Azure, and have already done most of the work towards getting the databases compatible with Azure SQL V12. My main area of concern right now is how to best perform and then automate the process of getting backups of our production databases restored into the corresponding Azure SQL DBs each night.

I’ve manually migrated a copy of the main database into Azure already, but I don’t know the best way to go about updating its data from the hosted SQL 2012 server on a regular schedule so that it’s not useless for testing purposes. I’d like to avoid doing a full restore each night, and I’m not sure if there’s a way to do this with only the change deltas. I'm sure there's a way with something like Azure Backup/ blob storage / Powershell, etc., but I haven't come across it yet.

Any help is greatly appreciated.

Thanks!

Hi Everyone,

I'm just looking for some guidance as to how best to implement Active Directory as well as better manageability in general for my company. We have no on-premises servers or AD right now, and aren't using AD in Azure, either. We use GSuite for our email provider, but we also have O365 Dynamics Online accounts under a different domain name, since it wouldn't let me use the domain name we'd already set up with GSuite. I have experience with upgrading pre-existing on-premises domains, as well as setting up an AD-less Azure Infrastructure from scratch, but I've never set up a new on-prem domain nor extended one from Azure to off-premises client devices. Here's some background info on our current systems:

Main office:

• 100MB symmetrical fiber internet
• pfSense firewall/router
• Dell PowerConnect gigabit PoE switch
• Ubiquiti APs
• Mix of desktops and laptops
• 15-20 full-time users at this location

Satellite office:

• 200MB down/50MB up cable internet
• Netgear consumer firewall/router/access point
• All laptops
• 5-10 full-time users at this location

Field technicians:

• Their own home/cellular ISP and routing solutions
• All laptops
• 10-20 full-time technicians that may only visit an office location a few times per year

Industrial Automation PCs:

• Verizon or AT&T Wireless data plans on public internet (Usually 10GB/mo allocated per modem)
• Sierra Wireless RV50 modem/gateway/router devices
• 120 IPCs are scattered over hundreds of square miles and run a mix of Win7Pro and Win10Ent IoT LTSB 2016

Azure Environment:

• Azure SQL VMs
• Service VMs
• Azure app services
• Redis cache, etc.

I'd like to be able to manage all of these devices in AD, if possible, including activating BitLocker and applying GPOs on all of them. Some groups of users, like DevOps and Developers, will need access to the Azure Infrastructure and the Industrial Automation PCs, while other groups, like Accounting, won't need any Azure or Industrial PC access. SSO would be nice, but probably won't be possible until we migrate our email to O365. Bandwidth usage is only a concern on the IPCs, as cellular data overages aren't cheap. We're open to buying a decent hypervisor for each office location if it's necessary to manage our resources successfully.

What's your take on the best way forward? Thanks for any help you can provide!

Very good points all around! And, to be honest, touching up the resume is definitely good advice at this point.
My boss (IT director) and I are in agreement about what needs to be done to move toward long-term success, scalability, and performance, so we hope to meet with our CEO, point out the application & IIS issues, and pitch a shift toward at least testing with PostgreSQL. If he agrees with the fact that these performance, code, and licensing problems are, and will only continue to be, huge yet surmountable issues that will make or break his company, then we'll let him know that we're happy to help with whatever we can.

@matteo-nunziati

You're very correct about the automation PCs--they're a horror show as far as security goes.
They autologon with admin privileges, and they rarely get updates due to bandwidth and manageability issues. To be clear, the automation PCs don't actually need to be joined to our organization Active Directory, and it'd probably be best if they weren't. If there's a different solution available to monitor/patch/secure them, I'm all for it. Unfortunately, we're stuck with Windows, as a lot of the automation tools we have to interface with only have Windows drivers and utilities available.

Hi Everyone,

I'm just looking for some guidance as to how best to implement Active Directory as well as better manageability in general for my company. We have no on-premises servers or AD right now, and aren't using AD in Azure, either. We use GSuite for our email provider, but we also have O365 Dynamics Online accounts under a different domain name, since it wouldn't let me use the domain name we'd already set up with GSuite. I have experience with upgrading pre-existing on-premises domains, as well as setting up an AD-less Azure Infrastructure from scratch, but I've never set up a new on-prem domain nor extended one from Azure to off-premises client devices. Here's some background info on our current systems:

Main office:

• 100MB symmetrical fiber internet
• pfSense firewall/router
• Dell PowerConnect gigabit PoE switch
• Ubiquiti APs
• Mix of desktops and laptops
• 15-20 full-time users at this location

Satellite office:

• 200MB down/50MB up cable internet
• Netgear consumer firewall/router/access point
• All laptops
• 5-10 full-time users at this location

Field technicians:

• Their own home/cellular ISP and routing solutions
• All laptops
• 10-20 full-time technicians that may only visit an office location a few times per year

Industrial Automation PCs:

• Verizon or AT&T Wireless data plans on public internet (Usually 10GB/mo allocated per modem)
• Sierra Wireless RV50 modem/gateway/router devices
• 120 IPCs are scattered over hundreds of square miles and run a mix of Win7Pro and Win10Ent IoT LTSB 2016

Azure Environment:

• Azure SQL VMs
• Service VMs
• Azure app services
• Redis cache, etc.

I'd like to be able to manage all of these devices in AD, if possible, including activating BitLocker and applying GPOs on all of them. Some groups of users, like DevOps and Developers, will need access to the Azure Infrastructure and the Industrial Automation PCs, while other groups, like Accounting, won't need any Azure or Industrial PC access. SSO would be nice, but probably won't be possible until we migrate our email to O365. Bandwidth usage is only a concern on the IPCs, as cellular data overages aren't cheap. We're open to buying a decent hypervisor for each office location if it's necessary to manage our resources successfully.

What's your take on the best way forward? Thanks for any help you can provide!

@JaredBusch Just what I was looking for. Thanks a lot, Jared!

Hello everyone,

I’m working on a project to migrate our hosted IIS/.NET/SQL 2012 environment into Azure, and have already done most of the work towards getting the databases compatible with Azure SQL V12. My main area of concern right now is how to best perform and then automate the process of getting backups of our production databases restored into the corresponding Azure SQL DBs each night.

I’ve manually migrated a copy of the main database into Azure already, but I don’t know the best way to go about updating its data from the hosted SQL 2012 server on a regular schedule so that it’s not useless for testing purposes. I’d like to avoid doing a full restore each night, and I’m not sure if there’s a way to do this with only the change deltas. I'm sure there's a way with something like Azure Backup/ blob storage / Powershell, etc., but I haven't come across it yet.

Any help is greatly appreciated.

Thanks!

Very good points all around! And, to be honest, touching up the resume is definitely good advice at this point.
My boss (IT director) and I are in agreement about what needs to be done to move toward long-term success, scalability, and performance, so we hope to meet with our CEO, point out the application & IIS issues, and pitch a shift toward at least testing with PostgreSQL. If he agrees with the fact that these performance, code, and licensing problems are, and will only continue to be, huge yet surmountable issues that will make or break his company, then we'll let him know that we're happy to help with whatever we can.

@Dashrender said:

From a platform perspective this seems strange to me that your devs are not the ones working to fix these issues.
Unless you are responsible for application performance as well as hardware performance?

I concur, but since the only dev on the main application (co-founder/co-owner/boss's boss) is convinced that it's hardware or some simple configuration setting somewhere that's causing the issue, I figure I should go ahead and investigate every avenue of improvement that I can touch!

@DustinB3403

I'm basically looking for the best ways to improve performance that I can control, i.e. any IIS/SQL/Server 2012 configuration or architecture changes that can be made that will require little to no work on the part of the developer(s). I've got full access to these machines but I have no software development experience, so I just want to do what I can to get things running more smoothly.

@scottalanmiller

Oh, I agree that it should run quite well on the current hardware, given the right setup. Here's some info from one of the hang reports in LeanSentry, which has been a pretty handy tool for IIS analytics:

[img]http://i.imgur.com/djV1cRb.png[/img]
[img]http://i.imgur.com/c0NCrlb.png[/img]

The blocked request location in this instance was a "Session in AcquireRequestState."

@scottalanmiller

You've got me! They've been leasing servers this whole time on a monthly basis, so over the last 3-4 years the company has probably paid $30k each for machines that might have been $7k-8k new. SQL Standard's been bundled into that monthly price at around ~$900/mo/SQL server (dual-proc hexacore machines), so full licenses could have easily been bought for that by now. Plus, the SQL servers are generally at maybe 10-15% CPU usage for the "master" server, and maybe 5% at most for the "slave" server, the latter of which is where the app server and clients pull most of their data. It's just been a lack of good long-term planning, really. I'm trying to help now that I'm here, but it would have been nice to have been here before everything was coded and put into production.

I do wonder what, from a technical standpoint, keeps us from using something like Postgresql, as we do industrial automation, and all of the data acquisition devices we utilize have Linux drivers available. Not that we'd ever have time to rewrite things to switch to it, but I wonder nonetheless.

@scottalanmiller

I wondered the exact same thing. It was set up that way by our lead dev, (who also co-founded our company), thinking that we were going to migrate EVERYTHING to Azure, but then realized upon testing that it's much slower there. I guess he's got a lot of API calls pointing there and either doesn't want / hasn't had time to move things local. IT wasn't consulted about this beforehand, so we became stuck with it after the fact. This is the only part of our setup that wasn't in place prior to my arrival, and I didn't know about it until a few weeks ago. I just thought we had part of one of our websites in Azure.