Help with Application Infrastructure / Architecture
-
@jn19 said:
You've got me! They've been leasing servers this whole time on a monthly basis, so over the last 3-4 years the company has probably paid $30k each for machines that might have been $7k-8k new. SQL Standard's been bundled into that monthly price at around ~$900/mo/SQL server (dual-proc hexacore machines), so full licenses could have easily been bought for that by now. Plus, the SQL servers are generally at maybe 10-15% CPU usage for the "master" server, and maybe 5% at most for the "slave" server, the latter of which is where the app server and clients pull most of their data. It's just been a lack of good long-term planning, really. I'm trying to help now that I'm here, but it would have been nice to have been here before everything was coded and put into production.
I do wonder what, from a technical standpoint, keeps us from using something like Postgresql, as we do industrial automation, and all of the data acquisition devices we utilize have Linux drivers available. Not that we'd ever have time to rewrite things to switch to it, but I wonder nonetheless.
Doesn't matter if they have Linux drivers... PostgreSQL is the database only, teh application layer can happily run on Windows. Not that that would be my first choice, just saying that using PostgreSQL is over the network and the platform for PostgreSQL itself isn't a factor for other things.
-
@jn19 said:
I'm basically looking for the best ways to improve performance that I can control, i.e. any IIS/SQL/Server 2012 configuration or architecture changes that can be made that will require little to no work on the part of the developer(s). I've got full access to these machines but I have no software development experience, so I just want to do what I can to get things running more smoothly.
The problem is... those aren't the places to fix things and you could drop a million dollars and do effectively nothing. It looks like you have a code problem, throwing money and hardware at it might do nothing.
-
@jn19 said:
@Dashrender said:
From a platform perspective this seems strange to me that your devs are not the ones working to fix these issues.
Unless you are responsible for application performance as well as hardware performance?I concur, but since the only dev on the main application (co-founder/co-owner/boss's boss) is convinced that it's hardware or some simple configuration setting somewhere that's causing the issue, I figure I should go ahead and investigate every avenue of improvement that I can touch!
Oh man... the same guy that caused all of the performance and cost problems already? That doesn't sound like a healthy situation.
-
@Dashrender said:
From a platform perspective this seems strange to me that your devs are not the ones working to fix these issues.
Well, one of the problems with devs causing issues is that often they caused them because they don't really know what they are doing and so can't fix them because they aren't sure why or how it all works.
This is not just suggested by several of the scenarios that the OP mentioned about how they got to where they are, but using SQL Server and IIS for SaaS apps, Redis on Azure, misunderstanding the goals of cloud and such all have the same "not necessarily but realistically... bad developers" problem. It's all "tech I heard about from my first year college professor" who, in turn, was a failed developer that's never worked in the real world and when put together is just a chain of disaster.
Someone in a position of decision making who understands the tech, even a little, looking at the cost associated with all of the Windows Server and SQL Server licenses and that scaling cost as they take the product public would nearly always put a stop to using those technologies before the first line of code was written. Sure, there are exceptions, but few and far between. Those technologies cost a fortune and creating licensing problems that are staggering.
It's hard to tell but it sounds like just one bad decision layered on another and people not willing to take ownership of their mistakes leading to an attempt to throw money (VC money, perhaps) at a problem to hide the fact that the person responsible doesn't want to take ownership of the issue.
-
@jn19 said:
I've got full access to these machines but I have no software development experience, so I just want to do what I can to get things running more smoothly.
I'm not saying quit, but this is when you prep your resume and start looking. I'm not being funny in any way. It's impossible to read the situation from here, but everything that we are hearing is that you have completely incompetent developers and management and they are driving the product into the ground and throwing money away like crazy and are trying to blame IT for their failings. This aren't the kinds of things that are likely to fix themselves down the road. This is the making of a bad situation - most likely just a company failing. But if this is supposed to be software to sell to customers, how will these problems play out at that point? How will paying customers react to being told to "buy faster desktops" or other insane things when the application isn't fast enough for them?
-
So your IIS worker process is blocked, that IS a code issue. Likely the only fix for this is adding IIS workers and throwing threads at the issue. That's about it.
-
@jn19 said:
Session in AcquireRequestState
https://www.leansentry.com/Guide/IIS-AspNet-Hangs
They agree, this is something only the developers can fix. Hardware is not going to solve your problem. Your founder is sending you on a wild goose chase, I'm afraid.
-
@scottalanmiller said:
@jn19 said:
I've got full access to these machines but I have no software development experience, so I just want to do what I can to get things running more smoothly.
I'm not saying quit, but this is when you prep your resume and start looking. I'm not being funny in any way. It's impossible to read the situation from here, but everything that we are hearing is that you have completely incompetent developers and management and they are driving the product into the ground and throwing money away like crazy and are trying to blame IT for their failings.
I was thinking along these same lines.
-
@scottalanmiller said:
@Dashrender said:
From a platform perspective this seems strange to me that your devs are not the ones working to fix these issues.
Well, one of the problems with devs causing issues is that often they caused them because they don't really know what they are doing and so can't fix them because they aren't sure why or how it all works.
I agree that this is just a bad situation for the OP. He's being asked to fix problems that are outside his wheelhouse. He seems to be doing a bang-up job so far, finding SQL tools, etc that can find timing problems, etc. But ultimately this comes down to the code. Something that nothing short of time will allow the OP to learn well enough to solve the real issues.
He's already shown that the hardware is yawning - it's not busy at all (at least the stats we've been shown - for example, we haven't been told how deep the drive queues are).
I don't think tuning IIS apps as part of an IT person's typical job. That's a dev's job.
-
@Dashrender said:
@scottalanmiller said:
@jn19 said:
I've got full access to these machines but I have no software development experience, so I just want to do what I can to get things running more smoothly.
I'm not saying quit, but this is when you prep your resume and start looking. I'm not being funny in any way. It's impossible to read the situation from here, but everything that we are hearing is that you have completely incompetent developers and management and they are driving the product into the ground and throwing money away like crazy and are trying to blame IT for their failings.
I was thinking along these same lines.
It's harsh and I feel bad for how often this ends up being a suggestion but... often these questions come up because of situations that IT cannot fix and are endemic to the organization.
-
@Dashrender said:
I don't think tuning IIS apps as part of an IT person's typical job. That's a dev's job.
A little of both. But it looks like the issue here is all code, not IIS itself.
-
Very good points all around! And, to be honest, touching up the resume is definitely good advice at this point.
My boss (IT director) and I are in agreement about what needs to be done to move toward long-term success, scalability, and performance, so we hope to meet with our CEO, point out the application & IIS issues, and pitch a shift toward at least testing with PostgreSQL. If he agrees with the fact that these performance, code, and licensing problems are, and will only continue to be, huge yet surmountable issues that will make or break his company, then we'll let him know that we're happy to help with whatever we can. -
Just following up, how is everything going over there @jn19
