This happened a couple of years ago, but it is still relevant because I know of at least one place where it is still happening even though Microsoft has fixed the issue that initially caused this hack to be put in place in the first place.
Problem In Production – Oh NO!
Here’s the problem we were seeing. We had several large PDF files that the client wanted to put up on the site so that their customers could download them. The problem that we were seeing is that even though the site worked fine under development and QA, the site seemed to consume a lot of memory under load.
Another symptoms we saw was that the file download took a lot of time to start, or timed out completely. But the main issue was the memory because when the memory was consumed, the site would restart, and because it was a worker process crash that caused it, the restart took the site down. This is not something you want to see happen when you are working with a client with world wide exposure. A client who, if I told you the name, I’m sure you would recognize.
In fact, it was the site not responding, or crashing, that first alerted us to the problem. So, as soon as we knew the site wasn’t responding, I’d log onto the server and fire up task manager to see what was going on.
Server Management Tip
One of the things I did early on managing the servers, is that I set up an entirely separate app pool for each of the web sites we are hosting. This way, we can monitor the memory and CPU activity of each one independently. For those of you who are interested, the way to make the pool name show up in task manager is to run the pool in ApplicationPoolIdentity, a setting you can get to through “Advanced Settings…” If you are only hosting one site, you shouldn’t need to do this, but in my case, with multiple sites, this told me a lot about what was going on with the sites and ultimately helped me track down what the problem was.
By the way, you should always run each of your sites under a separate application pool so that when one site is having trouble, every site on your server doesn’t have trouble. Imagine having an issue and not even knowing for sure which site was causing the trouble. It’s bad enough not knowing what page is causing the problem.
It Is a Memory Issue. Now What?
OK. So, I could see the memory increasing as soon as the site came back up. So, now to try to track down what was causing memory issues.
I tried various things along the way, none of them shed any light at all on the problem. I even got dotMemory from JetBrains to see if I could find anything. A memory leak maybe? No, none of my code had even a small leak.
The Fix Discovered
And then I finally stumbled on the issue. I can’t remember what it was that we installed that needed this, but what it did was change our web.config file so that it had a section in it that looks like this:
<configuration> ... lots of other configuration stuff ... <system.webServer> <modules runAllManagedModulesForAllRequests="true">
Make sure that runAllManagedModulesForAllRequest is either set to false, or doesn’t exist in your web.config file. If you need it to be turned on for some valid reason that I am not thinking of, at least put your static files on another site where this can be turned off.
Moral Of The Story
Now, the question that we should be asking now that we’ve figured it all out is, why wasn’t this caught earlier? When a site goes up, isn’t all of the functionality, including all of the links to all of the images, files to download, and other pages on the site and off the site supposed to be tested?
Yes, of course they are, but if you haven’t at least written down where all of this is, if you don’t have a systematic way of testing EVERYTHING, you WILL end up with these kind of errors once you put the site live. This is to say nothing of load testing along the way. This can all be prevented.
Other post in Testing
- Selenium with NUnit Presentation - February 17th, 2014
- “Debbie-Done” - March 4th, 2014
- When You Really Need All Of Your NUnit Test In One Class - March 18th, 2014
- TDD Isn’t All About Testing - March 25th, 2014
- Automated Web Application Functional Testing - April 1st, 2014
- What Not To Test - April 9th, 2014
- Don’t Comment Out That Test - April 24th, 2014
- The Proper Function of QA - May 1st, 2014
- TDD Saves Time – A Story - May 22nd, 2014
- Running Selenium In Parallel With Any .NET Unit Testing Tool - July 31st, 2014
- It is called "Unit Testing" for a reason - August 28th, 2014
- Is Your Architecture Crippling Your Unit Testing? - September 4th, 2014
- ASP.NET Server Performance Testing - November 6th, 2014
- NUnit & Visual Studio - December 4th, 2014
- NUnit Test Code Structure - February 5th, 2015
- Excuses For Not Testing - February 26th, 2015
- Why Johnny Can't do Test Driven Development - March 5th, 2015
- TDD Gamification - Turning Test Driven Development into a Game - April 23rd, 2015
- SpecFlow Tutorial - June 4th, 2015
- The Fallacy of Motion - July 23rd, 2015
- NUnit 2 vs NUnit 3. What you need to know. - December 3rd, 2015
- 8 Reasons Johnny Does Not Write Bug Free Code - September 20th, 2016