Investigating performance bottlenecks in your web application - Part 1

Recently I was responsible for improving the performance of a website implemented with ASP.NET MVC. Although the performance improvement does not have a straight forward approach and it strongly depends on your application structure and physical topology, here are the steps I chose and I believe they are valid in most applications with the same structure. Before I list these steps let me describe the structure of our case study:

We are using:
2 Web Servers: Windows Server 2008 R2, ASP.NET MVC 3 hosted in IIS 7
2 App Servers: Windows Server 2008 R2, WCF services hosted in IIS 7
2 DB Server: SQL Server 2008 R2
2 Reporting Servers: Sql Server Reporting Servers (SSRS)


The preliminary load test results revealed that the system was unable to cope beyond 50 concurrent users. The CPU usage in application servers was always flat around 100%. As a first rule, when you experience the high CPU usage the first thing you should blame is the APPLICATION itself! So you have to leverage your investigation skills to find the bottlenecks in the code.

An average CPU utilization of 20–30% on a web or application server is excellent, and
50–70% is a well utilized system. An average CPU utilization above 75% indicates that a system is reaching its computational capacity; however, there's usually no need to panic if you have an average of 90–95%, as you may be able to horizontally scale out by adding an additional server.

As long as we are using Windows Communication Foundation (WCF) in the application layer we can easily leverage the WCF diagnostic option to investigate the long running requests. To turn on the tracing on WCF services you need to add the following section in web.config file.

Here is more information about WCF diagnostics in MSDN.

As you see we log the tracing results in Tracing.svclog file. To view the content of this file you have to use SvcTraceViewer.exe located in "C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin" folder.

You can easily use this powerful tool to analyse the diagnostic traces that are generated by WCF. To get a reliable result you need to run the application for a reasonable period of time while the wcf tracing is on. In this way you could find out the number of method calls along with the process duration for each method. Choose the methods with unexpected duration time and try to modify the method to achieve a better performance.

Sometimes the service operation calls the other services or hits the database to fetch or update the data. So you will need to find out witch part of the service operation causes the bottleneck. Is it the database call or the operation process itself? You can plan different approaches based on your coding structure.

Using WCF tracing will help you to nominate the potential bottlenecks in service level. What if the bottleneck is in the other layers such as Repositories. As you may know people are using repository design patterns to separate the logic that retrieves the data and maps it to the entity model from the business logic that acts on the model. Persistence ignorance is the benefit of using this pattern. Hence we need to do tracing in repository level as well. If you are familiar with Aspect Oriented Programming, known as AOP, you can implement an Interceptor to inject a custom logger into all methods of your repositories. This logger wraps the service method and calculates the duration of the process.

As an example, let's assume that we are using Castle.Windsor as an IoC container. Here is the castle Interceptor I've implemented to do the job for us:

public class FunctionalTraceInterceptor : IInterceptor
  {
     public void Intercept(IInvocation invocation)
     {
         StringBuilder logEntry = new StringBuilder();
         DateTime startTime = DateTime.Now;

         invocation.Proceed();

         double duration = DateTime.Now.Subtract(startTime).TotalMilliseconds;
         logEntry.AppendLine(String.Format("{0},{1},{2}", invocation.InvocationTarget.ToString(), invocation.MethodInvocationTarget.Name, duration));
         ApplicationLogger.Inst.LogInfo("TraceAllFunctionCalls", logEntry.ToString());
    }
  }

IInterceptor is an interface located in Castle.Core.Interceptor namespace. ApplicationLogger is our singleton logger that uses log4net. You can implement your own logger instead. InvocationTarget is the service and MethodInvocationTarget is the service method.

To inject this interceptor into the services you can simply define it in castle configuration file:

  
    
        
          ${FunctionalTraceInterceptor}
        
    

    
    
  

SampleRepository is the repository that has implemented the ISampleRepository interface and we want to inject the interceptor to all of its public methods. You can use this solution for all the services you registered in IoC container.

The purpose of this post was introducing two common techniques for tracing the service calls. However the application is one of the components you need to modify to get a better performance. There are other parameters in performance improvement process that I will mention in the next posts.

JSON Model Binder doesn’t bind nullable properties in MVC 3

Model binding in the ASP.NET MVC framework is one of the features that make developers life easy. As you may know, a model binder in MVC provides a simple way to map posted form values to a .NET Framework type and pass the type to an action method as a parameter. In this post I’m not going to dig into the concepts of model binding as you can get a lot of information in many blogs.

One of the model binders in MVC 3 is JSON model binder. It is clear that it tries to bind json objects passed into the controller action via ajax call. To clarify the issue let’s start with a simple scenario.
Assume that we need to pass Student json object to a controller action. Here is the Student class:

public class Student
{
    public int Id { get; set; }
    public string Name { get; set; }
    public long? Grade { get; set; }
}

The action is called Save and accepts a parameter of type Student:

[HttpPost]
public ActionResult Save(Student student)
{
   // save the student 
   return View();
}

We are using jquery for ajax call. Just put a button on the view and bind the click event using following code:

So, on button click event we create a student object and initialize its properties. Then post that object into the Save action of the Student controller. Now put a breakpoint in action code and run the project. If you look at the student parameter passed into the Save action you will find out that the model binder has created the Student object and assigned all the properties except Grade!

Is that a bug? Yes it is! If you just change the Grade property type from long? To long you will see that the issue solved! I don’t know why the model binder cannot bind the nullable properties and we have to wait to see if it’s solved in the next version. However we cannot stop our projects and have to find a solution.

AtionFilterAttribute is our hero here. As you may know, we can apply a filter before executing the Action in the request pipeline of MVC. Please look at the documentations in MSDN for further info.
ActionFilter gives us the ability to inspect the executing context of the action (ActionExecutingContext). In this context, we have access to HttpContext and the HttpRequest. Also, we have read and write access to the action parameters. What we can do is to bind the incoming JSON data to the action parameter. Here is the code for our custom action filter:

public class JsonModelBinder : ActionFilterAttribute
    {
        public JsonModelBinder()
        {
        }
        
        public Type ActionParameterType { get; set; }
        public string ActionParameterName { get; set; }
        
        public override void OnActionExecuting(ActionExecutingContext filterContext)
        {
            HttpRequestBase request = filterContext.HttpContext.Request;
            Stream stream = request.InputStream;
            stream.Position = 0;
            filterContext.ActionParameters[ActionParameterName] =
                        (new DataContractJsonSerializer(ActionParameterType)).ReadObject(stream);
        }
        
    }

As you see, I’ve used the DataContractJsonSerializer to deserialize the object. This type is defined in System.Runtime.Serialization.Json namespace and you have to add a reference to System.Runtime.Serialization to use it.
The important assumption here is that the InputStream contains a JSON data. ActionParameterName and ActionParameterType are two properties that we have to initialize them when decorating the action with this attribute:

[HttpPost]
[JsonModelBinder(ActionParameterName="student", ActionParameterType=typeof(Student)]
public ActionResult Save(Student student)
{
     // save the student 
     return View();
}

Now let’s run the project again and look at the action parameter:

Bingo!