Infosys | Microsoft

Troubleshooting

While troubleshooting such problems one should start with following
1. The service application hosted on IIS should be using dedicated application pool (no other application is using the same application pool)

2. In case if the client application (asp.net) is also hosted on the same IIS server it should be using another dedicated application pool

Once these basic things are ensured, one should configure perfmon on IIS server with Instances counter from ServiceModelService 3.0.0.0. The instances count can help plot the service instances consumed from the application at any given point of time.

Another way to validate this is by taking memory dumps using DebugDiag, downloadable here.

Take full memory dump on server running WCF services (for IIS 6 and above, w3wp process id running WCF, or aspnet_wp for IIS 5). The Debugdiag tool needs to be installed on machine running IIS and the appropriate application Crash/Hang rule should be configured in this tool to capture the dump. Once the memory dump is captured on server, one can analyze this dump using WinDbg client, this tool can be downloaded from here

Once dump is obtained, next step is to start analyzing dumps. Steps to analyze the memory dumps are as follows

1) Open Crash Dump in WindDebug from File-> Open Crash dump

2) Set symbol path using command !symfix D:\Symbols

3) Load SOS file for .net 2.0 CLR using
   .Load C:\Windows\Microsoft.NET\Framework\v2.0.50727\SOS.dll

4)   To list addresses of all objects of type ServiceThrottle that exist on the managed heap. Use !dumpheap -type ServiceThrottle –short

------------------------------
0:000> !dumpheap -type ServiceThrottle -short
09f20b10
09f22a14
09f24a56
09f25023
0deaa456

5)For each address from the above output, list dumpobject address using command
!do 0deaa456

0:000> !do 0deaa456
Name: System.ServiceModel.Dispatcher.ServiceThrottle
MethodTable: 1b1ba2e4
EEClass: 1ae21f20
Size: 36(0x24) bytes
 (C:\WINNT\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT      Field         Offset                 Type VT     Attr      Value         Name
1b1ba328  400371c        4 ...cher.FlowThrottle  0   instance 0deaa454   calls
1b1ba328  400371d        8 ...cher.FlowThrottle  0  instance 0dea45b    sessions
1b175dbc  400371e        c ...her.QuotaThrottle  0  instance 00000000  dynamic
1b1ba328  400371f       10 ...cher.FlowThrottle  0  instance 00000000 instanceContexts
1b1b994c  4003720       14 ...l.ServiceHostBase 0  instance 0deaa168 host
793043b8  4003721       1c       System.Boolean 1 instance 0deaa234 isActive
79330508  4003722       18        System.Object  0 instance 0deaa3b8 thisLock

6) The ServiceThrottle object has many fields but sessions will help us progress. Type !do address of the sessions field to see what's inside of it.
!do 0dea45b
 
0:000> !do 0dea45b
Name: System.ServiceModel.Dispatcher.FlowThrottle
MethodTable: 1b1ba322
EEClass: 1ae21c43
Size: 36(0x24) bytes
 (C:\WINNT\assembly\GAC_MSIL\System.ServiceModel\3.0.0.0__b77a5c561934e089\System.ServiceModel.dll)
Fields:
      MT      Field       Offset                 Type VT     Attr        Value       Name
79332b38  50034f4       18         System.Int32  1 instance       10             capacity
79332b38  50034f5       1c         System.Int32  1 instance       10              count
79330508  50034f6        4        System.Object  0 instance 0deaa570 mutex
7931e84c  50034f7        8 ...ding.WaitCallback  0 instance 0deaa52c release
00000000  50034f8        c                               0 instance 0deaa57c waiters
793308ec  50034f9       10        System.String  0 instance 0deaa4b4 propertyName
793308ec  50034fa       14        System.String  0 instance 0deaa4f0  configName
 
The above highlighted capacity shows what the maximum capacity of WCF pool is; count indicates the current number of instances. In this example since both are 10, it indicates that no new WCF connection can be created unless the existing ones are being released.

Remedies

Unlike ASMX services, the WCF services needs to be explicitly instantiated and once the service call is completed the proxy connection needs to be explicitly closed. Not doing so correctly can result in to server side WCF connection exhaustion ultimately leading to service denial.
There may be scenarios where the concurrent users may be 2 or 3 and still the WCF connection pool may get exhausted, this can happen due to one of the possible condition

1.While working with the application, the concurrent browser sessions would have reached to 10 and hence connection pool got exhausted and couldn’t serve beyond 10 concurrent requests

2.WCF Service in turn calling Database stored procedures or other code which is long running and hasn’t returned the call to service, connection timeout settings are not hit and hence pool got exhausted.

3.WCF connections being leaked from some part of the code as WCF proxies were not closed appropriately.

For #2, the service side .config parameters which control settings are mentioned in serviceThrottling section depicted as
 
<serviceBehaviors>
        <behavior   name="My.Samples.ServiceLayer.CricketScoreServiceBehavior">
          <serviceThrottling
                 maxConcurrentCalls="10"  
                 maxConcurrentSessions="10” />
          <serviceMetadata httpGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="false" />
 </serviceBehaviors>

maxConcurrentCalls specifies maximum number of messages actively processing over a servicehost, where as maxConcurrentSessions controls how many maximum concurrent sessions can exists on the service host.  Refer the details about these settings here
Setting these to 10 may not be enough in scenarios where total No. of users or concurrent users are high.
For the above service, as soon as the total service instance reaches to 10, the next client call to service would go in wait state. It would remain in wait state and wait till the existing WCF connections are closed or timed out. Hence it is important that the timeout setting kicks in ASAP so that the client can acquire connection. These settings are controlled using the client side .config parameters which are ReceiveTimeout and InactivityTimeout

•InactivityTimeout - This inactivity timer fires if no messages, either application or infrastructure, are received within the timeout period. An infrastructure message is a message that is generated for the purpose of one of the protocols in the channel stack, such as a keep alive or an acknowledgment, rather than containing application data.

•ReceiveTimeout - This inactivity timer fires if no application messages are received within the timeout period. This specifies, for example, the maximum time a client may take to send at least one message to the server before the server will close the channel used by a session. This behavior ensures that clients cannot hold on to server resources for arbitrary long periods.

<system.serviceModel>
    <bindings>
      <wsHttpBinding>
        <binding name="WSHttpBinding_ICricketScoreService"
          closeTimeout="00:01:00"
          openTimeout="00:01:00"
          receiveTimeout="00:10:00"
          sendTimeout="00:01:00"
            <reliableSession ordered="true" inactivityTimeout="00:10:00"
            enabled="false" />
        </binding>
      </wsHttpBinding>
    </bindings>
</system.serviceModel>
 
 
Details about other WCF settings can be referred from here
Timeout settings are handy, but for #3 above, important that in the client side code one must ensure that the WCF proxy connections be explicitly closed. Following code snippet can be used as guidance for appropriate connection closing.

Try
  {
      CricketScoreService.ScoreClient _obj;
     _obj = new CricketScoreService.ScoreClient();
      //do other activities like calling Business layer method and once
      //done close the WCF connection as below
     _obj.Close();
  }
              Catch (Exception e) //catch service exception like faulted, etc.
      {
         if (obj!= null) &&
            (_obj.State == System.ServiceModel.CommunicationState.Faulted)
          {
           _obj.Abort();
          }
         Else if (obj!= null) &&
                 (_obj.State != System.ServiceModel.CommunicationState.Closed)
                {
                    _obj.Close();
                }
      }
   
 
Another recommended way to close WCF connection is by “using” statement as follows,
using (CricketScoreService.ScoreClient sc = new CricketScoreService.ScoreClient ())
       {
         sc.getScores(“111”);
       }

The using block automatically calls sc.Dispose() when it's done.
using statement doesn’t always ensure closed connection, hence it is not recommended to be used for closing connections. 
With ClientBase abstract class(which any WCFClient would inherit, check here), implementing the Dispose() method will make a call to Close(). When the ClientBase's Close() method is called, a web service call actually goes out to the WCF service, informing it that the connection session is no longer required. The problem with this mechanism is that when the Close() method is called, an exception can be thrown. After all, it involves yet another network call to a web service. It's for this reason that the using statement isn't recommended with WCF clients.

With this as background, watch out for WCF application hang problem for applications running in UAT, or production scenarios and this should come handy resolving it.