How RPC Client Throttling Works

If RPC average latency on the Exchange 2007 server is higher than normal, Exchange server users may start to experience noticeable service degradation. RPC average latency should be kept below 100 milliseconds in order to maintain an acceptable end-user experience.

RPC Client Throttling uses a computational method that is based on overall RPC average latencies and remote operation (ROP) statistics to determine whether a given client will start receiving back-off requests to limit the number of operations that the client can perform against the server. This method computes the average delay between ROPs over three sampling periods for a given minute. This logic uses a rolling average of the ROP rate for the last minute. Once this value is determined, it is then compared to the overall RPC average latency. If this computed value is lower than the overall RPC average latency, then back-off requests will be sent to the client.

Important:
Short spikes of high ROPS per second will not result in a back-off request being sent to the client as long as the average RPC latency for the last minute is low.

For example, if you have an RPC latency of 25 milliseconds, a well-behaved client can successfully send up to 40 ROPS per second (40*25=1000 milliseconds) with no throttling. If RPC latency increased to 100 milliseconds, and the client attempted to send the same amount of ROPS, the client would then start receiving back-off requests from the server. From this point, the server will only allow the client to send up to 10 ROPS per second until the next RPC average latency sample is again below 100 milliseconds.

The logic for back-off requests has two main parts. The first is how the server decides to throttle the client and the second is the determination to block the client from accessing the server. Inside the Information Store service, there is a client activity log that keeps track of each user’s activity against any given server. Each client activity log is associated with a users MAPI session. If the server makes a decision to throttle a client, a back-off request is added to the session back-off queue for that user's MAPI session. This back-off information is then used to determine if the client request is accepted or rejected by the server. If the user has multiple mailboxes open for a single MAPI session, this back-off information will apply to all mailboxes for that MAPI session.

Depending on the version of Microsoft Office Outlook being used, the server will send different back-off information to the client's RPC response stream.

• For Microsoft Office Outlook 2007 clients, a ropBackoff request is added to the user's back-off queue. This request contains information regarding the current back-off delay, and tells the client to delay sending any additional requests to the server for a specified time. The client will then retry sending requests after the specified time expires. The maximum time for this back-off delay is hard coded to 2000 milliseconds or two seconds.
  
  

  Note:
  We recommend using Outlook 2007 clients against an Exchange 2007 server for an optimal client experience.

  Note:
  The ropBackoff is a new feature in Outlook 2007 and the earlier Outlook clients do not understand this function.

• For Microsoft Office Outlook 2003 or earlier versions, a status code of RPC_S_SERVER_TOO_BUSY is sent. This status code mimics the response Exchange 2003 servers send to clients when Information Store RPC threads have been exhausted. The earlier versions are designed to catch this exception and then wait for a length of time before the call is retried as specified in the response from the server to the client. The default time for this delay is one second. If the client tries the request every second for one minute and still receives the same exception each time, the client gives up and disconnects its session to the Exchange server.
  
  

Monitoring for High RPC Operations per Second Conditions

To help detect when clients are being backed off, the performance counter “MSExchangeIS\RPC Client Backoff/sec” can be monitored on Exchange servers that have the Mailbox server role installed to determine the rate at which back-off requests are occurring.

Counter	Outlook 2007 expected values	Outlook 2003 and earlier expected values
MSExchangeIS\RPC Client Backoff/sec The rate that the server notifies the client to back-off.	50 per client	1 per client

For Outlook 2007 clients that are being throttled, you can expect the “MSExchangeIS\RPC Client Backoff/sec” counter to show approximately 50 back-offs per second for each client. For Outlook 2003 and earlier versions, you should expect, at most, one back-off per second for each client. The difference in the expected rates is caused by the granularity of the back-off method that is used for the respective clients. The smallest back-off period for Outlook 2003 is one second, whereas the smallest back-off period for Outlook 2007 is one millisecond.

When back-off information is sent to the client, the data contains two parts. The first part is the time that the back-off was inserted. The second part is the duration of the back-off period. The duration of the back-off period is calculated based on the following formula: (MSExchangeIS\RPC Average Latency * RPC throttling factor)/1000. By default, the RPC throttling factor is a value of 1000. This means that the back-off period is exactly the value of RPC average latency. The back-off value that the server sends to the client is the number of milliseconds until the back-off period expires. If the client sends another operation to the server before the specified back-off period expires, the server will return another ropBackoff with an updated delay value. This will continue to occur until the back-off period expires, thus removing the ropBackoff from the user's back-off queue. A sample of what this resembles in Performance Monitor can be seen in the following graphic:

Tuning for High RPC Operations per Second Conditions

On Exchange servers that have the Mailbox server role installed, a registry key is available to help tune the throttling computation. This registry key is called RPC Throttling Factor and is located under the HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem key.

Note:
This value is not set by default and must be added as a DWORD value under the previously mentioned key.

[HKLM\System\CurrentControlSet\Services\MSExchangeIS\ParametersSystem]

RPC Throttling Factor

DWORD: Can range between a value of 0-5000

To show how modification of the RPC Throttling Factor works, consider the scenario where you want to configure the server to backoff clients for a longer period of time, such as 2 x RPC Average Latency. You can do this by setting the RPC Throttling Factor to a value of 2000. This will increase the backoff period for each MAPI session by a factor of two. In this case, if RPC Averaged Latency was 50 milliseconds, a throttling factor of 2000 (50 ms * 2000/1000) will equal 100 milliseconds. Each backoff request will be extended to 100 milliseconds instead of the default 50 milliseconds, thus delaying additional client MAPI requests to the server for 100 milliseconds. Understand that such a configuration change may appear to the client as slow server performance. However, if this client has an adverse effect on overall server performance, resource usage on the Exchange server by this client will be minimized. This improves perceived server performance for all other users on the server.

Important:
If this problem should occur for many or all clients on the server, then a server-side investigation should be performed to detect any bottlenecks that may be occurring.

For an optimal experience with this throttling mechanism, a minimum of Outlook 2007 and Exchange 2007 is recommended.

You can disable throttling altogether by setting the throttling factor to a value of 0.

Important:
We do not recommend disabling client throttling in the Information Store. Making this change will allow poorly behaving clients to degrade performance on any given server.

If you believe that you may have to disable client throttling, we recommend that you first determine the cause of the performance bottleneck on the server. In most cases, identification and resolution of the cause will resolve the need for the server to send back-off requests to the clients.

For More Information