System.Web.HttpException: The remote host closed the connection. The error code is 0x80072746.

by Vignesh 6/13/2008 5:51:00 AM

The client had developed an AJAX based ASP.NET 2.0 web application which had been deployed to multiple IIS 6.0 servers. On only one particular server they noticed that the application does not perform as expected. It works for a few hours and then starts to fail with the following error logged in event log. The only way to recover from the problem is to reboot the server whenever the issue occurs.

Exception Type:System.Web.HttpException
The remote host closed the connection
. The error code is 0x80072746.
at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status,
Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
at System.Web.Hosting.ISAPIWorkerRequest.FlushResponse(Boolean finalFlush)
at System.Web.HttpResponse.Flush(Boolean finalFlush)
at System.Web.HttpResponse.Flush()
at System.Web.UI.HttpResponseWrapper.System.Web.UI.IHttpResponse.Flush()
at System.Web.UI.PageRequestManager.RenderFormCallback(HtmlTextWriter writer,
Control containerControl)
at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection
children)
at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)
at System.Web.UI.HtmlControls.HtmlForm.Render(HtmlTextWriter output)
at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer,
ControlAdapter adapter)
at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter
adapter)
at System.Web.UI.HtmlControls.HtmlForm.RenderControl(HtmlTextWriter writer)
at
System.Web.UI.HtmlFormWrapper.System.Web.UI.IHtmlForm.RenderControl(HtmlTextWriter
writer)
at System.Web.UI.PageRequestManager.RenderPageCallback(HtmlTextWriter writer,
Control pageControl)
at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection
children)
at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
at System.Web.UI.Page.Render(HtmlTextWriter writer)
at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer,
ControlAdapter adapter)
at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter
adapter)
at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint,
Boolean includeStagesAfterAsyncPoint)
Exception Type:System.Web.HttpException
The remote host closed the connection. The error code is 0x80072746.
at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status,

->A fiddler trace shows that the following error is returned to the client.

"Sys.WebForms.PageRequestManagerTimeoutException: The server request timed out"

->We then took a network trace between the client and the server and reproduced the error. From the client trace we can see that the server is sending a Reset flag to reset the connection.

Client_IP - YY.YY.YY.YYY
Server_IP - ZZ.Z.ZZ.ZZZ

Frame 16589 (54 bytes on wire, 54 bytes captured)
Ethernet II, Src: HewlettP_81:b4:09 (00:16:35:81:b4:09), Dst: HewlettP_5c:c2:6b
(00:17:08:5c:c2:6b)
Internet Protocol, Src: ZZ.Z.ZZ.ZZZ, Dst: YY.YY.YY.YYY
Transmission Control Protocol, Src Port: 2882 (2882), Dst Port: http (80), Seq:
1159, Ack: 26931, Len: 0
Source port: 2882 (2882)
Destination port: http (80)
Sequence number: 1159 (relative sequence number)
Acknowledgement number: 26931 (relative ack number)
Header length: 20 bytes
Flags: 0x14 (RST, ACK)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...1 .... = Acknowledgment: Set
.... 0... = Push: Not set
.... .1.. = Reset: Set
.... ..0. = Syn: Not set
.... ...0 = Fin: Not set
Window size: 0
Checksum: 0x4bbe [correct]

We were also able to reproduce the "Sys.WebForms.PageRequestManagerTimeoutException: The server request timed out" error locally on the server. The puzzling partConfused was that the same code works as expected on other servers. My focus shifted to narrowing this down to an environment issue so I started to hunting down any clues which would point us in the right direction. I tried opening the task manager and it came up as a blank window. On further digging we found the server very low on Free System PTE's. This was due to the /3GB switch being enabled in the boot.ini file.

On a 32 bit machine, remember that we only have a 4GB total address space to work with.  Using the /3GB switch allocates 1 GB to the kernel and 3 GB to the User-mode space.

Using this switch reduces the memory available in the following system pools:

-->Nonpaged Pool

-->Paged Pool

-->System Page Table Entries (PTEs)

If we have to allocate an additional 1GB of this address space to the user-mode space, then the System space is cut in half.  Drivers, Heap, Paged & NonPaged Memory all have only half the resources to work with now.  However, because of the way memory mapping works, cutting the kernel space in half does a lot more than just reducing the address space. A machine under medium to heavy load could quite easily run out of free PTE's - meaning that the system can no longer map system pages such as I/O space, kernel stacks and memory descriptor lists. When the memory reduction in the pools is too great in a specific server installation, the server or the applications may generate an error[s] or appear to stop responding.

The resolution was to disable the /3GB switch and reboot the machine. PartyThumbs-up . This also fixed the issue with the task manager!

There are other issues which I have seen when the Nonpaged Pool memory on the server is depleted. Please see http://support.microsoft.com/kb/933844 for more information.

My colleague has blogged about this at http://blogs.msdn.com/sudeepg/archive/2007/09/11/iis-not-serving-pages-page-cannot-be-displayed.aspx which has information on how to troubleshoot such issues.

Tags: , ,

Tips and Tricks | Troubleshooting | ASP.NET | AJAX

blog comments powered by Disqus

Comments


Disclaimer

The opinions expressed herein are our own personal opinions and do not represent our employer's view in any way.
© Copyright 2014, Rahul Soni