Saturday, December 1, 2012

NodeJS connection times out after 2 minutes

I've been working on an application that is a REST based API.  It is fairly simple with the only exception that it transfers largish amounts of data (50MB+).  This is not particularly hard for Node.js to do and I would say a great fit as you really can build an application with a small memory foot print.  I was really please to be able to stream the data back to client with the application using less memory than the data sent.  At the time I was using wget to call the API with simple requests so I could test my code was working.  This is pretty straight forward and wget is a great little tool when developing this type of stuff as most browsers would crash when they tried rendering the 50MB of data.  So I had really good way of testing the API each time I added a part to it.

All was going until I came across a strange situation after completing development of another API call.  This new call would timeout at 2 minutes.  I was perplexed as I didn't expect this as nothing had gone awry but the call had a lot of computation which meant no data would be sent to the client for several minutes.  First I thought it was wget's timeout... but it's default is 15 minutes so no way was it in the wrong.  Wget has a few timeout settings so I had a go messing around with them but no joy... no matter what I did it would always at 2 minutes timeout with this new API call.  So I tried a browser on the off chance something was screwy with wget but again they had the same issue.  It occurred to me then that it had something to do with the Node.js and an internal timeout.  I read the http.Server documentation inside out and there is no mention of a connection timeout when sending data...  I was confused and annoyed at this point.  This feature/behaviour is not described/documented in the Node.js API.  After a bit of searching I found a some information that the response connection times out after 2 minutes (120 seconds).

If you look source code of http.js you will see at line 1700 (v0.8.15) the code responsable for the timeout.  Unfortunately there is not much in the code that states why the timeout is hard coded.

socket.setTimeout(2 * 60 * 1000); // 2 minute timeout

The documentation does mention the http.ServerRequest.connection object but makes no mention of the http.ServerResponse.connection object.  I must admit I am surprise that there is no mention of this as how else would a response get back to the client?  The corrective action is pretty straight forward by setting the timeout on the response object.


http.createServer(function (req, res) {
  res.setTimeout(0);  // Never timeout
  /* 
     Do stuff
  */
});


I did hunt around the code base to really understand why there is a timeout of 2 minutes defined.  The best I could do is following which comes from the following commit https://github.com/joyent/node/commit/7a2e6d674a94e01a17e856b4d51ec229fad9af51



Default to 2 second timeout for http servers
Taking a performance hit on 'hello world' benchmark by enabling this by
default, but I think it's worth it. Hopefully we can improve performance by
resetting the timeout less often - ideally a 'hello world' benchmark would
only touch the one timer once - if it runs in less than 2 seconds. The rest
should be just link list manipulations.


Do note the committer ry mistakenly stated seconds instead of minutes in the commit notes but it would appear the is reason is performance.  It has been interesting investigating the cause of my grief but I don't feel the need to remove the 2 minute timeout as it does server a purpose and my situation of keeping a response open which has long periods (minutes) of no data being written is not a common use case.


1 comment:

Unknown said...

Thank you for this post that helped me a lot!
Note you now have the setTimeout() method in Server and ServerResponse: http://nodejs.org/api/http.html#http_response_settimeout_msecs_callback