Fixed timeouts with threaded sendfile() and subrequests. If a write event happens after sendfile() but before we've got the sendfile results in the main thread, this write event will be ignored. And if no more events will happen, the connection will hang. Removing the events works in the simple cases, but not always, as in some cases events are added back by an unrelated code. E.g., the upstream module adds write event in the ngx_http_upstream_init() to track client aborts. Fix is to use wev->complete instead. It is now set to 0 before a sendfile() task is posted, and it is set to 1 once a write event happens. If on completion of the sendfile() task wev->complete is 1, we know that an event happened while we were executing sendfile(), and the socket is still ready for writing even if sendfile() did not sent all the data or returned EAGAIN.

--- a/src/event/modules/ngx_epoll_module.c
+++ b/src/event/modules/ngx_epoll_module.c
@@ -840,6 +840,9 @@ ngx_epoll_process_events(ngx_cycle_t *cy
             }
 
             wev->ready = 1;
+#if (NGX_THREADS)
+            wev->complete = 1;
+#endif
 
             if (flags & NGX_POST_EVENTS) {
                 ngx_post_event(wev, &ngx_posted_events);