Mercurial > hg > nginx-tests
view memcached_keepalive_stale.t @ 1752:ba6e24e38f03
Tests: improved stop_daemons() to send signal again.
As was observed, it's possible that a signal to complete a uwsgi daemon
can be ignored while it is starting up, which results in tests hang due
to eternal waiting on child processes termination. Notably, it is seen
when running tests with a high number of prove jobs on a low-profile VM
against nginx with broken modules and/or configuration. To reproduce:
$ TEST_NGINX_GLOBALS=ERROR prove -j16 uwsgi*.t
Inspecting uwsgi under ktrace on FreeBSD confirms that a SIGTERM signal
is ignored at the very beginning of uwsgi startup. It is then replaced
with a default action after listen(), thus waiting until uwsgi is ready
to accept new TCP connections doesn't completely solve the hang window.
The fix is to retry sending a signal some time after waitpid(WNOHANG)
continuously demonstrated no progress with reaping a signaled process.
It is modelled after f13ead27f89c that improved stop() for nginx.
author | Sergey Kandaurov <pluknet@nginx.com> |
---|---|
date | Wed, 29 Dec 2021 22:29:23 +0300 |
parents | 882267679006 |
children |
line wrap: on
line source
#!/usr/bin/perl # (C) Maxim Dounin # Test for stale events handling in upstream keepalive. ############################################################################### use warnings; use strict; use Test::More; BEGIN { use FindBin; chdir($FindBin::Bin); } use lib 'lib'; use Test::Nginx; ############################################################################### select STDERR; $| = 1; select STDOUT; $| = 1; eval { require Cache::Memcached; }; plan(skip_all => 'Cache::Memcached not installed') if $@; my $t = Test::Nginx->new()->has(qw/http memcached upstream_keepalive rewrite/) ->has_daemon('memcached')->plan(1) ->write_file_expand('nginx.conf', <<'EOF'); %%TEST_GLOBALS%% daemon off; worker_processes 2; events { } http { %%TEST_GLOBALS_HTTP%% upstream memd { server 127.0.0.1:8081; keepalive 1; } server { listen 127.0.0.1:8080 sndbuf=32k; server_name localhost; location / { set $memcached_key $uri; memcached_pass memd; } } } EOF my $memhelp = `memcached -h`; my @memopts1 = (); if ($memhelp =~ /repcached/) { # repcached patches adds additional listen socket memcached # that should be different too push @memopts1, '-X', port(8082); } if ($memhelp =~ /-U/) { # UDP ports no longer off by default in memcached 1.2.7+ push @memopts1, '-U', '0'; } if ($memhelp =~ /-t/) { # for connection stats consistency in threaded memcached 1.3+ push @memopts1, '-t', '1'; } $t->run_daemon('memcached', '-l', '127.0.0.1', '-p', port(8081), @memopts1); $t->run(); $t->waitforsocket('127.0.0.1:' . port(8081)) or die "Unable to start memcached"; ############################################################################### my $memd1 = Cache::Memcached->new(servers => [ '127.0.0.1:' . port(8081) ], connect_timeout => 1.0); # It's possible that stale events occur, i.e. read event handler called # for just saved upstream connection without any data available for # read. We shouldn't close upstream connection in such situation. # # This happens due to reading from upstream connection on downstream write # events. More likely to happen with multiple workers due to use of posted # events. # # Stale event may only happen if reading response from upstream requires # entering event loop, i.e. response should be big enough. On the other # hand, it is less likely to occur with full client's connection output # buffer. # # We use here 2 workers, 20k response and set output buffer on clients # connection to 32k. This allows more or less reliably reproduce stale # events at least on FreeBSD testbed here. $memd1->set('/big', 'X' x 20480); my $total = $memd1->stats()->{total}->{total_connections}; for (1 .. 100) { http_get('/big'); } cmp_ok($memd1->stats()->{total}->{total_connections}, '<=', $total + 2, 'only one connection per worker used'); ###############################################################################