Mercurial > hg > nginx-tests

annotate stream_ssl_alpn.t @ 1752:ba6e24e38f03

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Tests: improved stop_daemons() to send signal again. As was observed, it's possible that a signal to complete a uwsgi daemon can be ignored while it is starting up, which results in tests hang due to eternal waiting on child processes termination. Notably, it is seen when running tests with a high number of prove jobs on a low-profile VM against nginx with broken modules and/or configuration. To reproduce: $ TEST_NGINX_GLOBALS=ERROR prove -j16 uwsgi*.t Inspecting uwsgi under ktrace on FreeBSD confirms that a SIGTERM signal is ignored at the very beginning of uwsgi startup. It is then replaced with a default action after listen(), thus waiting until uwsgi is ready to accept new TCP connections doesn't completely solve the hang window. The fix is to retry sending a signal some time after waitpid(WNOHANG) continuously demonstrated no progress with reaping a signaled process. It is modelled after f13ead27f89c that improved stop() for nginx.
author Sergey Kandaurov <pluknet@nginx.com>
date Wed, 29 Dec 2021 22:29:23 +0300
parents 38bd7e75fe69
children cdcd75657e52
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1739
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
1 #!/usr/bin/perl
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
2
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
3 # (C) Sergey Kandaurov
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
4 # (C) Nginx, Inc.
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
5
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
6 # Tests for stream ssl_alpn directive.
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
7
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
8 ###############################################################################
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
9
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
10 use warnings;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
11 use strict;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
12
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
13 use Test::More;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
14
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
15 BEGIN { use FindBin; chdir($FindBin::Bin); }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
16
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
17 use lib 'lib';
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
18 use Test::Nginx;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
19 use Test::Nginx::Stream qw/ stream /;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
20
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
21 ###############################################################################
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
22
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
23 select STDERR; $| = 1;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
24 select STDOUT; $| = 1;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
25
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
26 my $t = Test::Nginx->new()->has(qw/stream stream_ssl stream_return/)
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
27 ->has_daemon('openssl')->write_file_expand('nginx.conf', <<'EOF');
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
28
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
29 %%TEST_GLOBALS%%
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
30
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
31 daemon off;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
32
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
33 events {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
34 }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
35
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
36 stream {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
37 %%TEST_GLOBALS_STREAM%%
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
38
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
39 log_format test $status;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
40 access_log %%TESTDIR%%/test.log test;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
41
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
42 ssl_certificate_key localhost.key;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
43 ssl_certificate localhost.crt;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
44
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
45 server {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
46 listen 127.0.0.1:8080 ssl;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
47 return "X $ssl_alpn_protocol X";
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
48 ssl_alpn first second;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
49 }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
50 }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
51
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
52 EOF
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
53
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
54 eval { require IO::Socket::SSL; die if $IO::Socket::SSL::VERSION < 1.56; };
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
55 plan(skip_all => 'IO::Socket::SSL version >= 1.56 required') if $@;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
56
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
57 eval { IO::Socket::SSL->can_alpn() or die; };
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
58 plan(skip_all => 'IO::Socket::SSL with OpenSSL ALPN support required') if $@;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
59
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
60 eval { exists &Net::SSLeay::P_alpn_selected or die; };
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
61 plan(skip_all => 'Net::SSLeay with OpenSSL ALPN support required') if $@;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
62
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
63 $t->write_file('openssl.conf', <<EOF);
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
64 [ req ]
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
65 default_bits = 2048
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
66 encrypt_key = no
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
67 distinguished_name = req_distinguished_name
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
68 [ req_distinguished_name ]
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
69 EOF
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
70
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
71 my $d = $t->testdir();
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
72
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
73 foreach my $name ('localhost') {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
74 system('openssl req -x509 -new '
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
75 . "-config $d/openssl.conf -subj /CN=$name/ "
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
76 . "-out $d/$name.crt -keyout $d/$name.key "
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
77 . ">>$d/openssl.out 2>&1") == 0
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
78 or die "Can't create certificate for $name: $!\n";
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
79 }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
80
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
81 $t->try_run('no ssl_alpn')->plan(6);
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
82
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
83 ###############################################################################
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
84
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
85 is(get_ssl('first'), 'X first X', 'alpn match');
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
86 is(get_ssl('wrong', 'first'), 'X first X', 'alpn many');
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
87 is(get_ssl('wrong', 'second'), 'X second X', 'alpn second');
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
88 is(get_ssl(), 'X X', 'no alpn');
1743
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
89
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
90 SKIP: {
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
91 $t->{_configure_args} =~ /LibreSSL ([\d\.]+)/;
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
92 skip 'LibreSSL too old', 2 if defined $1 and $1 lt '3.4.0';
1744
38bd7e75fe69 Tests: skip ALPN rejection tests with OpenSSL < 1.1.0.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1743
diff changeset
93 $t->{_configure_args} =~ /OpenSSL ([\d\.]+)/;
38bd7e75fe69 Tests: skip ALPN rejection tests with OpenSSL < 1.1.0.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1743
diff changeset
94 skip 'OpenSSL too old', 2 if defined $1 and $1 lt '1.1.0';
1743
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
95
1739
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
96 ok(!get_ssl('wrong'), 'alpn mismatch');
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
97
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
98 $t->stop();
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
99
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
100 like($t->read_file('test.log'), qr/500$/, 'alpn mismatch - log');
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
101
1743
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
102 }
2318ed01ce53 Tests: skip ALPN rejection tests with too old LibreSSL.
Sergey Kandaurov <pluknet@nginx.com>
parents: 1739
diff changeset
103
1739
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
104 ###############################################################################
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
105
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
106 sub get_ssl {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
107 my (@alpn) = @_;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
108 my $s = stream('127.0.0.1:' . port(8080));
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
109
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
110 eval {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
111 local $SIG{ALRM} = sub { die "timeout\n" };
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
112 local $SIG{PIPE} = sub { die "sigpipe\n" };
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
113 alarm(8);
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
114 IO::Socket::SSL->start_SSL($s->{_socket},
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
115 SSL_alpn_protocols => [ @alpn ],
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
116 SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE(),
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
117 SSL_error_trap => sub { die $_[1] }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
118 );
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
119 alarm(0);
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
120 };
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
121 alarm(0);
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
122
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
123 if ($@) {
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
124 log_in("died: $@");
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
125 return undef;
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
126 }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
127
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
128 return $s->read();
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
129 }
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
130
f2fe58b4b59f Tests: ALPN tests in the stream module.
Sergey Kandaurov <pluknet@nginx.com>
parents:
diff changeset
131 ###############################################################################