|
698
|
1 <?xml version="1.0"?>
|
|
|
2
|
|
|
3 <!--
|
|
|
4 Copyright (C) Nginx, Inc.
|
|
|
5 -->
|
|
|
6
|
|
|
7 <!DOCTYPE article SYSTEM "../../../dtd/article.dtd">
|
|
|
8
|
|
|
9
|
|
|
10 <article name="Debugging nginx with DTrace pid provider"
|
|
|
11 link="/en/docs/nginx_dtrace_pid_provider.html"
|
|
|
12 lang="en"
|
|
|
13 rev="1"
|
|
|
14 toc="no">
|
|
|
15
|
|
|
16 <section>
|
|
|
17
|
|
|
18 <para>
|
|
|
19 This article assumes the reader has a general knowledge of nginx internals and
|
|
|
20 <link id="see_also">DTrace</link>.
|
|
|
21 </para>
|
|
|
22
|
|
|
23 <para>
|
|
|
24 Although nginx build with <link doc="debugging_log.xml">--with-debug</link> option
|
|
|
25 already provides a lot of information about request processing, it is sometimes desirable
|
|
|
26 to trace particular parts of code path more thoroughly and at the same time
|
|
|
27 omit the rest of debug output. DTrace pid provider (available on Solaris, OS X) is a useful
|
|
|
28 tool to explore userland programs internals, since it doesn't require any code changes and
|
|
|
29 it can help with the task. E.g. a simple DTrace script to trace and print nginx
|
|
|
30 functions calls may look like:
|
|
|
31
|
|
|
32 <programlisting>
|
|
|
33 #pragma D option flowindent
|
|
|
34
|
|
|
35 pid$target:nginx::entry {
|
|
|
36 }
|
|
|
37
|
|
|
38 pid$target:nginx::return {
|
|
|
39 }
|
|
|
40 </programlisting>
|
|
|
41
|
|
|
42 </para>
|
|
|
43
|
|
|
44 <para>
|
|
|
45 DTrace capabilities for function calls tracing provide only a limited amount
|
|
|
46 of useful information, though. Real-time inspection of function arguments
|
|
|
47 is typically more interesting, but also a bit more complicated. Examples below
|
|
|
48 are intended to help the reader become more familiar with DTrace and the
|
|
|
49 process of analyzing nginx behavior using DTrace.
|
|
|
50 </para>
|
|
|
51
|
|
|
52 <para>
|
|
|
53 One of the common scenarios for using DTrace with nginx is the following:
|
|
|
54 attach to the nginx worker to log request lines and request start times.
|
|
|
55 The corresponding function to attach is
|
|
|
56 <literal>ngx_http_process_request</literal>, and the argument in question
|
|
|
57 is a pointer to <literal>ngx_http_request_t</literal> structure.
|
|
|
58 DTrace script for such request logging can be as simple as:
|
|
|
59
|
|
|
60 <programlisting>
|
|
|
61
|
|
|
62 pid$target::*ngx_http_process_request:entry
|
|
|
63 {
|
|
|
64 this->request = (ngx_http_request_t *)copyin(arg0, sizeof(ngx_http_request_t));
|
|
|
65 this->request_line = stringof(copyin((uintptr_t)this->request->request_line.data,
|
|
|
66 this->request->request_line.len));
|
|
|
67 printf("request line = %s\n", this->request_line);
|
|
|
68 printf("request start sec = %d\n", this->request->start_sec);
|
|
|
69 }
|
|
|
70
|
|
|
71 </programlisting>
|
|
|
72 </para>
|
|
|
73
|
|
|
74 <para>
|
|
|
75 It should be noted that in the example above DTrace requires some knowledge about
|
|
|
76 <literal>ngx_http_process_request</literal> structure.
|
|
|
77 Unfortunately while it is possible to use a specific <literal>#include</literal>
|
|
|
78 directive in the DTrace script and then pass it to a C preprocessor
|
|
|
79 (with -C flag), that doesn't really work. Due to a lot of cross dependencies almost
|
|
|
80 all nginx header files have to be included. In turn, based on configure script
|
|
|
81 settings, nginx headers will include PCRE, OpenSSL and a variety of system header
|
|
|
82 files. While in theory all those header files related to a specific nginx build
|
|
|
83 might be included in DTrace script preprocessing and compilation, in reality
|
|
|
84 DTrace script most probably will fail to compile because of unknown syntax in
|
|
|
85 some header files.
|
|
|
86 </para>
|
|
|
87
|
|
|
88 <para>
|
|
|
89 The problem above can be solved by including only the relevant and
|
|
|
90 necessary structures and types definitions in the DTrace script. DTrace has to know
|
|
|
91 sizes of structures, types, and fields offsets. Thus dependencies can be further
|
|
|
92 reduced by manually optimizing structure definitions for use with DTrace.
|
|
|
93 </para>
|
|
|
94
|
|
|
95 <para>
|
|
|
96 Let's use DTrace script example above and see what structure definitions it needs
|
|
|
97 to work properly.
|
|
|
98 </para>
|
|
|
99
|
|
|
100 <para>
|
|
|
101 First of all <literal>objs/ngx_auto_config.h</literal> file generated by configure
|
|
|
102 should be included, because it defines a number of constants affecting various
|
|
|
103 <literal>#ifdef's</literal>. After that there's some basic types and definitions
|
|
|
104 like <literal>ngx_str_t</literal>, <literal>ngx_table_elt_t</literal>,
|
|
|
105 <literal>ngx_uint_t</literal> etc. should be put at the beginning of DTrace script.
|
|
|
106 These definitions are compact, commonly used and unlikely to be frequently changed.
|
|
|
107 </para>
|
|
|
108
|
|
|
109 <para>
|
|
|
110 Then there's the <literal>ngx_http_process_request_t</literal> structure which
|
|
|
111 contains a lot of pointers to other structures. Because these pointers are
|
|
|
112 really irrelevant to this script, and because they have the same size,
|
|
|
113 it is possible to just replace them with void pointers. Instead of changing
|
|
|
114 definitions, it is better to add appropriate typedefs, though:
|
|
|
115
|
|
|
116 <programlisting>
|
|
|
117 typedef ngx_http_upstream_t void;
|
|
|
118 typedef ngx_http_request_body_t void;
|
|
|
119 </programlisting>
|
|
|
120
|
|
|
121 Last but not least it is necessary to add definitions of two member structures:
|
|
|
122 <literal>ngx_http_headers_in_t</literal> and
|
|
|
123 <literal>ngx_http_headers_out_t</literal>, callback functions
|
|
|
124 declarations and constants definitions.
|
|
|
125 </para>
|
|
|
126
|
|
|
127 <para>
|
|
|
128 Final DTrace script can be downloaded
|
|
|
129 <link url="http://nginx.org/download/trace_process_request.d">here</link>.
|
|
|
130 </para>
|
|
|
131
|
|
|
132 <para>
|
|
|
133 The example below shows the output of the DTrace script:
|
|
|
134
|
|
|
135 <programlisting>
|
|
|
136 # dtrace -C -I ./objs -s trace_process_request.d -p 4848
|
|
|
137 dtrace: script 'trace_process_request.d' matched 1 probe
|
|
|
138 CPU ID FUNCTION:NAME
|
|
|
139 1 4 .XAbmO.ngx_http_process_request:entry request line = GET / HTTP/1.1
|
|
|
140 request start sec = 1349162898
|
|
|
141
|
|
|
142 0 4 .XAbmO.ngx_http_process_request:entry request line = GET /en/docs/nginx_dtrace_pid_provider.html HTTP/1.1
|
|
|
143 request start sec = 1349162899
|
|
|
144 </programlisting>
|
|
|
145
|
|
|
146 </para>
|
|
|
147
|
|
|
148 <para>Using similar techniques the reader should be able to trace other
|
|
|
149 nginx functions calls.
|
|
|
150 </para>
|
|
|
151
|
|
|
152 </section>
|
|
|
153
|
|
|
154 <section id="see_also"
|
|
|
155 name="See also">
|
|
|
156
|
|
|
157 <para>
|
|
|
158 <list type="bullet">
|
|
|
159
|
|
|
160 <listitem>
|
|
|
161 <link url="http://docs.oracle.com/cd/E19253-01/817-6223/index.html">
|
|
|
162 Solaris Dynamic Tracing Guide</link>
|
|
|
163 </listitem>
|
|
|
164
|
|
|
165 <listitem>
|
|
|
166 <link url="http://dtrace.org/blogs/brendan/2011/02/09/dtrace-pid-provider/">
|
|
|
167 Introduction article on DTrace pid provider</link>
|
|
|
168 </listitem>
|
|
|
169
|
|
|
170 </list>
|
|
|
171 </para>
|
|
|
172
|
|
|
173 </section>
|
|
|
174
|
|
|
175 </article>
|
