+ +

+ + +This article assumes the reader has a general knowledge of nginx internals and +DTrace. + + + +Although nginx build with --with-debug option +already provides a lot of information about request processing, it is sometimes desirable +to trace particular parts of code path more thoroughly and at the same time +omit the rest of debug output. DTrace pid provider (available on Solaris, OS X) is a useful +tool to explore userland programs internals, since it doesn't require any code changes and +it can help with the task. E.g. a simple DTrace script to trace and print nginx +functions calls may look like: + + +#pragma D option flowindent + +pid$target:nginx::entry { +} + +pid$target:nginx::return { +} + + + + + +DTrace capabilities for function calls tracing provide only a limited amount +of useful information, though. Real-time inspection of function arguments +is typically more interesting, but also a bit more complicated. Examples below +are intended to help the reader become more familiar with DTrace and the +process of analyzing nginx behavior using DTrace. + + + +One of the common scenarios for using DTrace with nginx is the following: +attach to the nginx worker to log request lines and request start times. +The corresponding function to attach is +ngx_http_process_request, and the argument in question +is a pointer to ngx_http_request_t structure. +DTrace script for such request logging can be as simple as: + + + +pid$target::*ngx_http_process_request:entry +{ + this->request = (ngx_http_request_t *)copyin(arg0, sizeof(ngx_http_request_t)); + this->request_line = stringof(copyin((uintptr_t)this->request->request_line.data, + this->request->request_line.len)); + printf("request line = %s\n", this->request_line); + printf("request start sec = %d\n", this->request->start_sec); +} + + + + + +It should be noted that in the example above DTrace requires some knowledge about +ngx_http_process_request structure. +Unfortunately while it is possible to use a specific #include +directive in the DTrace script and then pass it to a C preprocessor +(with -C flag), that doesn't really work. Due to a lot of cross dependencies almost +all nginx header files have to be included. In turn, based on configure script +settings, nginx headers will include PCRE, OpenSSL and a variety of system header +files. While in theory all those header files related to a specific nginx build +might be included in DTrace script preprocessing and compilation, in reality +DTrace script most probably will fail to compile because of unknown syntax in +some header files. + + + +The problem above can be solved by including only the relevant and +necessary structures and types definitions in the DTrace script. DTrace has to know +sizes of structures, types, and fields offsets. Thus dependencies can be further +reduced by manually optimizing structure definitions for use with DTrace. + + + +Let's use DTrace script example above and see what structure definitions it needs +to work properly. + + + +First of all objs/ngx_auto_config.h file generated by configure +should be included, because it defines a number of constants affecting various +#ifdef's. After that there's some basic types and definitions +like ngx_str_t, ngx_table_elt_t, +ngx_uint_t etc. should be put at the beginning of DTrace script. +These definitions are compact, commonly used and unlikely to be frequently changed. + + + +Then there's the ngx_http_process_request_t structure which +contains a lot of pointers to other structures. Because these pointers are +really irrelevant to this script, and because they have the same size, +it is possible to just replace them with void pointers. Instead of changing +definitions, it is better to add appropriate typedefs, though: + + +typedef ngx_http_upstream_t void; +typedef ngx_http_request_body_t void; + + +Last but not least it is necessary to add definitions of two member structures: +ngx_http_headers_in_t and +ngx_http_headers_out_t, callback functions +declarations and constants definitions. + + + +Final DTrace script can be downloaded +here. + + + +The example below shows the output of the DTrace script: + + +# dtrace -C -I ./objs -s trace_process_request.d -p 4848 +dtrace: script 'trace_process_request.d' matched 1 probe +CPU ID FUNCTION:NAME + 1 4 .XAbmO.ngx_http_process_request:entry request line = GET / HTTP/1.1 +request start sec = 1349162898 + + 0 4 .XAbmO.ngx_http_process_request:entry request line = GET /en/docs/nginx_dtrace_pid_provider.html HTTP/1.1 +request start sec = 1349162899 + + + + +Using similar techniques the reader should be able to trace other +nginx functions calls. + + +

+ +

+ + + + + + +Solaris Dynamic Tracing Guide + + + + +Introduction article on DTrace pid provider + + + + + +

+ +