Mercurial > hg > nginx-quic
annotate src/core/ngx_slab.c @ 4207:4fc91bae6f83
Better recheck of dead upstream servers.
Previously nginx used to mark backend again as live as soon as fail_timeout
passes (10s by default) since last failure. On the other hand, detecting
dead backend takes up to 60s (proxy_connect_timeout) in typical situation
"backend is down and doesn't respond to any packets". This resulted in
suboptimal behaviour in the above situation (up to 23% of requests were
directed to dead backend with default settings).
More detailed description of the problem may be found here (in Russian):
http://mailman.nginx.org/pipermail/nginx-ru/2011-August/042172.html
Fix is to only allow one request after fail_timeout passes, and
mark backend as "live" only if this request succeeds.
Note that with new code backend will not be marked "live" unless "check"
request is completed, and this may take a while in some specific workloads
(e.g. streaming). This is believed to be acceptable.
author | Maxim Dounin <mdounin@mdounin.ru> |
---|---|
date | Wed, 12 Oct 2011 14:22:48 +0000 |
parents | 6b87e875e87a |
children | 1b779cb69dc8 4919fb357a5d |
rev | line source |
---|---|
441
da8c5707af39
nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files
Igor Sysoev <igor@sysoev.ru>
parents:
357
diff
changeset
|
1 |
da8c5707af39
nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files
Igor Sysoev <igor@sysoev.ru>
parents:
357
diff
changeset
|
2 /* |
444
42d11f017717
nginx-0.1.0-2004-09-29-20:00:49 import; remove years from copyright
Igor Sysoev <igor@sysoev.ru>
parents:
441
diff
changeset
|
3 * Copyright (C) Igor Sysoev |
441
da8c5707af39
nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files
Igor Sysoev <igor@sysoev.ru>
parents:
357
diff
changeset
|
4 */ |
da8c5707af39
nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files
Igor Sysoev <igor@sysoev.ru>
parents:
357
diff
changeset
|
5 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
6 #include <ngx_config.h> |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
7 #include <ngx_core.h> |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
8 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
9 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
10 #define NGX_SLAB_PAGE_MASK 3 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
11 #define NGX_SLAB_PAGE 0 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
12 #define NGX_SLAB_BIG 1 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
13 #define NGX_SLAB_EXACT 2 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
14 #define NGX_SLAB_SMALL 3 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
15 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
16 #if (NGX_PTR_SIZE == 4) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
17 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
18 #define NGX_SLAB_PAGE_FREE 0 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
19 #define NGX_SLAB_PAGE_BUSY 0xffffffff |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
20 #define NGX_SLAB_PAGE_START 0x80000000 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
21 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
22 #define NGX_SLAB_SHIFT_MASK 0x0000000f |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
23 #define NGX_SLAB_MAP_MASK 0xffff0000 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
24 #define NGX_SLAB_MAP_SHIFT 16 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
25 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
26 #define NGX_SLAB_BUSY 0xffffffff |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
27 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
28 #else /* (NGX_PTR_SIZE == 8) */ |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
29 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
30 #define NGX_SLAB_PAGE_FREE 0 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
31 #define NGX_SLAB_PAGE_BUSY 0xffffffffffffffff |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
32 #define NGX_SLAB_PAGE_START 0x8000000000000000 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
33 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
34 #define NGX_SLAB_SHIFT_MASK 0x000000000000000f |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
35 #define NGX_SLAB_MAP_MASK 0xffffffff00000000 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
36 #define NGX_SLAB_MAP_SHIFT 32 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
37 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
38 #define NGX_SLAB_BUSY 0xffffffffffffffff |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
39 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
40 #endif |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
41 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
42 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
43 #if (NGX_DEBUG_MALLOC) |
1771
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
44 |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
45 #define ngx_slab_junk(p, size) ngx_memset(p, 0xD0, size) |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
46 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
47 #else |
1771
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
48 |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
49 #if (NGX_FREEBSD) |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
50 |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
51 #define ngx_slab_junk(p, size) \ |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
52 if (ngx_freebsd_debug_malloc) ngx_memset(p, 0xD0, size) |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
53 |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
54 #else |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
55 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
56 #define ngx_slab_junk(p, size) |
1771
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
57 |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
58 #endif |
b794d78e52c5
take MALLOC_OPTIONS=J into account in slab allocator
Igor Sysoev <igor@sysoev.ru>
parents:
1645
diff
changeset
|
59 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
60 #endif |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
61 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
62 static ngx_slab_page_t *ngx_slab_alloc_pages(ngx_slab_pool_t *pool, |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
63 ngx_uint_t pages); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
64 static void ngx_slab_free_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page, |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
65 ngx_uint_t pages); |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
66 static void ngx_slab_error(ngx_slab_pool_t *pool, ngx_uint_t level, |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
67 char *text); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
68 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
69 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
70 static ngx_uint_t ngx_slab_max_size; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
71 static ngx_uint_t ngx_slab_exact_size; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
72 static ngx_uint_t ngx_slab_exact_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
73 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
74 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
75 void |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
76 ngx_slab_init(ngx_slab_pool_t *pool) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
77 { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
78 u_char *p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
79 size_t size; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
80 ngx_int_t m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
81 ngx_uint_t i, n, pages; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
82 ngx_slab_page_t *slots; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
83 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
84 /* STUB */ |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
85 if (ngx_slab_max_size == 0) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
86 ngx_slab_max_size = ngx_pagesize / 2; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
87 ngx_slab_exact_size = ngx_pagesize / (8 * sizeof(uintptr_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
88 for (n = ngx_slab_exact_size; n >>= 1; ngx_slab_exact_shift++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
89 /* void */ |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
90 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
91 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
92 /**/ |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
93 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
94 pool->min_size = 1 << pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
95 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
96 p = (u_char *) pool + sizeof(ngx_slab_pool_t); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
97 size = pool->end - p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
98 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
99 ngx_slab_junk(p, size); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
100 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
101 slots = (ngx_slab_page_t *) p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
102 n = ngx_pagesize_shift - pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
103 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
104 for (i = 0; i < n; i++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
105 slots[i].slab = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
106 slots[i].next = &slots[i]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
107 slots[i].prev = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
108 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
109 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
110 p += n * sizeof(ngx_slab_page_t); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
111 |
1645
31622d9f2c0d
the simple expression has the same precision without overflow
Igor Sysoev <igor@sysoev.ru>
parents:
1018
diff
changeset
|
112 pages = (ngx_uint_t) (size / (ngx_pagesize + sizeof(ngx_slab_page_t))); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
113 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
114 ngx_memzero(p, pages * sizeof(ngx_slab_page_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
115 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
116 pool->pages = (ngx_slab_page_t *) p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
117 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
118 pool->free.prev = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
119 pool->free.next = (ngx_slab_page_t *) p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
120 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
121 pool->pages->slab = pages; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
122 pool->pages->next = &pool->free; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
123 pool->pages->prev = (uintptr_t) &pool->free; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
124 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
125 pool->start = (u_char *) |
1016
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
126 ngx_align_ptr((uintptr_t) p + pages * sizeof(ngx_slab_page_t), |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
127 ngx_pagesize); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
128 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
129 m = pages - (pool->end - pool->start) / ngx_pagesize; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
130 if (m > 0) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
131 pages -= m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
132 pool->pages->slab = pages; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
133 } |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
134 |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
135 pool->log_ctx = &pool->zero; |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
136 pool->zero = '\0'; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
137 } |
357
e260514b9ad4
nginx-0.0.7-2004-06-16-23:36:07 import
Igor Sysoev <igor@sysoev.ru>
parents:
diff
changeset
|
138 |
e260514b9ad4
nginx-0.0.7-2004-06-16-23:36:07 import
Igor Sysoev <igor@sysoev.ru>
parents:
diff
changeset
|
139 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
140 void * |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
141 ngx_slab_alloc(ngx_slab_pool_t *pool, size_t size) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
142 { |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
143 void *p; |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
144 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
145 ngx_shmtx_lock(&pool->mutex); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
146 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
147 p = ngx_slab_alloc_locked(pool, size); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
148 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
149 ngx_shmtx_unlock(&pool->mutex); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
150 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
151 return p; |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
152 } |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
153 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
154 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
155 void * |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
156 ngx_slab_alloc_locked(ngx_slab_pool_t *pool, size_t size) |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
157 { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
158 size_t s; |
881
5ed08a4c3bd3
fix building on 64-bit platforms
Igor Sysoev <igor@sysoev.ru>
parents:
860
diff
changeset
|
159 uintptr_t p, n, m, mask, *bitmap; |
5ed08a4c3bd3
fix building on 64-bit platforms
Igor Sysoev <igor@sysoev.ru>
parents:
860
diff
changeset
|
160 ngx_uint_t i, slot, shift, map; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
161 ngx_slab_page_t *page, *prev, *slots; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
162 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
163 if (size >= ngx_slab_max_size) { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
164 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
165 ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0, |
967 | 166 "slab alloc: %uz", size); |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
167 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
168 page = ngx_slab_alloc_pages(pool, (size + ngx_pagesize - 1) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
169 >> ngx_pagesize_shift); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
170 if (page) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
171 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
172 p += (uintptr_t) pool->start; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
173 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
174 } else { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
175 p = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
176 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
177 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
178 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
179 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
180 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
181 if (size > pool->min_size) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
182 shift = 1; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
183 for (s = size - 1; s >>= 1; shift++) { /* void */ } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
184 slot = shift - pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
185 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
186 } else { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
187 size = pool->min_size; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
188 shift = pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
189 slot = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
190 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
191 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
192 ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0, |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
193 "slab alloc: %uz slot: %ui", size, slot); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
194 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
195 slots = (ngx_slab_page_t *) ((u_char *) pool + sizeof(ngx_slab_pool_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
196 page = slots[slot].next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
197 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
198 if (page->next != page) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
199 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
200 if (shift < ngx_slab_exact_shift) { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
201 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
202 do { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
203 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
204 bitmap = (uintptr_t *) (pool->start + p); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
205 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
206 map = (1 << (ngx_pagesize_shift - shift)) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
207 / (sizeof(uintptr_t) * 8); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
208 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
209 for (n = 0; n < map; n++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
210 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
211 if (bitmap[n] != NGX_SLAB_BUSY) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
212 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
213 for (m = 1, i = 0; m; m <<= 1, i++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
214 if ((bitmap[n] & m)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
215 continue; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
216 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
217 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
218 bitmap[n] |= m; |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
219 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
220 i = ((n * sizeof(uintptr_t) * 8) << shift) |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
221 + (i << shift); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
222 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
223 if (bitmap[n] == NGX_SLAB_BUSY) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
224 for (n = n + 1; n < map; n++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
225 if (bitmap[n] != NGX_SLAB_BUSY) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
226 p = (uintptr_t) bitmap + i; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
227 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
228 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
229 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
230 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
231 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
232 prev = (ngx_slab_page_t *) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
233 (page->prev & ~NGX_SLAB_PAGE_MASK); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
234 prev->next = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
235 page->next->prev = page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
236 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
237 page->next = NULL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
238 page->prev = NGX_SLAB_SMALL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
239 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
240 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
241 p = (uintptr_t) bitmap + i; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
242 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
243 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
244 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
245 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
246 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
247 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
248 page = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
249 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
250 } while (page); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
251 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
252 } else if (shift == ngx_slab_exact_shift) { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
253 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
254 do { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
255 if (page->slab != NGX_SLAB_BUSY) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
256 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
257 for (m = 1, i = 0; m; m <<= 1, i++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
258 if ((page->slab & m)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
259 continue; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
260 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
261 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
262 page->slab |= m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
263 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
264 if (page->slab == NGX_SLAB_BUSY) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
265 prev = (ngx_slab_page_t *) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
266 (page->prev & ~NGX_SLAB_PAGE_MASK); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
267 prev->next = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
268 page->next->prev = page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
269 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
270 page->next = NULL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
271 page->prev = NGX_SLAB_EXACT; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
272 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
273 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
274 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
275 p += i << shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
276 p += (uintptr_t) pool->start; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
277 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
278 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
279 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
280 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
281 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
282 page = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
283 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
284 } while (page); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
285 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
286 } else { /* shift > ngx_slab_exact_shift */ |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
287 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
288 n = ngx_pagesize_shift - (page->slab & NGX_SLAB_SHIFT_MASK); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
289 n = 1 << n; |
1016
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
290 n = ((uintptr_t) 1 << n) - 1; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
291 mask = n << NGX_SLAB_MAP_SHIFT; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
292 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
293 do { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
294 if ((page->slab & NGX_SLAB_MAP_MASK) != mask) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
295 |
881
5ed08a4c3bd3
fix building on 64-bit platforms
Igor Sysoev <igor@sysoev.ru>
parents:
860
diff
changeset
|
296 for (m = (uintptr_t) 1 << NGX_SLAB_MAP_SHIFT, i = 0; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
297 m & mask; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
298 m <<= 1, i++) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
299 { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
300 if ((page->slab & m)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
301 continue; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
302 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
303 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
304 page->slab |= m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
305 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
306 if ((page->slab & NGX_SLAB_MAP_MASK) == mask) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
307 prev = (ngx_slab_page_t *) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
308 (page->prev & ~NGX_SLAB_PAGE_MASK); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
309 prev->next = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
310 page->next->prev = page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
311 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
312 page->next = NULL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
313 page->prev = NGX_SLAB_BIG; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
314 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
315 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
316 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
317 p += i << shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
318 p += (uintptr_t) pool->start; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
319 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
320 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
321 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
322 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
323 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
324 page = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
325 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
326 } while (page); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
327 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
328 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
329 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
330 page = ngx_slab_alloc_pages(pool, 1); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
331 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
332 if (page) { |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
333 if (shift < ngx_slab_exact_shift) { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
334 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
335 bitmap = (uintptr_t *) (pool->start + p); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
336 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
337 s = 1 << shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
338 n = (1 << (ngx_pagesize_shift - shift)) / 8 / s; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
339 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
340 if (n == 0) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
341 n = 1; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
342 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
343 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
344 bitmap[0] = (2 << n) - 1; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
345 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
346 map = (1 << (ngx_pagesize_shift - shift)) / (sizeof(uintptr_t) * 8); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
347 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
348 for (i = 1; i < map; i++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
349 bitmap[i] = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
350 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
351 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
352 page->slab = shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
353 page->next = &slots[slot]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
354 page->prev = (uintptr_t) &slots[slot] | NGX_SLAB_SMALL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
355 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
356 slots[slot].next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
357 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
358 p = ((page - pool->pages) << ngx_pagesize_shift) + s * n; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
359 p += (uintptr_t) pool->start; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
360 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
361 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
362 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
363 } else if (shift == ngx_slab_exact_shift) { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
364 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
365 page->slab = 1; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
366 page->next = &slots[slot]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
367 page->prev = (uintptr_t) &slots[slot] | NGX_SLAB_EXACT; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
368 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
369 slots[slot].next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
370 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
371 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
372 p += (uintptr_t) pool->start; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
373 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
374 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
375 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
376 } else { /* shift > ngx_slab_exact_shift */ |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
377 |
881
5ed08a4c3bd3
fix building on 64-bit platforms
Igor Sysoev <igor@sysoev.ru>
parents:
860
diff
changeset
|
378 page->slab = ((uintptr_t) 1 << NGX_SLAB_MAP_SHIFT) | shift; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
379 page->next = &slots[slot]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
380 page->prev = (uintptr_t) &slots[slot] | NGX_SLAB_BIG; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
381 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
382 slots[slot].next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
383 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
384 p = (page - pool->pages) << ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
385 p += (uintptr_t) pool->start; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
386 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
387 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
388 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
389 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
390 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
391 p = 0; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
392 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
393 done: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
394 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
395 ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0, "slab alloc: %p", p); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
396 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
397 return (void *) p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
398 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
399 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
400 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
401 void |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
402 ngx_slab_free(ngx_slab_pool_t *pool, void *p) |
357
e260514b9ad4
nginx-0.0.7-2004-06-16-23:36:07 import
Igor Sysoev <igor@sysoev.ru>
parents:
diff
changeset
|
403 { |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
404 ngx_shmtx_lock(&pool->mutex); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
405 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
406 ngx_slab_free_locked(pool, p); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
407 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
408 ngx_shmtx_unlock(&pool->mutex); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
409 } |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
410 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
411 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
412 void |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
413 ngx_slab_free_locked(ngx_slab_pool_t *pool, void *p) |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
414 { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
415 size_t size; |
1018
06306a20b8bb
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
1016
diff
changeset
|
416 uintptr_t slab, m, *bitmap; |
06306a20b8bb
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
1016
diff
changeset
|
417 ngx_uint_t n, type, slot, shift, map; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
418 ngx_slab_page_t *slots, *page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
419 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
420 ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0, "slab free: %p", p); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
421 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
422 if ((u_char *) p < pool->start || (u_char *) p > pool->end) { |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
423 ngx_slab_error(pool, NGX_LOG_ALERT, "ngx_slab_free(): outside of pool"); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
424 goto fail; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
425 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
426 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
427 n = ((u_char *) p - pool->start) >> ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
428 page = &pool->pages[n]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
429 slab = page->slab; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
430 type = page->prev & NGX_SLAB_PAGE_MASK; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
431 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
432 switch (type) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
433 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
434 case NGX_SLAB_SMALL: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
435 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
436 shift = slab & NGX_SLAB_SHIFT_MASK; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
437 size = 1 << shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
438 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
439 if ((uintptr_t) p & (size - 1)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
440 goto wrong_chunk; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
441 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
442 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
443 n = ((uintptr_t) p & (ngx_pagesize - 1)) >> shift; |
1016
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
444 m = (uintptr_t) 1 << (n & (sizeof(uintptr_t) * 8 - 1)); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
445 n /= (sizeof(uintptr_t) * 8); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
446 bitmap = (uintptr_t *) ((uintptr_t) p & ~(ngx_pagesize - 1)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
447 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
448 if (bitmap[n] & m) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
449 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
450 if (page->next == NULL) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
451 slots = (ngx_slab_page_t *) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
452 ((u_char *) pool + sizeof(ngx_slab_pool_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
453 slot = shift - pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
454 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
455 page->next = slots[slot].next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
456 slots[slot].next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
457 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
458 page->prev = (uintptr_t) &slots[slot] | NGX_SLAB_SMALL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
459 page->next->prev = (uintptr_t) page | NGX_SLAB_SMALL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
460 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
461 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
462 bitmap[n] &= ~m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
463 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
464 n = (1 << (ngx_pagesize_shift - shift)) / 8 / (1 << shift); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
465 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
466 if (n == 0) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
467 n = 1; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
468 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
469 |
1016
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
470 if (bitmap[0] & ~(((uintptr_t) 1 << n) - 1)) { |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
471 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
472 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
473 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
474 map = (1 << (ngx_pagesize_shift - shift)) / (sizeof(uintptr_t) * 8); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
475 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
476 for (n = 1; n < map; n++) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
477 if (bitmap[n]) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
478 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
479 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
480 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
481 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
482 ngx_slab_free_pages(pool, page, 1); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
483 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
484 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
485 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
486 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
487 goto chunk_already_free; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
488 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
489 case NGX_SLAB_EXACT: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
490 |
1016
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
491 m = (uintptr_t) 1 << |
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
492 (((uintptr_t) p & (ngx_pagesize - 1)) >> ngx_slab_exact_shift); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
493 size = ngx_slab_exact_size; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
494 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
495 if ((uintptr_t) p & (size - 1)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
496 goto wrong_chunk; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
497 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
498 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
499 if (slab & m) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
500 if (slab == NGX_SLAB_BUSY) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
501 slots = (ngx_slab_page_t *) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
502 ((u_char *) pool + sizeof(ngx_slab_pool_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
503 slot = ngx_slab_exact_shift - pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
504 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
505 page->next = slots[slot].next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
506 slots[slot].next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
507 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
508 page->prev = (uintptr_t) &slots[slot] | NGX_SLAB_EXACT; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
509 page->next->prev = (uintptr_t) page | NGX_SLAB_EXACT; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
510 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
511 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
512 page->slab &= ~m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
513 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
514 if (page->slab) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
515 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
516 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
517 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
518 ngx_slab_free_pages(pool, page, 1); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
519 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
520 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
521 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
522 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
523 goto chunk_already_free; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
524 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
525 case NGX_SLAB_BIG: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
526 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
527 shift = slab & NGX_SLAB_SHIFT_MASK; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
528 size = 1 << shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
529 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
530 if ((uintptr_t) p & (size - 1)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
531 goto wrong_chunk; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
532 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
533 |
1016
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
534 m = (uintptr_t) 1 << ((((uintptr_t) p & (ngx_pagesize - 1)) >> shift) |
d0e8c81d3bb7
fix slab allocator on 64-bit platfroms
Igor Sysoev <igor@sysoev.ru>
parents:
967
diff
changeset
|
535 + NGX_SLAB_MAP_SHIFT); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
536 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
537 if (slab & m) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
538 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
539 if (page->next == NULL) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
540 slots = (ngx_slab_page_t *) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
541 ((u_char *) pool + sizeof(ngx_slab_pool_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
542 slot = shift - pool->min_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
543 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
544 page->next = slots[slot].next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
545 slots[slot].next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
546 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
547 page->prev = (uintptr_t) &slots[slot] | NGX_SLAB_BIG; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
548 page->next->prev = (uintptr_t) page | NGX_SLAB_BIG; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
549 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
550 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
551 page->slab &= ~m; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
552 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
553 if (page->slab & NGX_SLAB_MAP_MASK) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
554 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
555 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
556 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
557 ngx_slab_free_pages(pool, page, 1); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
558 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
559 goto done; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
560 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
561 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
562 goto chunk_already_free; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
563 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
564 case NGX_SLAB_PAGE: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
565 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
566 if ((uintptr_t) p & (ngx_pagesize - 1)) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
567 goto wrong_chunk; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
568 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
569 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
570 if (slab == NGX_SLAB_PAGE_FREE) { |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
571 ngx_slab_error(pool, NGX_LOG_ALERT, |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
572 "ngx_slab_free(): page is already free"); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
573 goto fail; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
574 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
575 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
576 if (slab == NGX_SLAB_PAGE_BUSY) { |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
577 ngx_slab_error(pool, NGX_LOG_ALERT, |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
578 "ngx_slab_free(): pointer to wrong page"); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
579 goto fail; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
580 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
581 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
582 n = ((u_char *) p - pool->start) >> ngx_pagesize_shift; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
583 size = slab & ~NGX_SLAB_PAGE_START; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
584 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
585 ngx_slab_free_pages(pool, &pool->pages[n], size); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
586 |
2711
6b87e875e87a
fix building by BCC without NGX_DEBUG_MALLOC
Igor Sysoev <igor@sysoev.ru>
parents:
2611
diff
changeset
|
587 ngx_slab_junk(p, size << ngx_pagesize_shift); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
588 |
2711
6b87e875e87a
fix building by BCC without NGX_DEBUG_MALLOC
Igor Sysoev <igor@sysoev.ru>
parents:
2611
diff
changeset
|
589 return; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
590 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
591 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
592 /* not reached */ |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
593 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
594 return; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
595 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
596 done: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
597 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
598 ngx_slab_junk(p, size); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
599 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
600 return; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
601 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
602 wrong_chunk: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
603 |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
604 ngx_slab_error(pool, NGX_LOG_ALERT, |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
605 "ngx_slab_free(): pointer to wrong chunk"); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
606 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
607 goto fail; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
608 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
609 chunk_already_free: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
610 |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
611 ngx_slab_error(pool, NGX_LOG_ALERT, |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
612 "ngx_slab_free(): chunk is already free"); |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
613 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
614 fail: |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
615 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
616 return; |
357
e260514b9ad4
nginx-0.0.7-2004-06-16-23:36:07 import
Igor Sysoev <igor@sysoev.ru>
parents:
diff
changeset
|
617 } |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
618 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
619 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
620 static ngx_slab_page_t * |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
621 ngx_slab_alloc_pages(ngx_slab_pool_t *pool, ngx_uint_t pages) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
622 { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
623 ngx_slab_page_t *page, *p; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
624 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
625 for (page = pool->free.next; page != &pool->free; page = page->next) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
626 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
627 if (page->slab >= pages) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
628 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
629 if (page->slab > pages) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
630 page[pages].slab = page->slab - pages; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
631 page[pages].next = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
632 page[pages].prev = page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
633 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
634 p = (ngx_slab_page_t *) page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
635 p->next = &page[pages]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
636 page->next->prev = (uintptr_t) &page[pages]; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
637 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
638 } else { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
639 p = (ngx_slab_page_t *) page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
640 p->next = page->next; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
641 page->next->prev = page->prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
642 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
643 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
644 page->slab = pages | NGX_SLAB_PAGE_START; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
645 page->next = NULL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
646 page->prev = NGX_SLAB_PAGE; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
647 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
648 if (--pages == 0) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
649 return page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
650 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
651 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
652 for (p = page + 1; pages; pages--) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
653 p->slab = NGX_SLAB_PAGE_BUSY; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
654 p->next = NULL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
655 p->prev = NGX_SLAB_PAGE; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
656 p++; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
657 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
658 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
659 return page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
660 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
661 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
662 |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
663 ngx_slab_error(pool, NGX_LOG_CRIT, "ngx_slab_alloc() failed: no memory"); |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
664 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
665 return NULL; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
666 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
667 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
668 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
669 static void |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
670 ngx_slab_free_pages(ngx_slab_pool_t *pool, ngx_slab_page_t *page, |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
671 ngx_uint_t pages) |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
672 { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
673 ngx_slab_page_t *prev; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
674 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
675 page->slab = pages--; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
676 |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
677 if (pages) { |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
678 ngx_memzero(&page[1], pages * sizeof(ngx_slab_page_t)); |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
679 } |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
680 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
681 if (page->next) { |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
682 prev = (ngx_slab_page_t *) (page->prev & ~NGX_SLAB_PAGE_MASK); |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
683 prev->next = page->next; |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
684 page->next->prev = page->prev; |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
685 } |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
686 |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
687 page->prev = (uintptr_t) &pool->free; |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
688 page->next = pool->free.next; |
966
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
689 |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
690 page->next->prev = (uintptr_t) page; |
715336c243e1
many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked()
Igor Sysoev <igor@sysoev.ru>
parents:
881
diff
changeset
|
691 |
860
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
692 pool->free.next = page; |
201d017ea470
slab allocator in shared memory
Igor Sysoev <igor@sysoev.ru>
parents:
826
diff
changeset
|
693 } |
2611
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
694 |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
695 |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
696 static void |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
697 ngx_slab_error(ngx_slab_pool_t *pool, ngx_uint_t level, char *text) |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
698 { |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
699 ngx_log_error(level, ngx_cycle->log, 0, "%s%s", text, pool->log_ctx); |
2bce3f6416c6
improve ngx_slab_alloc() error logging
Igor Sysoev <igor@sysoev.ru>
parents:
2610
diff
changeset
|
700 } |