nginx: src/core/ngx_slab.c annotate

annotate src/core/ngx_slab.c @ 4159:718f2154b813 stable-1.0

Merging r3933, r4154: A new fix for the case when ssl_session_cache defined, but ssl is not enabled in any server. The previous r1033 does not help when unused zone becomes used after reconfiguration, so it is backed out. The initial thought was to make SSL modules independed from SSL implementation and to keep OpenSSL code dependance as much as in separate files.

author	Igor Sysoev <igor@sysoev.ru>
date	Fri, 30 Sep 2011 13:36:33 +0000
parents	6b87e875e87a
children	1b779cb69dc8 4919fb357a5d

rev	line source
441 da8c5707af39 nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files Igor Sysoev <igor@sysoev.ru> parents: 357 diff changeset	1
da8c5707af39 nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files Igor Sysoev <igor@sysoev.ru> parents: 357 diff changeset	2 /*
444 42d11f017717 nginx-0.1.0-2004-09-29-20:00:49 import; remove years from copyright Igor Sysoev <igor@sysoev.ru> parents: 441 diff changeset	3 * Copyright (C) Igor Sysoev
441 da8c5707af39 nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files Igor Sysoev <igor@sysoev.ru> parents: 357 diff changeset	4 */
da8c5707af39 nginx-0.1.0-2004-09-28-12:34:51 import; set copyright and remove unused files Igor Sysoev <igor@sysoev.ru> parents: 357 diff changeset	5
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	6 #include <ngx_config.h>
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	7 #include <ngx_core.h>
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	8
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	9
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	10 #define NGX_SLAB_PAGE_MASK 3
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	11 #define NGX_SLAB_PAGE 0
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	12 #define NGX_SLAB_BIG 1
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	13 #define NGX_SLAB_EXACT 2
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	14 #define NGX_SLAB_SMALL 3
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	15
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	16 #if (NGX_PTR_SIZE == 4)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	17
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	18 #define NGX_SLAB_PAGE_FREE 0
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	19 #define NGX_SLAB_PAGE_BUSY 0xffffffff
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	20 #define NGX_SLAB_PAGE_START 0x80000000
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	21
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	22 #define NGX_SLAB_SHIFT_MASK 0x0000000f
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	23 #define NGX_SLAB_MAP_MASK 0xffff0000
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	24 #define NGX_SLAB_MAP_SHIFT 16
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	25
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	26 #define NGX_SLAB_BUSY 0xffffffff
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	27
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	28 #else /* (NGX_PTR_SIZE == 8) */
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	29
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	30 #define NGX_SLAB_PAGE_FREE 0
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	31 #define NGX_SLAB_PAGE_BUSY 0xffffffffffffffff
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	32 #define NGX_SLAB_PAGE_START 0x8000000000000000
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	33
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	34 #define NGX_SLAB_SHIFT_MASK 0x000000000000000f
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	35 #define NGX_SLAB_MAP_MASK 0xffffffff00000000
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	36 #define NGX_SLAB_MAP_SHIFT 32
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	37
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	38 #define NGX_SLAB_BUSY 0xffffffffffffffff
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	39
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	40 #endif
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	41
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	42
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	43 #if (NGX_DEBUG_MALLOC)
1771 b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	44
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	45 #define ngx_slab_junk(p, size) ngx_memset(p, 0xD0, size)
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	46
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	47 #else
1771 b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	48
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	49 #if (NGX_FREEBSD)
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	50
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	51 #define ngx_slab_junk(p, size) \
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	52 if (ngx_freebsd_debug_malloc) ngx_memset(p, 0xD0, size)
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	53
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	54 #else
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	55
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	56 #define ngx_slab_junk(p, size)
1771 b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	57
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	58 #endif
b794d78e52c5 take MALLOC_OPTIONS=J into account in slab allocator Igor Sysoev <igor@sysoev.ru> parents: 1645 diff changeset	59
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	60 #endif
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	61
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	62 static ngx_slab_page_t ngx_slab_alloc_pages(ngx_slab_pool_t pool,
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	63 ngx_uint_t pages);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	64 static void ngx_slab_free_pages(ngx_slab_pool_t pool, ngx_slab_page_t page,
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	65 ngx_uint_t pages);
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	66 static void ngx_slab_error(ngx_slab_pool_t *pool, ngx_uint_t level,
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	67 char *text);
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	68
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	69
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	70 static ngx_uint_t ngx_slab_max_size;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	71 static ngx_uint_t ngx_slab_exact_size;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	72 static ngx_uint_t ngx_slab_exact_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	73
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	74
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	75 void
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	76 ngx_slab_init(ngx_slab_pool_t *pool)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	77 {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	78 u_char *p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	79 size_t size;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	80 ngx_int_t m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	81 ngx_uint_t i, n, pages;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	82 ngx_slab_page_t *slots;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	83
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	84 /* STUB */
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	85 if (ngx_slab_max_size == 0) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	86 ngx_slab_max_size = ngx_pagesize / 2;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	87 ngx_slab_exact_size = ngx_pagesize / (8 * sizeof(uintptr_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	88 for (n = ngx_slab_exact_size; n >>= 1; ngx_slab_exact_shift++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	89 /* void */
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	90 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	91 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	92 /**/
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	93
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	94 pool->min_size = 1 << pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	95
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	96 p = (u_char *) pool + sizeof(ngx_slab_pool_t);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	97 size = pool->end - p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	98
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	99 ngx_slab_junk(p, size);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	100
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	101 slots = (ngx_slab_page_t *) p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	102 n = ngx_pagesize_shift - pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	103
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	104 for (i = 0; i < n; i++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	105 slots[i].slab = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	106 slots[i].next = &slots[i];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	107 slots[i].prev = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	108 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	109
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	110 p += n * sizeof(ngx_slab_page_t);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	111
1645 31622d9f2c0d the simple expression has the same precision without overflow Igor Sysoev <igor@sysoev.ru> parents: 1018 diff changeset	112 pages = (ngx_uint_t) (size / (ngx_pagesize + sizeof(ngx_slab_page_t)));
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	113
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	114 ngx_memzero(p, pages * sizeof(ngx_slab_page_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	115
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	116 pool->pages = (ngx_slab_page_t *) p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	117
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	118 pool->free.prev = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	119 pool->free.next = (ngx_slab_page_t *) p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	120
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	121 pool->pages->slab = pages;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	122 pool->pages->next = &pool->free;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	123 pool->pages->prev = (uintptr_t) &pool->free;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	124
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	125 pool->start = (u_char *)
1016 d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	126 ngx_align_ptr((uintptr_t) p + pages * sizeof(ngx_slab_page_t),
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	127 ngx_pagesize);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	128
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	129 m = pages - (pool->end - pool->start) / ngx_pagesize;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	130 if (m > 0) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	131 pages -= m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	132 pool->pages->slab = pages;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	133 }
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	134
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	135 pool->log_ctx = &pool->zero;
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	136 pool->zero = '\0';
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	137 }
357 e260514b9ad4 nginx-0.0.7-2004-06-16-23:36:07 import Igor Sysoev <igor@sysoev.ru> parents: diff changeset	138
e260514b9ad4 nginx-0.0.7-2004-06-16-23:36:07 import Igor Sysoev <igor@sysoev.ru> parents: diff changeset	139
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	140 void *
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	141 ngx_slab_alloc(ngx_slab_pool_t *pool, size_t size)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	142 {
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	143 void *p;
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	144
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	145 ngx_shmtx_lock(&pool->mutex);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	146
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	147 p = ngx_slab_alloc_locked(pool, size);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	148
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	149 ngx_shmtx_unlock(&pool->mutex);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	150
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	151 return p;
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	152 }
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	153
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	154
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	155 void *
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	156 ngx_slab_alloc_locked(ngx_slab_pool_t *pool, size_t size)
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	157 {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	158 size_t s;
881 5ed08a4c3bd3 fix building on 64-bit platforms Igor Sysoev <igor@sysoev.ru> parents: 860 diff changeset	159 uintptr_t p, n, m, mask, *bitmap;
5ed08a4c3bd3 fix building on 64-bit platforms Igor Sysoev <igor@sysoev.ru> parents: 860 diff changeset	160 ngx_uint_t i, slot, shift, map;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	161 ngx_slab_page_t page, prev, *slots;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	162
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	163 if (size >= ngx_slab_max_size) {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	164
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	165 ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0,
967 1cb178720355 style fix Igor Sysoev <igor@sysoev.ru> parents: 966 diff changeset	166 "slab alloc: %uz", size);
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	167
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	168 page = ngx_slab_alloc_pages(pool, (size + ngx_pagesize - 1)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	169 >> ngx_pagesize_shift);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	170 if (page) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	171 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	172 p += (uintptr_t) pool->start;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	173
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	174 } else {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	175 p = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	176 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	177
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	178 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	179 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	180
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	181 if (size > pool->min_size) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	182 shift = 1;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	183 for (s = size - 1; s >>= 1; shift++) { /* void */ }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	184 slot = shift - pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	185
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	186 } else {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	187 size = pool->min_size;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	188 shift = pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	189 slot = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	190 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	191
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	192 ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0,
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	193 "slab alloc: %uz slot: %ui", size, slot);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	194
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	195 slots = (ngx_slab_page_t ) ((u_char ) pool + sizeof(ngx_slab_pool_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	196 page = slots[slot].next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	197
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	198 if (page->next != page) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	199
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	200 if (shift < ngx_slab_exact_shift) {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	201
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	202 do {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	203 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	204 bitmap = (uintptr_t *) (pool->start + p);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	205
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	206 map = (1 << (ngx_pagesize_shift - shift))
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	207 / (sizeof(uintptr_t) * 8);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	208
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	209 for (n = 0; n < map; n++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	210
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	211 if (bitmap[n] != NGX_SLAB_BUSY) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	212
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	213 for (m = 1, i = 0; m; m <<= 1, i++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	214 if ((bitmap[n] & m)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	215 continue;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	216 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	217
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	218 bitmap[n] \|= m;
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	219
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	220 i = ((n * sizeof(uintptr_t) * 8) << shift)
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	221 + (i << shift);
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	222
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	223 if (bitmap[n] == NGX_SLAB_BUSY) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	224 for (n = n + 1; n < map; n++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	225 if (bitmap[n] != NGX_SLAB_BUSY) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	226 p = (uintptr_t) bitmap + i;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	227
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	228 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	229 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	230 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	231
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	232 prev = (ngx_slab_page_t *)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	233 (page->prev & ~NGX_SLAB_PAGE_MASK);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	234 prev->next = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	235 page->next->prev = page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	236
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	237 page->next = NULL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	238 page->prev = NGX_SLAB_SMALL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	239 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	240
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	241 p = (uintptr_t) bitmap + i;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	242
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	243 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	244 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	245 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	246 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	247
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	248 page = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	249
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	250 } while (page);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	251
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	252 } else if (shift == ngx_slab_exact_shift) {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	253
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	254 do {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	255 if (page->slab != NGX_SLAB_BUSY) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	256
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	257 for (m = 1, i = 0; m; m <<= 1, i++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	258 if ((page->slab & m)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	259 continue;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	260 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	261
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	262 page->slab \|= m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	263
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	264 if (page->slab == NGX_SLAB_BUSY) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	265 prev = (ngx_slab_page_t *)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	266 (page->prev & ~NGX_SLAB_PAGE_MASK);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	267 prev->next = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	268 page->next->prev = page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	269
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	270 page->next = NULL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	271 page->prev = NGX_SLAB_EXACT;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	272 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	273
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	274 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	275 p += i << shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	276 p += (uintptr_t) pool->start;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	277
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	278 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	279 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	280 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	281
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	282 page = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	283
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	284 } while (page);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	285
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	286 } else { /* shift > ngx_slab_exact_shift */
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	287
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	288 n = ngx_pagesize_shift - (page->slab & NGX_SLAB_SHIFT_MASK);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	289 n = 1 << n;
1016 d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	290 n = ((uintptr_t) 1 << n) - 1;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	291 mask = n << NGX_SLAB_MAP_SHIFT;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	292
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	293 do {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	294 if ((page->slab & NGX_SLAB_MAP_MASK) != mask) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	295
881 5ed08a4c3bd3 fix building on 64-bit platforms Igor Sysoev <igor@sysoev.ru> parents: 860 diff changeset	296 for (m = (uintptr_t) 1 << NGX_SLAB_MAP_SHIFT, i = 0;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	297 m & mask;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	298 m <<= 1, i++)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	299 {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	300 if ((page->slab & m)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	301 continue;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	302 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	303
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	304 page->slab \|= m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	305
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	306 if ((page->slab & NGX_SLAB_MAP_MASK) == mask) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	307 prev = (ngx_slab_page_t *)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	308 (page->prev & ~NGX_SLAB_PAGE_MASK);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	309 prev->next = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	310 page->next->prev = page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	311
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	312 page->next = NULL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	313 page->prev = NGX_SLAB_BIG;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	314 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	315
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	316 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	317 p += i << shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	318 p += (uintptr_t) pool->start;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	319
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	320 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	321 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	322 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	323
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	324 page = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	325
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	326 } while (page);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	327 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	328 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	329
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	330 page = ngx_slab_alloc_pages(pool, 1);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	331
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	332 if (page) {
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	333 if (shift < ngx_slab_exact_shift) {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	334 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	335 bitmap = (uintptr_t *) (pool->start + p);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	336
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	337 s = 1 << shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	338 n = (1 << (ngx_pagesize_shift - shift)) / 8 / s;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	339
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	340 if (n == 0) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	341 n = 1;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	342 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	343
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	344 bitmap[0] = (2 << n) - 1;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	345
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	346 map = (1 << (ngx_pagesize_shift - shift)) / (sizeof(uintptr_t) * 8);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	347
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	348 for (i = 1; i < map; i++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	349 bitmap[i] = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	350 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	351
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	352 page->slab = shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	353 page->next = &slots[slot];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	354 page->prev = (uintptr_t) &slots[slot] \| NGX_SLAB_SMALL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	355
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	356 slots[slot].next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	357
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	358 p = ((page - pool->pages) << ngx_pagesize_shift) + s * n;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	359 p += (uintptr_t) pool->start;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	360
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	361 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	362
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	363 } else if (shift == ngx_slab_exact_shift) {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	364
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	365 page->slab = 1;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	366 page->next = &slots[slot];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	367 page->prev = (uintptr_t) &slots[slot] \| NGX_SLAB_EXACT;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	368
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	369 slots[slot].next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	370
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	371 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	372 p += (uintptr_t) pool->start;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	373
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	374 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	375
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	376 } else { /* shift > ngx_slab_exact_shift */
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	377
881 5ed08a4c3bd3 fix building on 64-bit platforms Igor Sysoev <igor@sysoev.ru> parents: 860 diff changeset	378 page->slab = ((uintptr_t) 1 << NGX_SLAB_MAP_SHIFT) \| shift;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	379 page->next = &slots[slot];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	380 page->prev = (uintptr_t) &slots[slot] \| NGX_SLAB_BIG;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	381
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	382 slots[slot].next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	383
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	384 p = (page - pool->pages) << ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	385 p += (uintptr_t) pool->start;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	386
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	387 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	388 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	389 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	390
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	391 p = 0;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	392
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	393 done:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	394
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	395 ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0, "slab alloc: %p", p);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	396
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	397 return (void *) p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	398 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	399
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	400
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	401 void
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	402 ngx_slab_free(ngx_slab_pool_t pool, void p)
357 e260514b9ad4 nginx-0.0.7-2004-06-16-23:36:07 import Igor Sysoev <igor@sysoev.ru> parents: diff changeset	403 {
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	404 ngx_shmtx_lock(&pool->mutex);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	405
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	406 ngx_slab_free_locked(pool, p);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	407
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	408 ngx_shmtx_unlock(&pool->mutex);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	409 }
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	410
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	411
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	412 void
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	413 ngx_slab_free_locked(ngx_slab_pool_t pool, void p)
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	414 {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	415 size_t size;
1018 06306a20b8bb fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 1016 diff changeset	416 uintptr_t slab, m, *bitmap;
06306a20b8bb fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 1016 diff changeset	417 ngx_uint_t n, type, slot, shift, map;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	418 ngx_slab_page_t slots, page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	419
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	420 ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, ngx_cycle->log, 0, "slab free: %p", p);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	421
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	422 if ((u_char ) p < pool->start \|\| (u_char ) p > pool->end) {
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	423 ngx_slab_error(pool, NGX_LOG_ALERT, "ngx_slab_free(): outside of pool");
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	424 goto fail;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	425 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	426
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	427 n = ((u_char *) p - pool->start) >> ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	428 page = &pool->pages[n];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	429 slab = page->slab;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	430 type = page->prev & NGX_SLAB_PAGE_MASK;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	431
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	432 switch (type) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	433
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	434 case NGX_SLAB_SMALL:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	435
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	436 shift = slab & NGX_SLAB_SHIFT_MASK;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	437 size = 1 << shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	438
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	439 if ((uintptr_t) p & (size - 1)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	440 goto wrong_chunk;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	441 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	442
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	443 n = ((uintptr_t) p & (ngx_pagesize - 1)) >> shift;
1016 d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	444 m = (uintptr_t) 1 << (n & (sizeof(uintptr_t) * 8 - 1));
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	445 n /= (sizeof(uintptr_t) * 8);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	446 bitmap = (uintptr_t *) ((uintptr_t) p & ~(ngx_pagesize - 1));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	447
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	448 if (bitmap[n] & m) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	449
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	450 if (page->next == NULL) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	451 slots = (ngx_slab_page_t *)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	452 ((u_char *) pool + sizeof(ngx_slab_pool_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	453 slot = shift - pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	454
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	455 page->next = slots[slot].next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	456 slots[slot].next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	457
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	458 page->prev = (uintptr_t) &slots[slot] \| NGX_SLAB_SMALL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	459 page->next->prev = (uintptr_t) page \| NGX_SLAB_SMALL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	460 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	461
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	462 bitmap[n] &= ~m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	463
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	464 n = (1 << (ngx_pagesize_shift - shift)) / 8 / (1 << shift);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	465
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	466 if (n == 0) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	467 n = 1;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	468 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	469
1016 d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	470 if (bitmap[0] & ~(((uintptr_t) 1 << n) - 1)) {
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	471 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	472 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	473
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	474 map = (1 << (ngx_pagesize_shift - shift)) / (sizeof(uintptr_t) * 8);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	475
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	476 for (n = 1; n < map; n++) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	477 if (bitmap[n]) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	478 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	479 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	480 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	481
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	482 ngx_slab_free_pages(pool, page, 1);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	483
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	484 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	485 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	486
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	487 goto chunk_already_free;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	488
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	489 case NGX_SLAB_EXACT:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	490
1016 d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	491 m = (uintptr_t) 1 <<
d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	492 (((uintptr_t) p & (ngx_pagesize - 1)) >> ngx_slab_exact_shift);
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	493 size = ngx_slab_exact_size;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	494
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	495 if ((uintptr_t) p & (size - 1)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	496 goto wrong_chunk;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	497 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	498
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	499 if (slab & m) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	500 if (slab == NGX_SLAB_BUSY) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	501 slots = (ngx_slab_page_t *)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	502 ((u_char *) pool + sizeof(ngx_slab_pool_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	503 slot = ngx_slab_exact_shift - pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	504
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	505 page->next = slots[slot].next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	506 slots[slot].next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	507
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	508 page->prev = (uintptr_t) &slots[slot] \| NGX_SLAB_EXACT;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	509 page->next->prev = (uintptr_t) page \| NGX_SLAB_EXACT;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	510 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	511
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	512 page->slab &= ~m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	513
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	514 if (page->slab) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	515 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	516 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	517
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	518 ngx_slab_free_pages(pool, page, 1);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	519
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	520 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	521 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	522
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	523 goto chunk_already_free;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	524
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	525 case NGX_SLAB_BIG:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	526
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	527 shift = slab & NGX_SLAB_SHIFT_MASK;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	528 size = 1 << shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	529
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	530 if ((uintptr_t) p & (size - 1)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	531 goto wrong_chunk;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	532 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	533
1016 d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	534 m = (uintptr_t) 1 << ((((uintptr_t) p & (ngx_pagesize - 1)) >> shift)
d0e8c81d3bb7 fix slab allocator on 64-bit platfroms Igor Sysoev <igor@sysoev.ru> parents: 967 diff changeset	535 + NGX_SLAB_MAP_SHIFT);
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	536
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	537 if (slab & m) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	538
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	539 if (page->next == NULL) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	540 slots = (ngx_slab_page_t *)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	541 ((u_char *) pool + sizeof(ngx_slab_pool_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	542 slot = shift - pool->min_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	543
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	544 page->next = slots[slot].next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	545 slots[slot].next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	546
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	547 page->prev = (uintptr_t) &slots[slot] \| NGX_SLAB_BIG;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	548 page->next->prev = (uintptr_t) page \| NGX_SLAB_BIG;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	549 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	550
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	551 page->slab &= ~m;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	552
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	553 if (page->slab & NGX_SLAB_MAP_MASK) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	554 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	555 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	556
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	557 ngx_slab_free_pages(pool, page, 1);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	558
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	559 goto done;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	560 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	561
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	562 goto chunk_already_free;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	563
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	564 case NGX_SLAB_PAGE:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	565
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	566 if ((uintptr_t) p & (ngx_pagesize - 1)) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	567 goto wrong_chunk;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	568 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	569
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	570 if (slab == NGX_SLAB_PAGE_FREE) {
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	571 ngx_slab_error(pool, NGX_LOG_ALERT,
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	572 "ngx_slab_free(): page is already free");
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	573 goto fail;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	574 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	575
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	576 if (slab == NGX_SLAB_PAGE_BUSY) {
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	577 ngx_slab_error(pool, NGX_LOG_ALERT,
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	578 "ngx_slab_free(): pointer to wrong page");
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	579 goto fail;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	580 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	581
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	582 n = ((u_char *) p - pool->start) >> ngx_pagesize_shift;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	583 size = slab & ~NGX_SLAB_PAGE_START;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	584
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	585 ngx_slab_free_pages(pool, &pool->pages[n], size);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	586
2711 6b87e875e87a fix building by BCC without NGX_DEBUG_MALLOC Igor Sysoev <igor@sysoev.ru> parents: 2611 diff changeset	587 ngx_slab_junk(p, size << ngx_pagesize_shift);
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	588
2711 6b87e875e87a fix building by BCC without NGX_DEBUG_MALLOC Igor Sysoev <igor@sysoev.ru> parents: 2611 diff changeset	589 return;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	590 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	591
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	592 /* not reached */
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	593
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	594 return;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	595
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	596 done:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	597
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	598 ngx_slab_junk(p, size);
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	599
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	600 return;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	601
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	602 wrong_chunk:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	603
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	604 ngx_slab_error(pool, NGX_LOG_ALERT,
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	605 "ngx_slab_free(): pointer to wrong chunk");
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	606
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	607 goto fail;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	608
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	609 chunk_already_free:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	610
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	611 ngx_slab_error(pool, NGX_LOG_ALERT,
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	612 "ngx_slab_free(): chunk is already free");
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	613
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	614 fail:
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	615
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	616 return;
357 e260514b9ad4 nginx-0.0.7-2004-06-16-23:36:07 import Igor Sysoev <igor@sysoev.ru> parents: diff changeset	617 }
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	618
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	619
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	620 static ngx_slab_page_t *
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	621 ngx_slab_alloc_pages(ngx_slab_pool_t *pool, ngx_uint_t pages)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	622 {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	623 ngx_slab_page_t page, p;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	624
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	625 for (page = pool->free.next; page != &pool->free; page = page->next) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	626
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	627 if (page->slab >= pages) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	628
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	629 if (page->slab > pages) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	630 page[pages].slab = page->slab - pages;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	631 page[pages].next = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	632 page[pages].prev = page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	633
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	634 p = (ngx_slab_page_t *) page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	635 p->next = &page[pages];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	636 page->next->prev = (uintptr_t) &page[pages];
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	637
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	638 } else {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	639 p = (ngx_slab_page_t *) page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	640 p->next = page->next;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	641 page->next->prev = page->prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	642 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	643
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	644 page->slab = pages \| NGX_SLAB_PAGE_START;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	645 page->next = NULL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	646 page->prev = NGX_SLAB_PAGE;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	647
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	648 if (--pages == 0) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	649 return page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	650 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	651
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	652 for (p = page + 1; pages; pages--) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	653 p->slab = NGX_SLAB_PAGE_BUSY;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	654 p->next = NULL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	655 p->prev = NGX_SLAB_PAGE;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	656 p++;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	657 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	658
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	659 return page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	660 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	661 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	662
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	663 ngx_slab_error(pool, NGX_LOG_CRIT, "ngx_slab_alloc() failed: no memory");
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	664
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	665 return NULL;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	666 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	667
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	668
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	669 static void
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	670 ngx_slab_free_pages(ngx_slab_pool_t pool, ngx_slab_page_t page,
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	671 ngx_uint_t pages)
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	672 {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	673 ngx_slab_page_t *prev;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	674
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	675 page->slab = pages--;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	676
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	677 if (pages) {
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	678 ngx_memzero(&page[1], pages * sizeof(ngx_slab_page_t));
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	679 }
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	680
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	681 if (page->next) {
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	682 prev = (ngx_slab_page_t *) (page->prev & ~NGX_SLAB_PAGE_MASK);
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	683 prev->next = page->next;
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	684 page->next->prev = page->prev;
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	685 }
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	686
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	687 page->prev = (uintptr_t) &pool->free;
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	688 page->next = pool->free.next;
966 715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	689
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	690 page->next->prev = (uintptr_t) page;
715336c243e1 many bug fixes and ngx_slab_alloc_locked()/ngx_slab_free_locked() Igor Sysoev <igor@sysoev.ru> parents: 881 diff changeset	691
860 201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	692 pool->free.next = page;
201d017ea470 slab allocator in shared memory Igor Sysoev <igor@sysoev.ru> parents: 826 diff changeset	693 }
2611 2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	694
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	695
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	696 static void
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	697 ngx_slab_error(ngx_slab_pool_t pool, ngx_uint_t level, char text)
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	698 {
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	699 ngx_log_error(level, ngx_cycle->log, 0, "%s%s", text, pool->log_ctx);
2bce3f6416c6 improve ngx_slab_alloc() error logging Igor Sysoev <igor@sysoev.ru> parents: 2610 diff changeset	700 }

Mercurial > hg > nginx

annotate src/core/ngx_slab.c @ 4159:718f2154b813 stable-1.0