From: Lee Schermerhorn Subject: slab - handle memoryless nodes V2a References: bnc#436025, bnc#570492 Patch-mainline: not yet The slab cache, since [apparently] 2.6.21, does not handle memoryless nodes well. Specifically, the "fast path"-- ____cache_alloc()--will never succeed, but will be called twice: once speculatively [expected to succeed] and once in the fallback path. This adds significant overhead to all kmem cache allocations, incurring a significant regression relative to earlier kernels [from before slab.c was reorganized]. This patch addresses the regression by modifying slab.c to treat the first fallback node in a memoryless node's general zonelist as the "slab local node" -- i.e., the local node for the purpose of slab allocations. This is, in fact, the node from which all "local" allocations for cpus attached to a memoryless node will be satisfied. The new function numa_slab_nid(gfp_t) replaces all calls to numa_node_id() in slab.c. numa_slab_id() will simply return numa_node_id() for nodes with memory, but will return the first node in the local node's zonelist selected by the gfp flags. Effects of the patch: We first noticed the effects of the slab reorganization running the AIM benchmark on a distro based on 2.6.27. The effect is even more pronounced in the hackbench results. The platform in an HP rx8640 numa platform, configured with "0% Cell Local Memory". In this configuration, all memory appears in a "pseudo-node"--an artifact of the firmware--and is interleaved across all the physical nodes' memory on a cacheline granularity. All cpus are presented as attached to memoryless nodes. Here are the results of running hackbench at various load levels with and without the patch on the same platform configured for 0% CLM and "100% CLM". Command: hackbench N process 100, for N = 10..100 by 10 100% CLM 0% CLM Tasks no with no with patch patch %diff patch patch %diff 400 0.246 0.281 14.23% 2.962 0.410 -86.16% 800 0.418 0.421 0.72% 6.224 0.793 -87.26% 1200 0.548 0.532 -2.92% 9.058 1.090 -87.97% 1600 0.654 0.716 9.48% 12.473 1.562 -87.48% 2000 0.871 0.859 -1.38% 15.484 1.889 -87.80% 2400 0.994 1.043 4.93% 18.689 2.309 -87.65% 2800 1.196 1.195 -0.08% 22.069 2.606 -88.19% 3200 1.322 1.344 1.66% 25.642 2.988 -88.35% 3600 1.474 1.519 3.05% 28.003 3.418 -87.79% 4000 1.682 1.750 4.04% 30.887 3.858 -87.51% In the 100% CLM case, the regression does not appear, because all nodes have local memory. Note that the patch has >10% overhead on the first run, but then varies widely from run to run [more below]. For the 0%CLM configuration, the patch reduced the run time by 86-88%. The following runs extend the number of hackbench tasks using: hackbench N process 100, for N = 100 to 400 by 20 We didn't run the 0%CLM/no-patch runs as they were taking too long for our liking. We wanted to see how the patched kernel performed as we extended the range. 100% CLM 0% CLM Tasks no with no with patch patch %diff patch patch %diff 4800 1.879 2.117 12.67% not-run 4.458 5600 2.100 2.352 12.00% not-run 5.207 6400 2.532 2.447 -3.36% not-run 5.669 8000 2.799 2.792 -0.25% not-run 6.651 8000 3.244 3.030 -6.60% not-run 7.366 8800 3.282 3.550 8.17% not-run 8.169 9600 3.595 3.738 3.98% not-run 8.582 10400 3.811 4.045 6.14% not-run 9.705 11200 4.090 4.162 1.76% not-run 9.760 12000 4.408 4.419 0.25% not-run 10.141 12800 4.665 4.787 2.62% not-run 11.628 13600 5.095 5.069 -0.51% not-run 11.735 14400 5.347 5.464 2.19% not-run 12.621 15200 5.620 5.831 3.75% not-run 13.452 16000 5.870 6.161 4.96% not-run 14.069 The 0% CLM configuration with the patch performs worse than the 100% CLM configuration. In the 0% CLM case we had 64 ia64 cores beating on a single zone in the interleaved memory-only pseudo-node. In the 100% CLM case, we have 16 cores allocating memory locally to each of 4 nodes, demonstating the difference between [pseudo-]SMP and NUMA behavior. Note, again, that the first run[s] have higher % difference between the patched and unpatched kernels for the 100% CLM config, and then vary quite a bit run to run. To get a feel for the average overhead, we ran 40 runs at the 16000 task load point with more interations to increase the runtime per run: hackbench 400 process 200 These were run on the 100% CLM configuration, as this best represents most NUMA platforms: No patch with Patch %diff Average of 40: 9.796 9.857 0.623 Signed-off-by: Lee Schermerhorn Acked-by: Nick Piggin mm/slab.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 78 insertions(+), 13 deletions(-) --- a/mm/slab.c +++ b/mm/slab.c @@ -305,7 +305,7 @@ struct kmem_list3 { struct array_cache **alien; /* on other nodes */ unsigned long next_reap; /* updated without locking */ int free_touched; /* updated without locking */ -}; +} __attribute__((aligned(sizeof(long)))); /* * Need this for bootstrapping a per node allocator. @@ -968,6 +968,11 @@ static int transfer_objects(struct array #define drain_alien_cache(cachep, alien) do { } while (0) #define reap_alien(cachep, l3) do { } while (0) +static inline int numa_slab_nid(struct kmem_cache *cachep, gfp_t flags) +{ + return 0; +} + static inline struct array_cache **alloc_alien_cache(int node, int limit, gfp_t gfp) { return (struct array_cache **)BAD_ALIEN_MAGIC; @@ -999,6 +1004,64 @@ static inline void *____cache_alloc_node static void *____cache_alloc_node(struct kmem_cache *, gfp_t, int); static void *alternate_node_alloc(struct kmem_cache *, gfp_t); +/* + * slow path for numa_slab_nid(), below + */ +static noinline int __numa_slab_nid(struct kmem_cache *cachep, + int node, gfp_t flags) +{ + struct zonelist *zonelist; + struct zone *zone; + enum zone_type highest_zoneidx = gfp_zone(flags); + + if (likely(node_state(node, N_NORMAL_MEMORY))) + return node; + + /* + * memoryless node: consult its zonelist. + * Cache the fallback node, if cache pointer provided. + */ + zonelist = &NODE_DATA(node)->node_zonelists[0]; + (void)first_zones_zonelist(zonelist, highest_zoneidx, + NULL, + &zone); + if (cachep) + cachep->nodelists[node] = + (struct kmem_list3 *)((unsigned long)zone->node << 1 | 1); + return zone->node; +} + +/* + * "Local" node for slab is first node in zonelist with memory. + * For nodes with memory this will be the actual local node. + * + * Use nodelist[numa_node_id()] to cache the fallback node for + * memoryless nodes. We'll be loading that member soon anyway, + * or already have, when called for cache refill, ... Use low + * bit of "pointer" as flag for "memoryless_node", indicating + * that the fallback nodes is stored here [<<1]. + */ +#define memoryless_node(L3L) ((L3L) & 1) +static inline int numa_slab_nid(struct kmem_cache *cachep, gfp_t flags) +{ + int node = numa_node_id(); + + if (likely(cachep)){ + unsigned long l3l = (unsigned long)cachep->nodelists[node]; + + if (likely(l3l)) { + if (unlikely(memoryless_node(l3l))) + node = (int)(l3l >> 1); + return node; + } + } + + /* + * !cachep || !l3l - the slow path + */ + return __numa_slab_nid(cachep, node, flags); +} + static struct array_cache **alloc_alien_cache(int node, int limit, gfp_t gfp) { struct array_cache **ac_ptr; @@ -1098,7 +1161,7 @@ static inline int cache_free_alien(struc struct array_cache *alien = NULL; int node; - node = numa_node_id(); + node = numa_slab_nid(cachep, GFP_KERNEL); /* * Make sure we are not freeing a object from another node to the array @@ -1443,7 +1506,7 @@ void __init kmem_cache_init(void) * 6) Resize the head arrays of the kmalloc caches to their final sizes. */ - node = numa_node_id(); + node = numa_slab_nid(NULL, GFP_KERNEL); /* 1) create the cache_cache */ INIT_LIST_HEAD(&cache_chain); @@ -2079,7 +2142,7 @@ static int __init_refok setup_cpu_cache( } } } - cachep->nodelists[numa_node_id()]->next_reap = + cachep->nodelists[numa_slab_nid(cachep, GFP_KERNEL)]->next_reap = jiffies + REAPTIMEOUT_LIST3 + ((unsigned long)cachep) % REAPTIMEOUT_LIST3; @@ -2411,7 +2474,7 @@ static void check_spinlock_acquired(stru { #ifdef CONFIG_SMP check_irq_off(); - assert_spin_locked(&cachep->nodelists[numa_node_id()]->list_lock); + assert_spin_locked(&cachep->nodelists[numa_slab_nid(cachep, GFP_KERNEL)]->list_lock); #endif } @@ -2438,7 +2501,7 @@ static void do_drain(void *arg) { struct kmem_cache *cachep = arg; struct array_cache *ac; - int node = numa_node_id(); + int node = numa_slab_nid(cachep, GFP_KERNEL); check_irq_off(); ac = cpu_cache_get(cachep); @@ -2975,7 +3038,7 @@ static void *cache_alloc_refill(struct k retry: check_irq_off(); - node = numa_node_id(); + node = numa_slab_nid(cachep, flags); if (unlikely(must_refill)) goto force_grow; ac = cpu_cache_get(cachep); @@ -3185,7 +3248,7 @@ static void *alternate_node_alloc(struct if (in_interrupt() || (flags & __GFP_THISNODE)) return NULL; - nid_alloc = nid_here = numa_node_id(); + nid_alloc = nid_here = numa_slab_nid(cachep, flags); if (cpuset_do_slab_mem_spread() && (cachep->flags & SLAB_MEM_SPREAD)) nid_alloc = cpuset_mem_spread_node(); else if (current->mempolicy) @@ -3360,6 +3423,7 @@ __cache_alloc_node(struct kmem_cache *ca { unsigned long save_flags; void *ptr; + int slab_node = numa_slab_nid(cachep, flags); flags &= gfp_allowed_mask; @@ -3372,7 +3436,7 @@ __cache_alloc_node(struct kmem_cache *ca local_irq_save(save_flags); if (nodeid == -1) - nodeid = numa_node_id(); + nodeid = slab_node; if (unlikely(!cachep->nodelists[nodeid])) { /* Node not bootstrapped yet */ @@ -3380,7 +3444,7 @@ __cache_alloc_node(struct kmem_cache *ca goto out; } - if (nodeid == numa_node_id()) { + if (nodeid == slab_node) { /* * Use the locally cached objects if possible. * However ____cache_alloc does not allow fallback @@ -3425,7 +3489,8 @@ __do_cache_alloc(struct kmem_cache *cach * ____cache_alloc_node() knows how to locate memory on other nodes */ if (!objp) - objp = ____cache_alloc_node(cache, flags, numa_node_id()); + objp = ____cache_alloc_node(cache, flags, + numa_slab_nid(cache, flags)); out: return objp; @@ -3522,7 +3587,7 @@ static void cache_flusharray(struct kmem { int batchcount; struct kmem_list3 *l3; - int node = numa_node_id(); + int node = numa_slab_nid(cachep, GFP_KERNEL); batchcount = ac->batchcount; #if DEBUG @@ -4172,7 +4237,7 @@ static void cache_reap(struct work_struc { struct kmem_cache *searchp; struct kmem_list3 *l3; - int node = numa_node_id(); + int node = numa_slab_nid(NULL, GFP_KERNEL); struct delayed_work *work = to_delayed_work(w); if (!mutex_trylock(&cache_chain_mutex))