Mimallocv3

[DinoV]

No description provided.

[python-cla-bot]

The following commit authors need to sign the Contributor License Agreement:

• daanl@outlook.com

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #35256703 | 📁 Comparing e19e2dd against main (415ebff)

  🔍 Preview build  

11 files changed · ± 11 modified

± Modified

• license.html
• py-modindex.html
• c-api/extension-modules.html
• c-api/file.html
• c-api/import.html
• c-api/intro.html
• library/bdb.html
• library/ctypes.html
• library/gc.html
• whatsnew/3.16.html
• and 1 more...

[StanFromIreland]

@daanx can you please sign the CLA?

[daanx]

@daanx can you please sign the CLA?

Ah yes, but not really for the current PR as that includes the full mimalloc v3 vendored sources and that is a separate project under the MIT license.

I can of course sign off for the changes I made on the CPython side. The current PR contains both, maybe we can split it? I think commit cedd47b is the CPython part only, while the other commits are just importing mimalloc v3 as a vendored library.

[daanx]

Using MI_WIN_INIT_USE_FLS is not really encouraged for mimalloc. If we can use the default that would be best.

[DinoV]

Using MI_WIN_INIT_USE_FLS is not really encouraged for mimalloc. If we can use the default that would be best.

There's a crash on the Windows builds w/o it currently (and it seems to show up basically at startup but the freeze programs make it through it). I was planning on digging into it more later but this is what Claude came up as a fix - I just don't have immediate access to a Windows machine to see where the actual crash is coming from. But that's why it's there for now.

Right now I'm just trying to get things to green CI and then I'll clean up the final result.

[DinoV]

@corona10 Hey here it is!

[corona10]

For TSAN fail see: 2599016

IIRC, it was already happened at

cpython/Objects/mimalloc/os.c

Lines 245 to 268 in 34fe59b

	else {
	// if not aligned, free it, overallocate, and unmap around it
	// NOTE(sgross): this warning causes issues in Python tests
	// _mi_warning_message("unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x%zx bytes, address: %p, alignment: 0x%zx, commit: %d)\n", size, p, alignment, commit);
	mi_os_prim_free(p, size, commit, stats);
	if (size >= (SIZE_MAX - alignment)) return NULL; // overflow
	const size_t over_size = size + alignment;
	if (mi_os_mem_config.must_free_whole) { // win32 virtualAlloc cannot free parts of an allocate block
	// over-allocate uncommitted (virtual) memory
	p = mi_os_prim_alloc(over_size, 1 /*alignment*/, false /* commit? */, false /* allow_large */, is_large, is_zero, stats);
	if (p == NULL) return NULL;
	// set p to the aligned part in the full region
	// note: this is dangerous on Windows as VirtualFree needs the actual base pointer
	// this is handled though by having the `base` field in the memid's
	*base = p; // remember the base
	p = mi_align_up_ptr(p, alignment);
	// explicitly commit only the aligned part
	if (commit) {
	_mi_os_commit(p, size, NULL, stats);
	}
	}

[corona10]

About guardrail for memory pressure

any changes expected to increase memory above 15% (with a hard limit of 20%) should be discussed with the SC in advance.