Skip to content

Link only the bootstrap modules into the build helper programs - #160537

Draft
itamaro wants to merge 2 commits into
python:mainfrom
itamaro:build-helpers-bootstrap-only
Draft

itamaro wants to merge 2 commits into
python:mainfrom
itamaro:build-helpers-bootstrap-only

Conversation

@itamaro

@itamaro itamaro commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Problem

Programs/_freeze_module and _bootstrap_python are linked with $(LIBRARY_OBJS_OMIT_FROZEN), i.e. with every static module from all Setup files (Setup.local, and all of Setup.stdlib under MODULE_BUILDTYPE=static). As a result a builtin module cannot depend on anything the helpers themselves produce:

  • a module whose source includes a header generated by $(FREEZE_MODULE) is a dependency cycle (make: Circular ... dependency dropped, then a compile failure);
  • a module referencing a symbol from Modules/getpath.o fails to link _freeze_module, which links Modules/getpath_noop.o instead.

CPython itself already carries a workaround for the second case: the _Py_Get_Getpath_CodeObject() stub in getpath_noop.c, there only because _testinternalcapi gets linked into _freeze_module in MODULE_BUILDTYPE=static builds. That option is not available to out-of-tree Setup.local modules.

The Windows build already does this right: PCbuild/_freeze_module.vcxproj compiles a fixed source list with PC/config_minimal.c.

Change

Link the helpers with the core objects plus the modules of Modules/Setup.bootstrap only. The final libpython/python link is unchanged.

  • Modules/makesetup: new -b Setup option marking a Setup file as the bootstrap modules. Their objects and libraries are substituted for _BOOTSTRAP_MODOBJS_ / _BOOTSTRAP_MODLIBS_, and a second table, Modules/config_bootstrap.c, is generated with only those modules. A bootstrap module overridden by an earlier rule (e.g. in Setup.local) keeps the overriding static module for the helpers too.
  • Makefile.pre.in: BOOTSTRAP_LIBRARY_OBJS (core + Modules/config_bootstrap.o + $(BOOTSTRAP_MODOBJS)) replaces LIBRARY_OBJS_OMIT_FROZEN in the _freeze_module and _bootstrap_python links. getbuildinfo gets a separate object for the helpers, since its prerequisites include $(MODOBJS).
  • configure.ac / configure: pass -b Modules/Setup.bootstrap and move config_bootstrap.c into Modules/.
  • Modules/getpath_noop.c: drop the now unnecessary stub.

Testing

Linux x86-64, gcc 13:

  • Default build, empty Setup.local: same 37 builtin / 75 shared as before; test_frozen test_embed test_sysconfig test_importlib test_zipimport test_site test_getpath test_capi pass; a second make is a no-op.
  • Default build with a Setup.local containing (a) a builtin whose source #includes a header generated by $(FREEZE_MODULE) and (b) a builtin calling _Py_Get_Getpath_CodeObject(): fails on main (circular dependency / missing symbol), builds and works on this branch. nm confirms neither module (nor _asyncio) is in the helpers, while posix is.
  • MODULE_BUILDTYPE=static: 105 builtins in python; _testinternalcapi and _ssl are no longer in _freeze_module (37 MB vs 63 MB for python), which exercises the removed stub. Same with a Setup.local override of a bootstrap module (errno): the override reaches the helpers. test_frozen test_embed test_getpath test_sysconfig test_capi test_import test_errno pass.

Notes

  • MODULE_DEPS_STATIC=Modules/config.c still recompiles the bootstrap objects whenever Setup.local changes, which relinks the helpers and regenerates all frozen headers — the same cascade as today. A follow-up could emit MODULE_DEPS_BOOTSTRAP=Modules/config_bootstrap.c for -b modules, provided config_bootstrap.c is only rewritten when its content changes.
  • Cross builds (iOS, Android, wasm) don't build the helpers and are unaffected; not run here.
  • configure was edited in sync with configure.ac; make regen-configure still to be run.

itamaro and others added 2 commits October 23, 2026 15:10
Programs/_freeze_module and _bootstrap_python were linked with
$(LIBRARY_OBJS_OMIT_FROZEN), i.e. every static module from all Setup
files (Setup.local, Setup.stdlib with MODULE_BUILDTYPE=static, ...).
That makes it impossible for a builtin module to depend on anything the
helpers produce: a frozen module header, Modules/getpath.o, or generated
sources. A Setup.local module including a header generated by
$(FREEZE_MODULE) is a dependency cycle ("Circular ... dependency
dropped"), and one referencing a getpath.o symbol fails to link, since
_freeze_module links Modules/getpath_noop.o instead. CPython itself
needed a workaround for the latter: a stub of
_Py_Get_Getpath_CodeObject() in getpath_noop.c for _testinternalcapi
in MODULE_BUILDTYPE=static builds.

Mirror what PCbuild/_freeze_module.vcxproj already does on Windows
(a fixed source list plus PC/config_minimal.c): link the helpers with
the core objects plus the modules of Modules/Setup.bootstrap only.

* makesetup: new "-b Setup" option marking a Setup file as the bootstrap
  modules. Their objects and libraries are substituted for
  _BOOTSTRAP_MODOBJS_ / _BOOTSTRAP_MODLIBS_, and a second table,
  config_bootstrap.c, is generated with only those modules. A bootstrap
  module overridden by an earlier rule (Setup.local) keeps the
  overriding static module for the helpers too.
* Makefile.pre.in: BOOTSTRAP_LIBRARY_OBJS (core + Modules/config_bootstrap.o
  + $(BOOTSTRAP_MODOBJS)) replaces LIBRARY_OBJS_OMIT_FROZEN in the
  _freeze_module and _bootstrap_python links. getbuildinfo gets a
  separate object for the helpers, as its prerequisites include
  $(MODOBJS).
* configure.ac / configure: pass -b Modules/Setup.bootstrap and move
  config_bootstrap.c into Modules/ (configure hand-edited in sync).
* Modules/getpath_noop.c: drop the now unnecessary stub.

The final libpython/python link is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD8kPnXaAMJZk2LvSHPXYA
Setup.local builtin modules that depend on build-helper output; see
Tools/build/bootstrap-split-repro/README.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YD8kPnXaAMJZk2LvSHPXYA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant