Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions Lib/test/test_json/test_recursion.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,22 +91,35 @@ def test_highly_nested_objects_decoding(self):
@support.skip_emscripten_stack_overflow()
@support.requires_resource('cpu')
def test_highly_nested_objects_encoding(self):
# See #12051
# See issue gh-56260

# With max_depth=2_000 on Linux, the Python implementation takes around
# 2 seconds to raise RecursionError, whereas the C implementation only
# takes around 160 ms.
max_depth = 2_000
l, d = [], {}
for x in range(500_000):
for x in range(max_depth * 10):
l, d = [l], {'k':d}
with self.assertRaises(RecursionError):
with support.infinite_recursion(5000):

# Don't use assertRaises(RecursionError): on FreeBSD 16 with a default
# stack size of 512 MiB, the C implementation doesn't raise
# RecursionError.
try:
with support.infinite_recursion(max_depth):
self.dumps(l)
with self.assertRaises(RecursionError):
with support.infinite_recursion(5000):
except RecursionError:
pass
try:
with support.infinite_recursion(max_depth):
self.dumps(d)
except RecursionError:
pass

@support.skip_if_huge_c_stack()
@support.skip_emscripten_stack_overflow()
@support.skip_wasi_stack_overflow()
def test_endless_recursion(self):
# See #12051
# See issue gh-56260
class EndlessJSONEncoder(self.json.JSONEncoder):
def default(self, o):
"""If check_circular is False, this will keep adding another list."""
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
On FreeBSD, call ``pthread_attr_get_np()`` to get the stack size (and guard
size) of the current thread. Patch by Victor Stinner.
16 changes: 13 additions & 3 deletions Python/ceval.c
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
#include "ceval.h"
#include "pycore_long.h"

#ifdef HAVE_PTHREAD_NP_H
# include <pthread_np.h> // pthread_attr_get_np()
#endif


int
Py_GetRecursionLimit(void)
{
Expand Down Expand Up @@ -147,13 +152,18 @@ hardware_stack_limits(uintptr_t *base, uintptr_t *top, uintptr_t sp)
// limits compared to the old stack size estimation. musl doesn't give
// us a way to detect it, so we have to assume any non-glibc,
// non-Android Linux is musl and mustn't rely on pthread_getattr_np.
# if defined(HAVE_PTHREAD_GETATTR_NP) && !defined(_AIX) && \
!defined(__NetBSD__) && \
(defined(__GLIBC__) || defined(__ANDROID__) || !defined(__linux__))
# if (defined(HAVE_PTHREAD_GETATTR_NP) || defined(_Py_HAVE_PTHREAD_ATTR_GET_NP)) \
&& !defined(_AIX) && !defined(__NetBSD__) \
&& (defined(__GLIBC__) || defined(__ANDROID__) || !defined(__linux__))
size_t stack_size, guard_size;
void *stack_addr;
pthread_attr_t attr;
#ifdef _Py_HAVE_PTHREAD_ATTR_GET_NP
int err = pthread_attr_init(&attr);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is already called here; is it safe to call here again?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pthread_attr_init() should be called every time you allocate a new pthread_attr_t attr on the stack memory: https://www.man7.org/linux/man-pages/man3/pthread_attr_init.3.html. Here the strange part is that the pthread_getattr_np() code path doesn't call pthread_attr_init(), but it calls pthread_attr_destroy().

At least, I can say that calling pthread_getattr_np() on a fresh pthread_attr_t attr variable without calling pthread_attr_init() fails on FreeBSD. So I had to add pthread_attr_init() call.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote a follow-up PR #159857 to also call pthread_attr_init() on Linux.

err |= pthread_attr_get_np(pthread_self(), &attr);
#else
int err = pthread_getattr_np(pthread_self(), &attr);
#endif
if (err == 0) {
err = pthread_attr_getguardsize(&attr, &guard_size);
err |= pthread_attr_getstack(&attr, &stack_addr, &stack_size);
Expand Down
1 change: 1 addition & 0 deletions Tools/build/check_capi_macros_ignored.txt
Original file line number Diff line number Diff line change
Expand Up @@ -433,6 +433,7 @@ HAVE_PTHREAD_GET_NAME_NP
HAVE_PTHREAD_H
HAVE_PTHREAD_INIT
HAVE_PTHREAD_KILL
HAVE_PTHREAD_NP_H
HAVE_PTHREAD_SETNAME_NP
HAVE_PTHREAD_SET_NAME_NP
HAVE_PTHREAD_SIGMASK
Expand Down
49 changes: 49 additions & 0 deletions configure

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 4 additions & 1 deletion configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -3195,7 +3195,8 @@ AC_CHECK_HEADERS([ \
alloca.h asm/types.h bluetooth.h conio.h direct.h dlfcn.h endian.h errno.h fcntl.h grp.h \
io.h langinfo.h libintl.h libutil.h linux/auxvec.h sys/auxv.h linux/fs.h linux/limits.h linux/memfd.h \
linux/netfilter_ipv4.h linux/random.h linux/soundcard.h linux/sched.h \
linux/tipc.h linux/wait.h netdb.h net/ethernet.h netinet/in.h netpacket/packet.h poll.h process.h pthread.h pty.h \
linux/tipc.h linux/wait.h netdb.h net/ethernet.h netinet/in.h netpacket/packet.h \
poll.h process.h pthread.h pthread_np.h pty.h \
sched.h setjmp.h shadow.h signal.h spawn.h sys/audioio.h sys/bsdtty.h sys/devpoll.h \
sys/endian.h sys/epoll.h sys/event.h sys/eventfd.h sys/file.h sys/ioctl.h sys/kern_control.h \
sys/loadavg.h sys/lock.h sys/memfd.h sys/mkdev.h sys/mman.h sys/modem.h sys/param.h sys/pidfd.h sys/poll.h \
Expand Down Expand Up @@ -5586,6 +5587,8 @@ AC_CHECK_FUNCS([ \

PY_CHECK_FUNC_PRIVATE([freadlink], [@%:@include <unistd.h>])
PY_CHECK_FUNC_PRIVATE([strerror_r], [@%:@include <string.h>])
# FreeBSD pthread_attr_get_np()
PY_CHECK_FUNC_PRIVATE([pthread_attr_get_np], [@%:@include <pthread_np.h>])


# os.statx uses Linux's statx function. AIX also has a function named statx,
Expand Down
6 changes: 6 additions & 0 deletions pyconfig.h.in
Original file line number Diff line number Diff line change
Expand Up @@ -1138,6 +1138,9 @@
/* Define to 1 if you have the 'pthread_kill' function. */
#undef HAVE_PTHREAD_KILL

/* Define to 1 if you have the <pthread_np.h> header file. */
#undef HAVE_PTHREAD_NP_H

/* Define to 1 if you have the 'pthread_setname_np' function. */
#undef HAVE_PTHREAD_SETNAME_NP

Expand Down Expand Up @@ -2214,6 +2217,9 @@
/* Define if you have the 'PR_SET_VMA_ANON_NAME' constant. */
#undef _Py_HAVE_PR_SET_VMA_ANON_NAME

/* Define if you have the 'pthread_attr_get_np' function. */
#undef _Py_HAVE_PTHREAD_ATTR_GET_NP

/* Define if you have the 'sinpi' function. */
#undef _Py_HAVE_SINPI

Expand Down
Loading