PyPy v7.3.18: release of python 2.7, 3.10 and 3.11 beta, released 2025-02-06

The PyPy team is proud to release version 7.3.18 of PyPy.

This release includes a python 3.11 interpreter. We are labelling it “beta” because it is the first one. In the next release we will drop 3.10 and remove the “beta” label. There are a particularly large set of bugfixes in this release thanks to @devdanzin using fusil on the 3.10 builds, originally written by Victor Stinner. Other significant changes:

  • We have updated libffi shipped in our portable builds. We also now statically link to libffi where possible which reduces the number of shared object dependencies.
  • We have added code to be able to show the native function names when profiling with VMProf. So far only Linux supports this feature.
  • We have added a PEP 768-inspired remote debugging facility. See Remote Debugging.
  • The HPy backend has been updated to latest HPy HEAD

The release includes three different interpreters:

  • PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
  • PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.19.
  • PyPy3.11, which is an interpreter supporting the syntax and the features of Python 3.11, including the stdlib for CPython 3.10.11.

The interpreters are based on much the same codebase, thus the triple release. This is a micro release, all APIs are compatible with the other 7.3 releases. It follows after 7.3.17 release on August 28, 2024.

We recommend updating. You can find links to download the releases here:

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our blog via a pull request to https://github.com/pypy/pypy.org

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: bug fixes, PyPy and RPython documentation improvements, or general help with making RPython’s JIT even better.

If you are a python library maintainer and use C-extensions, please consider making a HPy / CFFI / cppyy version of your library that would be performant on PyPy. In any case, both cibuildwheel and the multibuild system support building wheels for PyPy.

VMProf Native Symbol Names

When running VMProf profiling with native profiling enabled, PyPy did so far not produce function names for C functions. The output looked like this:

pypy -m vmprof ~/projects/gitpypy/lib-python/2.7/test/pystone.py
Pystone(1.1) time for 50000 passes = 0.0109887
This machine benchmarks at 4.55011e+06 pystones/second
 vmprof output:
 %:      name:                location:
 100.0%  entry_point          <builtin>/app_main.py:874
 100.0%  run_command_line     <builtin>/app_main.py:601
 100.0%  run_toplevel         <builtin>/app_main.py:93
 100.0%  _run_module_as_main  /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:150
 100.0%  _run_code            /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:62
 100.0%  <module>             /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/site-packages/vmprof/__main__.py:1
 100.0%  main                 /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/site-packages/vmprof/__main__.py:30
 100.0%  run_path             /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:238
 100.0%  _run_module_code     /home/user/bin/pypy-c-jit-170203-99a72243b541-linux64/lib-python/2.7/runpy.py:75
 100.0%  <module>             /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:3
 100.0%  main                 /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:60
 100.0%  pystones             /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:67
 100.0%  Proc0                /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:79
 76.9%   <unknown code>
 69.2%   <unknown code>
 53.8%   <unknown code>
 53.8%   <unknown code>
 46.2%   <unknown code>
 46.2%   <unknown code>
 38.5%   <unknown code>
 38.5%   Proc8                /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:212
 30.8%   <unknown code>
 ...

We can now symbolify these C functions and give function names and which shared library they come from, at least on Linux:

Pystone(1.1) time for 50000 passes = 0.218967
This machine benchmarks at 228345 pystones/second
 vmprof output:
 %:      name:                                           location:
 100.0%  entry_point                                     <builtin>/app_main.py:889
 100.0%  run_command_line                                <builtin>/app_main.py:616
 100.0%  run_toplevel                                    <builtin>/app_main.py:95
 100.0%  _run_module_as_main                             /home/user/projects/gitpypy/lib-python/2.7/runpy.py:150
 100.0%  _run_code                                       /home/user/projects/gitpypy/lib-python/2.7/runpy.py:62
 100.0%  <module>                                        /home/user/projects/gitpypy/site-packages/vmprof/__main__.py:1
 100.0%  main                                            /home/user/projects/gitpypy/site-packages/vmprof/__main__.py:30
 100.0%  run_module                                      /home/user/projects/gitpypy/lib-python/2.7/runpy.py:179
 100.0%  _run_module_code                                /home/user/projects/gitpypy/lib-python/2.7/runpy.py:75
 100.0%  <module>                                        /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:3
 100.0%  main                                            /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:60
 100.0%  pystones                                        /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:67
 100.0%  Proc0                                           /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:79
 95.5%   n:pypy_g_execute_frame:0:pypy-c
 91.4%   n:pypy_g_PyFrame_dispatch:0:pypy-c
 63.8%   n:pypy_g_PyFrame_dispatch_bytecode:0:pypy-c
 49.8%   Proc1                                           /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:137
 17.6%   copy                                            /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:53
 13.6%   n:pypy_g_PyFrame_CALL_FUNCTION:0:pypy-c
 10.4%   Proc8                                           /home/user/projects/gitpypy/lib-python/2.7/test/pystone.py:212
 8.6%    n:pypy_g_STORE_ATTR_slowpath:0:pypy-c

This becomes even more useful when using the VMProf Firefox converter, which uses the Firefox Profiler Web UI to visualize profiling output:

_images/2025-vmprof-firefox.png

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It’s fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

  • x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
  • 64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

PyPy supports Windows 32-bit, Linux PPC64 big- and little-endian, Linux ARM 32 bit, RISC-V RV64IMAFD Linux, and s390x Linux but does not release binaries. Please reach out to us if you wish to sponsor binary releases for those platforms. Downstream packagers provide binary builds for debian, Fedora, conda, OpenBSD, FreeBSD, Gentoo, and more.

Changelog

For all versions

  • Update cffi backend to 1.18.0-dev
  • Refactor use of Python2 unicode in rpython to use only utf-8. This affects windows calls to rposix routines. Now all the system calls on windows should use the FunctionW variants instead of the FunctionA ones.
  • Update to vmprof-0.4.17
  • Update vmprof native profiling to show more native symbols
  • Implement PEP 768-inspired remote debugging facility. See Remote Debugging.
  • Add many more int optimization rules.
  • Bump macosx-version-min to 10.13 on 2.7 to match 3.10, 3.11

Bugfixes

  • Make sure that tracing tail-recursive infinite recursion ends (#5021)
  • Revive tools/gcdump.py which uses PYPYLOG
  • Fix socket.socket.sendto for AF_PACKET protocol (#5024)
  • Fix inf/nan formatting with thousands separator (#5018)
  • Fixup int/long confusion on 32-bit builds
  • Fix the gc.get_stats output to not show incorrect numbers (#5005)
  • Use simple interactive console if stdin is closed (#2981)
  • Use HOMEBREW_CELLAR to find tcl library on macOS (#5096)
  • Don’t segfault in unicodedb when looking up invalid codepoints (#5113)
  • Fix segfault in pyexpat (#5112)
  • Guard against list mutation in the list repr (#5117)
  • Check input for divide-by-zero in __pypy__.intops (#5129)
  • Check input for valid c in mulmod(a, b, c), mod, and floordiv (#5128)

Speedups and enhancements

  • Make the opencoder encoding support varsized ints. This shrinks the memory usage of the JIT and is supposed to support really long trace limits without recompiling pypy
  • Implement a try_cast_erased function in RPython
  • Copy CPython’s threading implementation for windows
  • Only use largefile glibc interfaces on 32-bit build (#5071)
  • Add a DSL for integer optimizations, use it to create some optimizations, see the blogpost
  • Optimize overflowing int*int multiplication that produces a long result
  • Resolve names of native functions in vmprof (#5114)
  • Make internal streamio replace_crlf_with_lf do only one copy, not two since we often read programmatically generated python code.
  • Avoid keeping refs on the frame stack when calling functions
  • Fix reverse JIT unrolling, which was done too eagerly
  • Write abs(int) in a branch-free way, producing fewer JIT traces
  • On windows, use wchar_t for main(..., argv). Solves long standing issue around calling pypy.exe <unicode-named-file>.py
  • Fix cffi backend for struct-in-a-struct (#python-cffi/cffi#147)
  • Make str.format somewhat more jit friendly
  • Use BCryptGenRandom on windows in rurandom (#5039)
  • If mode is 0700 in mkdir on windows, restrict the directory to the current user
  • Add support for unicode version 14 to RPython

Python 3.10

Bugfixes

  • Sync Python.h with upstream, add import.h (#5013)
  • Prefer static sysconfigdata if it exists and do not create static sysconfigdata on portable builds (#5015)
  • Fix python2-isms in complex, in both str and format
  • Do not segfault in reversed.__setstate__ (#5029)
  • Fix weird edge case where a index of a memoryview releases the underlying buffer
  • Fix _ssl.read when get_shutdown is true, should no longer error
  • Always initialize locale by calling setlocale(LC_CTYPE, '')
  • Fix when metaclass __new__ has extra args
  • Fix venv when src is a source build
  • Also create python.exe and python3.exe when creating a venv in a source build on windows
  • Sync 'user_base' between site.py and sysconfig.py on windows (#5073)
  • Allow unterminated string error to propagate in the tokenizer (#5076)
  • Fix subtle problem with Py_SIZE(PyListObject) since it assumes PyListObject is a PyVarObject.
  • Fix race in Thread.join() (#5080) and threadpool stopping (#4994)
  • Fix logic in packaging tklib for darwin (#5082)
  • Fix an infinite loop in the jump threading optimization in the bytecode compiler (#5090)
  • Make __doc__ a proper descr on methods
  • Limit _string.formatter* functions to reject bytes (#5111)
  • Remove internal calls of utf8(bytes), fix error handler, add test (#5110, #5111)
  • Make linecache.checkcache more resilient in the presence of __del__ and multithreading (#5109)
  • Remove extraneous exports from _dbm.__all__ (#5115)
  • Add missing _ensure_initialized in _curses.putp (#5116)
  • Check self.ssl for pathological use of _ssl (#5124)
  • Use os.fsencode for dbm.*.open (#5115)
  • Check for NULL ssl certificate (#5120)
  • Check before calling _dealloc_warn_w (#5123)
  • Fix _curses tests and hide _mk_acs (#5122)
  • Use pkg-config for cffi _tkinter, fix tk/tcl_path for portable builds (#5064, #5096)
  • Test, fix infinite recursion when creating a pyobj from w_obj when the pyobj is a list subtype that overrides __len__ in c
  • Be more careful in the order of StringIO.__init__ since decoding can fail (#5126)
  • Refactor hashlib _keccak_init to be a regular class method (#5127)
  • Fix list.pop and list.insert to use __index__
  • Check code validity in _pickle_support.builtin_code (#5130)
  • Check for bad result when calling nl_langinfo (#5132)
  • Backport CPython fix to not write incomplete pyc files (python/cpython#126606)
  • Do not initialize values if stringio newline is wrong (#5140)
  • Initialize buffer view readonly flag properly (#5136)
  • Only increment pos in PyDict_Next after checking len (#5142)
  • In builtin hex(x), oct(x), bin(x), use int.__format__ not x.__format__
  • Invalidate the method cache in the presence of overridden mros (#5149)
  • Fix new failure in lib-python/3/test/test_descr when mro() sets __bases__
  • Avoid segfault when creating memoryview of ctypes array with 0 shape (#5156)
  • Fix signature of sendfunc
  • Backport the CPython fixes to pyrepl to PyPy (#4990)
  • Fix win32console._write return value (:issue: 5139)
  • Backport changes to site.py from CPython3.13 to get pyrepl’s PYTHON_HISTORY working

Speedups and enhancements

  • Move _remove_dead_weakref to the _weakref module (#5068)
  • Make unicodedata.normalize O(1) for ascii strings (#5052)
  • Add PyContextVar_Reset, Py_FatalError (#5081) (#5086)
  • Make Py_FatalError a macro that adds the current function name, like CPython
  • Many error message tweaks for test compliance with CPython
  • Make unmarshaling use unrolling_iterable instead of a function ptr table, which should speed it up slightly.
  • Add _ssl.keylog_filename which is useful for debugging ssl problems (#5141)
  • Allocate less when using PyErr_NoMemory to raise an error rather than segfault
  • Add docstrings to tupleobject
  • Add audit events to syslog and _sqlite3