PyPy v7.3.6: release of python 2.7, 3.7, and 3.8-beta

The PyPy team is proud to release version 7.3.6 of PyPy, which includes three different interpreters:

  • PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
  • PyPy3.7, which is an interpreter supporting the syntax and the features of Python 3.7, including the stdlib for CPython 3.7.12.
  • PyPy3.8, which is an interpreter supporting the syntax and the features of Python 3.8, including the stdlib for CPython 3.8.12. Since this is our first release of the interpreter, we relate to this as “beta” quality. We welcome testing of this version, if you discover incompatibilites, please report them so we can gain confidence in the version.

The interpreters are based on much the same codebase, thus the multiple release. This is a micro release, all APIs are compatible with the other 7.3 releases. Highlights of the release, since the release of 7.3.5 in May 2021, include:

  • We have merged a backend for HPy, the better C-API interface. The backend implements version 0.0.3.
  • Translation of PyPy into a binary, known to be slow, is now about 40% faster. On a modern machine, PyPy3.8 can translate in about 20 minutes.
  • PyPy Windows 64 is now available on conda-forge, along with over 600 commonly used binary packages. This new offering joins the more than 1000 conda packages for PyPy on Linux and macOS. Many thanks to the conda-forge maintainers for pushing this forward over the past 18 months.
  • Speed improvements were made to io, sum, _ssl and more. These were done in response to user feedback.
  • The 3.8 version of the release contains a beta-quality improvement to the JIT to better support compiling huge Python functions by breaking them up into smaller pieces.
  • The release of Python3.8 required a concerted effort. We were greatly helped by @isidentical (Batuhan Taskaya) and other new contributors.
  • The 3.8 package now uses the same layout as CPython, and many of the PyPy-specific changes to sysconfig, distutils.sysconfig, and distutils.commands.install.py have been removed. The stdlib now is located in <base>/lib/pypy3.8 on posix systems, and in <base>/Lib on Windows. The include files on windows remain the same, on posix they are in <base>/include/pypy3.8. Note we still use the pypy prefix to prevent mixing the files with CPython (which uses python.

We recommend updating. You can find links to download the v7.3.6 releases here:

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our blog via a pull request to https://github.com/pypy/pypy.org

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on PyPy, or general help with making RPython’s JIT even better. Since the previous release, we have accepted contributions from 7 new contributors, thanks for pitching in, and welcome to the project!

If you are a python library maintainer and use C-extensions, please consider making a CFFI / cppyy version of your library that would be performant on PyPy. In any case both cibuildwheel and the multibuild system support building wheels for PyPy.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython 2.7, 3.7, and soon 3.8. It’s fast (PyPy and CPython 3.7.4 performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This PyPy release supports:

  • x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 64 bits, OpenBSD, FreeBSD)
  • big- and little-endian variants of PPC64 running Linux,
  • s390x running Linux
  • 64-bit ARM machines running Linux.

PyPy does support Windows 32-bit and ARM 32 bit processors, but does not release binaries. Please reach out to us if you wish to sponsor releases for those platforms.

Changelog

Bugfixes shared across versions

  • Backport fix for bpo 44022 in httplib
  • Make dotviewer Python3 compatible and add some features (like rudimentary record support).
  • Fix error reporting when the error position is the last character of a JSON bytestring (issue 3514)
  • Set non-volatile xmm registers in the JIT for windows 64-bit calling conventions. Fixes a bug where the JIT was not restoring registers when returning from a call
  • Support multiple tags in hg_archive, fixes platform._sys_version() which was reporting the first tag (i.e. rc1) when it should have reported the last tag (i.e. final)
  • Fix position bugs in the astcompiler stemming from the fact that SyntaxErrors use 1-based column offsets
  • Fix an obscure corner of %-formatting when a tuple is passed in that triggered infinite recursion in the __repr__ of a named_tuple with an __iter__ (issue 3555)
  • Fixed a PowerPC JIT bug that caused a stack overflow (issue 3463)

Speedups and enhancements shared across versions

  • Speed up RPython typing
  • Speed up RPython graph cycle finding by removing leaves before checking for cycles
  • Speed up C compilation on GCC by using the pre-compiled header feature
  • When switching from an unwrapped list strategy to the ObjectListStrategy, try to cache wrapped elements. This prevents the memory blowup on code like l = [0] * N; l[0] = "abc" (issue 2881).
  • Specialize builtin sum for lists/tuples (issue 3492).
  • Update to cffi 1.14.6
  • Use libffi-7.dll on windows instead of an old vendored version of libffi
  • Fix for a corner case missing an optimization: interpindirect2app() on a function with signature 'self, space, __args__' was missing the optimization for that signature, which works with interp2app(). It’s used in _hpy_universal.
  • Add an option to the packaging script to force non-portable packaging (issue 3538)
  • Switch to “powersort” merging strategy by Munro and Wild instead the timsort algorithm (bpo 34561)
  • Check env keys for '=' when calling os.execve
  • Provide a PyPy-only scoped way to malloc buffers in cffi and use it in ssl.read

C-API (cpyext) and C-extensions

We are no longer backporting changes to the cpyext compatibility layer to PyPy2.7.

Python 3.7+ bugfixes

  • Fix MemoryError on zip.read in shutil._unpack_zipfile for large files bpo 43650
  • Fix some issues around the obj field of memoryview, and add missing toreadonly.
  • Fix re.sub() with no match and with unusual types of arguments (issue 3515)
  • Fix _socket.sethostname() failure when passed bytes
  • Switch sys.implementation.version and sys.implementation.hexversion to pypy_version_info (i.e. (7, 3. 6) not (3, 7. 10)) (issue 3129)
  • Fix pypy -Wonce (issue 3411)

Python 3.7+ speedups and enhancements

  • Speed up cached imports by re-implementing (a subset of) bpo 22557. This brings PyPy3.7 very close to the speed of PyPy2 (issue 3431)
  • Ignore finalizers on built-in io classes if we know the stream is closed. Also find some other optimizations around io operations.
  • Add more fields to sysconfig.get_config_var via _sysconfigdata (issue 3483)
  • Add a sys.implementation._multiarch field like CPython on linux and darwin
  • Add a lib_pypy\_sysconfigdata__*.py file like CPython on linux, darwin during packaging via sysconfig._generate_posix_vars() (issue 3483)
  • Slightly adapt the packaging and cffi-module build scripts for compatibility with conda-forge.
  • Create pypy.exe, pypyw.exe, python.exe, pythonw.exe when packaging for windows
  • Speed up _ssl error processing by moving the class out of _PySSL_errno and creating a fast-path for instantiation (issue 3490)
  • Support HPy 0.0.2
  • Use CPython list of consts in os.{confstr,pathconf,sysconf}_names (issue 3502)
  • Add _winapi.GetFileType and FILE_TYPE_* values (issue 3531)
  • Allow ctypes.POINTER() to cast ctypes.array` (issue 3546)
  • Update the stdlib to v3.7.12

Python 3.7 C-API

  • Add PEP 495 c-api TimeAndFold datetime constructors (issue 2987)
  • Allow NULL in PyErr_WriteUnraisable (issue 3353)
  • Support *TimeZone* functions in datetime
  • Add slot functions so int(x) and float(x) work properly where x is a c-extension class
  • When creating a PyUnicodeObject, use the compact form to store the data directly on the object and not via an additional buffer. This is used in pythran via _PyUnicode_COMPACT_DATA even though it is a “private” interface.
  • Add PyGILState_Check, PyContextVar_New, PyContextVar_Get, PyContextVar_Set
  • Add PyExc_WindowsError (issue 3472)
  • Add frame.f_back, assuming the user is aware of the dangers of examinig the stack
  • Fix typo in import.h