PyPy v7.3.4: release of 2.7 and 3.7

The PyPy team is proud to release the version 7.3.4 of PyPy, which includes two different interpreters:

  • PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
  • PyPy3.7, which is an interpreter supporting the syntax and the features of Python 3.7, including the stdlib for CPython 3.7.10. We no longer refer to this as beta-quality as the last incompatibilities with CPython (in the re module) have been fixed.

We are no longer releasing a Python3.6 version, as we focus on updating to Python 3.8. We have begun streaming the advances towards this goal on Saturday evenings European time on https://www.twitch.tv/pypyproject. If Python3.6 is important to you, please reach out as we could offer sponsored longer term support.

The two interpreters are based on much the same codebase, thus the multiple release. This is a micro release, all APIs are compatible with the other 7.3 releases. Highlights of the release include binary Windows 64 support, faster numerical instance fields, and a preliminary HPy backend.

A new contributor (Ondrej Baranovič - thanks!) took us up on the challenge to get windows 64-bit support. The work has been merged and for the first time we are releasing a 64-bit Windows binary package.

The release contains the biggest change to PyPy’s implementation of the instances of user-defined classes in many years. The optimization was motivated by the report of performance problems running a numerical particle emulation. We implemented an optimization that stores int and float instance fields in an unboxed way, as long as these fields are type-stable (meaning that the same field always stores the same type, using the principle of type freezing). This gives significant performance improvements on numerical pure-Python code, and other code where instances store many integers or floating point numbers.

There were also a number of optimizations for methods around strings and bytes, following user reported performance problems. If you are unhappy with PyPy’s performance on some code of yours, please report an issue!

A major new feature is prelminary support for the Universal mode of HPy: a new way of writing c-extension modules to totally encapsulate PyObject*. The goal, as laid out in the HPy documentation and recent HPy blog post, is to enable a migration path for c-extension authors who wish their code to be performant on alternative interpreters like GraalPython (written on top of the Java virtual machine), RustPython, and PyPy. Thanks to Oracle and IBM for sponsoring work on HPy.

Support for the vmprof statistical profiler has been extended to ARM64 via a built-in backend.

Several issues exposed in the 7.3.3 release were fixed. Many of them came from the great work ongoing to ship PyPy-compatible binary packages in conda-forge. A big shout out to them for taking this on.

Development of PyPy takes place on https://foss.heptapod.net/pypy/pypy. We have seen an increase in the number of drive-by contributors who are able to use gitlab + mercurial to create merge requests.

The CFFI backend has been updated to version 1.14.5 and the cppyy backend to 1.14.2. We recommend using CFFI rather than C-extensions to interact with C, and using cppyy for performant wrapping of C++ code for Python.

As always, we strongly recommend updating to the latest versions. Many fixes are the direct result of end-user bug reports, so please continue reporting issues as they crop up.

You can find links to download the v7.3.4 releases here:

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our renovated blog site via a pull request to https://github.com/pypy/pypy.org

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on PyPy, or general help with making RPython’s JIT even better. Since the previous release, we have accepted contributions from 10 new contributors, thanks for pitching in, and welcome to the project!

If you are a python library maintainer and use C-extensions, please consider making a cffi / cppyy version of your library that would be performant on PyPy. In any case both cibuildwheel and the multibuild system support building wheels for PyPy.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython 2.7, 3.7, and soon 3.8. It’s fast (PyPy and CPython 3.7.4 performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This PyPy release supports:

  • x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32/64 bits, OpenBSD, FreeBSD)
  • big- and little-endian variants of PPC64 running Linux,
  • s390x running Linux
  • 64-bit ARM machines running Linux.

PyPy does support ARM 32 bit processors, but does not release binaries.

Changelog

Bugfixes shared across versions

  • Test, fix xml default attribute values (issue 3333, bpo 42151)
  • Rename _hashlib.Hash to HASH to match CPython
  • Fix loading system libraries with ctypes on macOS Big Sur (issue 3314)
  • Fix __thread_id in greenlets (issue 3381)
  • Reject XML entity declarations in plist files (bpo 42051)
  • Make compare_digest more constant-time (bpo 40791)
  • Only use ‘&’ as a query string separator in url parsing (bpo 42967)
  • Fix __r*__ reverse methods on weakref proxies
  • Restore pickle of dict iterators on default (python2.7)

Speedups and enhancements shared across versions

  • Introduce a new RPython decorator @llhelper_error_value, which officializes the fact that you can raise RPython exceptions from llhelpers, and makes it possible to specify what is the C value to return in case of errors. Useful for HPY
  • Introduce a new RPython decorator @never_allocate which ensures a class is never instantiated at runtime. Useful for objects that are required to be constant-folded away
  • Upstream internal cparser tool from pypy/ to rpython/
  • Make set.update(<non-set>) more jit-friendly by - unrolling it if the number of args is small (usually 1) - jitting the adding of new elements which fixes test_unpack_ex on PyPy3.7 as a side-effect
  • Fix position of elif clauses in the AST
  • Make the exe stack larger on windows
  • Implement constcharpsize2str in rffi and refactor code to use it
  • Add a versions.json to https://downloads.python.org/pypy/versions.json for the gitlab actions python install action
  • Add symlinks for python, python3 to the package tarballs (not on Windows)
  • Fix a missing error: bare except:-clauses should come last in codegen
  • Copy manifest from CPython and link it into pypy.exe (issue 3363)
  • Preserve None passed as REG_BINARY instead of crashing or changing it to an empty string in winreg (bpo 21151)
  • Add REG_QWORD* and Reg{Dis,En}ableReflectionKey, and RegDeleteKeyEx to winreg
  • Backport msvc detection from python3, which probably breaks using Visual Studio 2008 (MSVC9, or the version that used to be used to build CPython2.7 on Windows)
  • Optimize chains of longlong2float(float2longlong(x)) and vice versa
  • Optimize instances of maps with integer and float fields by storing them unboxed in a float array (on 32bit machines this is only done for float fields). The float array is stored in one of the storage slots of the instance. Once a field proves to be type-unstable we give up on type-specializing this class and revert all instances once we touch them the next time to the default representation.
  • Update the version of Tk/Tcl on windows to 8.6
  • Updated cppyy API to cppyy_backend 1.14.2: consistent types for Win64, support for new builtin types
  • Refactor the intbound analysis in the JIT
  • Faster str.replace and bytes.replace implementations.
  • Implement vmprof support for aarch64
  • Fast path for unicode.upper/lower, unicodedb.toupper/lower for ascii, latin-1
  • Add a JIT driver for re.split
  • Expose os.memfd_create on Linux for glibc>2.27 (not on portable builds)
  • Add a shortcut for re.sub doing zero replacements for things like escaping characters)
  • Expose the physical size of a list in __pypy__.list_get_physical_size
  • Clean up the icon bundled with the exe in windows
  • Add a fast path for list[:] = l2
  • Update packaged OpenSSL to 1.1.1k
  • Make ARM builds portable

C-API (cpyext) and C-extensions

  • make order of arguments of PyDescr_NewGetSet consistent with CPython, related to issue 2267
  • Fix parsing inf and friends in PyOS_string_to_double (issue 3375)
  • Fix signature of PyEval_EvalCode
  • Change parameter type of PyModule_New to const char*, add PyModule_Check and PyModule_CheckExact
  • Add PyUnicode_Contains (issue 3400)
  • Fix PyObject_Format for type objects (issue 3404)
  • Move inttypes.h into pyport.h (issue 3407)
  • Sync Py_.*Flags values with sys.flags (issue 3409)
  • Make PyUnicode_Check a macro, but still export the function from the shared library for backwards compatibility

Python 3.7+

  • Update the re module to the Python 3.7 implementation
  • Fix the crypt thread lock (issue 3395) and fix input encoding (issue 3378)
  • Fixes utf_8_decode for final=False (issue 3348)
  • Test, fix for time.strftime(u'%y\ud800%m', time.localtime(192039127))
  • CALL_FUNCTION_KW with keyword arguments is now much faster, because the data structure storing the arguments can be removed by the JIT
  • Fix the repr of subclasses
  • Better error message for object.__init__ with too many parameters
  • Fix bug in codecs where using a function from the parser turns warnings into SyntaxErrors a bit too eagerly
  • Produce proper deprecation warnings from the compiler, with the right filename and line number
  • Fixes for circular imports (bpo 30024) and stack usage (bpo 31286)
  • A type annotated assignment was incorrectly handled in the scoping rules, leading to a crash in complex situations (issue 3355)
  • Fix a segfault in nonblocking bufferio reads (issue 3172)
  • Use correct slot for sni_callback attribute in _ssl (issue 3359)
  • Hang on to servername_callback handle in _ssl so it will not be deleted until the context is deleted (issue 3396)
  • Implement set_wakeup_fd(warn_on_full_buffer) (issue 3227)
  • Round-trip invalid UTF-16 data in winreg without a UnicodeDecodeError (issue 3342)
  • Truncate REG_SZ at first NULL in winreg to match reg.exe behaviour (bpo 25778)
  • Fix for surrogates in winreg input value (issue 3345)
  • In sysconfig, INCLUDEPY and INCLUDEDIR should point to the original directory even in a virtualenv (issue 3364)
  • Add LDLIBRARY to sysconfig for posgresql
  • Prevent overflow in _hash_long on win64 using method from CPython
  • Raise ValueError when argv[0] of execv and friends is empty (bpo 28732)
  • Allow compiler to inherit flags from __future__.annotations (issue 3371)
  • Provide a PyPy BytesBuilder alternative to io.BytesIO in pure-python pickle
  • Generalize venv to copy all *.exe and *.dll for windows
  • The evaluation order of keys and values of large dict literals was wrong in 3.7 (in lower versions it was the same way, but in 3.7 the evaluation order of small dicts changed), issue 3380
  • Cache the imported re module in _sre (going through __import__ is unfortunately quite expensive on 3.x)
  • Mention a repeated keyword argument in the error message
  • Stop emitting the STORE_ANNOTATION and BINARY_DIVIDE bytecodes, update pyc magic number
  • Fix site.py to be closer to upstream to enable pip install --user and pip install --local
  • No longer call eval() on content received via HTTP in CJK codec tests (bpo 41944)
  • Add missing c_/f_/contiguous flags on memoryview
  • Fix xml.ElementTree.extend not working on iterators (issue 3181, bpo 43399)
  • Python -m now adds starting directory to sys.path (bpo 33053)
  • Reimplement heapq.merge() using a linked tournament tree (bpo 38938)
  • Fix shring of cursors in sqllite3 (issues 3351 and 3403)
  • Fix remaining sqllite3 incompatibilities
  • Fix CALL_METHOD_KW to not lose the immutability of the keyword name tuple

Python 3.7 C-API

  • Change char * to const char * in PyStructSequence_Field, PyStructSequence_Desc, PyGetSetDef, wrapperbase
  • Implement METH_FASTCALL (issue 3357)
  • Add pystrtod.h and expose constants
  • Clean up some char * -> const char * misnaming (issue 3362)
  • Accept NULL input to PyLong_AsUnsignedLongLongMask
  • Add PyImport_GetModule (issue 3385)
  • Converting utf-8 to 1-byte buffers must consider latin-1 encoding (issue 3413)
  • Fix value of .__module__ and .__name__ on the result of PyType_FromSpec
  • Add missing PyFile_FromFd