How Python Is Evolving
How does Python evolve? Stack Overflow explored the issue on its podcast, with Pablo Galindo Salgado — one of Python’s core developers, and one of the five people on Python’s steering council.
Salgado was also the release manager for the last two major releases, Python 3.10 (in October of 2021) and 3.11 (October 2022), which he jokes on the podcast “sounds very flashy, but it’s just the person who pushes the buttons to get the binaries out, which is not the most fashionable job, but someone needs to do it and from time to time is good…
“One time we actually broke GitHub by the way.”
Python 3.11 is finally released. In the CPython release team, we have put a lot of effort into making 3.11 the best version of Python possible. Better tracebacks, faster Python, exception groups and except*, typing improvements and much more. Get it here:https://t.co/WNlHRrD8Gg pic.twitter.com/WHpc86TTwd
— Pablo Galindo Salgado (@pyblogsal) October 24, 2022
What comes through in the talk is a passion for the language that happens at several levels. It starts with the developers and committee who maintain and expand the language for its loyal community. Yet that same community is also helping to grow the language, not just by applying it to new problem domains, but also by collaborating in open processes to identify its most-needed new features. Python is fine-tuning the language for data science and AI applications, and looking at ways to avoid performance hits from Python’s Global Interpreter Lock.
But while the future may hold a faster and more performant Python, it’s this community of passionate and committed users who will help to make it happen…
Change and Compatibility
Change is always happening. As a core developer Salgado helped implement Python’s new parser and the Python compiler. Version 3.12 of Python included improvements for Python’s formatted string literals (or “f-strings”, the special strings with variables that get replaced by values at runtime). And Salgado told Dice.com in January that longer-term goals include WebAssembly compatibility and improving Python’s typing system, its user interface, and its error messages.
But another long-term goal is maintaining backwards compatibility. On an earlier Stack Overflow podcast, Salgado admits the number of people still using Python 2 “is not zero… But I’m happy to say that it’s very, very few. I think it’s in the dozens…”
Python’s break of backwards compatibility with Python 2 remains a lingering memory — which still hangs over their effort to change Python’s Global Interpreter Lock. “We know what happened with Python 2 to Python 3,” Salgado says later in the interview. “We have learned that lesson, and we don’t want to repeat that even in the slightest, and that’s very important for us.”
Interfacing with AI
Galindo was asked about companies interested in applying generative AI and large language models — and whether he sees that leading to new Python frameworks or language features. But he responded with a candid observation about the core team. “We have been really bad at predicting what was the new big thing in Python.” Whether it’s web development and the Django web framework, data science, or (today), generative AI, “we are always a bit behind.”
Python responds partly by focusing on flexibility, says Salgado. “We try to make all possible workflows, or as many as possible, available and possible with Python.” As an example, Salgado says he’s seen AI frameworks — written in speedy compiled languages like Rust, C++ or Swift — that still have a layer of Python on top for interacting. It’s this flexibility that “allows you to keep going and it gives you this momentum as a developer” — and of course, the Python layer on top allows quick iterations and the easy interactivity of an interpreted language. “You run a little experiment and something happens and you can have instant feedback — and then you can move things around, and you don’t need to wait for compilation and what not.”
Salgado believes communities working on AI with large codebases appreciate the benefits of Python’s optional “hint” syntax for indicating variable types (and the way it enables type-checking tools). Physicists appreciate the syntax modification that lets them alert type-checking tools about the use of oddly-shaped multidimensional arrays, he says on the podcast.
Salgado also looked back to when Python added its matrix multiplication operator. It’s a standalone @ symbol signifying a popular math operation where the value in one matrix’s columns get multipled by the row values in another matrix in a process that ultimately creates a third matrix. Salgado remembered this as a moment when “we actually changed the language to help these communities… that was mainly for the data science people,” where this operation is very commonly used.
In a January interview with Dice.com, Salgado highlighted the matrix multiplication operator a good example of how the language gets evolved. As Salgado put it in 2021, “Python is developed out in the open, primarily by volunteers. As a result, any problems must be resolved through a partnership between the community, the core development team, and the Python Steering Council.
“Sometimes this makes things a bit more challenging because there isn’t a coordinated effort to approach some of these challenges. On the other hand, many members of the core dev team and the Python community at large are interested in these problems and are active in investigating different solutions…
“The fact that the development is done in the open, primarily by volunteers, creates some challenges, but also allows anyone to give their input and participate. This often results in much better solutions than those that would be reached by a very controlling set of developers, all of whom have similar backgrounds.”
About that Global Interpreter Lock
The podcasters had to ask about Python’s pesky Global Interpreter Lock, which prevents parallel processing (so at best, Salgado says, “the threads are kind of switching between themselves, but you cannot ever have two threads running Python code at the same time.”) But the Python core developers are addressing it — exploring an optional thread-limiting lock — specifically because they’ve sensed a need from within its community of users. “Right now we are feeling that in the data science world, and especially in the AI world, there is all these super parallel libraries underneath in C++ and they cannot really leverage the whole power of the thing because of this restriction.” So for AI/data science communities, “it will allow them to leverage these parallel workflows when you don’t have a lot of communication in the threads maybe, or you’re leveraging the actual raw power of the C++ libraries.”
People have approached the problem, but it’s difficult. (“Obviously you can solve it but it means that now Python is twice as long, so that’s probably not acceptable.”) But the Python user community came through again, when last year Sam Gross, a software engineer at Meta AI, actually implemented a proof-of-concept version of Python’s CPython reference language without the Global Interpreter Lock — and “in a way that doesn’t have that much impact on performance,” says Salgado.
Salgado describes the possibility as “a very impactful thing… and especially on the people doing artificial intelligence and data science because it will allow them to leverage these parallel workflows when you don’t have a lot of communication in the threads maybe, or you’re leveraging the actual raw power of the C++ libraries.”
When asked how Python can keeps up with trends and activity (besides adding new features), Salgado says they’re more focused on maintaining the productivity of developers — and “Right now, I think for us what is more important is that Python doesn’t get in the way.” He acknowledges the Global Interpreter Lock as a case “when it does, unfortunately…”
An Exciting Future
So when the interviewers closed by asking what’s exciting in Python’s future, Salgado revisited the possibility of changing Global Interpreter Lock. “I would like to think that there is a world where this can actually happen, just because this has been blocking a lot of possible workflows.”
But that’s not the only effort to speed Python. Salgado was also excited about Guido von Rossum’s team at Microsoft and their work on speeding up Pyton’s CPython reference implementation (the default and most widely used implementation). “It’s not going to be as fast as C++, so if anyone is thinking that, this will be a reality check… But free speed is important… you don’t need to touch anything and then you get free speed!”
Long ago the seminal Zen of Python document intoned that “There should be one — and preferably only one — obvious way to do it.” So the interviewers ask whether that creates a tension for the language’s steering community — or a fear of introducing multiple ways of doing things?
Pleading guilty, Salgado admits that Python breaks that rule most clearly with stream formatting. “I think we have six ways to do it!” But he adds that in the real world, “It’s impossible to be omniscient and know the best way to do it.” So when it comes to striving for one way to do it, Salgado has a ready response.
“I would modify the sentence to say, there has to be only one very good way to do it.”
