Larry Hastings - Removing Python's GIL: The Gilectomy - PyCon 2016
Larry Hastings - Removing Python's GIL: The Gilectomy - PyCon 2016 on HackerTube: recommended by 5 Hacker News commenters across 5 separate months, 2016–2023.
Larry Hastings explains what Python's GIL is, what problems it solves and how it works, and presents his Gilectomy project to remove it, including the performance and C-extension obstacles.
What HN commenters said
11 Hacker News comments link this video. Each one links back to its thread.
gshulegaardAug 2017Strongly recommends
I feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it. I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and about performance when removing the GIL): ▶ this video I am not sure I would, personally, trade single-threaded performance for enabling multi-threaded applications. I view Python as a high-level rapid prototyping language that is well suited for business logic and glue code. And for that type of workload I would value single-threaded performance over support for multi-threading. Even now, a year later, the Gilectomy project is still slightly off performance-wise (although it looks really really close :) ): [link] As noted elsewhere, multi-processing offers adequate parallelization for this type of logic. Also, coroutines and async libraries such as gevent and asyncio offer easily approachable event loops for maximizing single-threaded resource utilization. It's true that multi-processing is not a replacement for multi-threading. There definitely are tasks and workloads where multi-processing and its inherent overhead make it unsuitable as a solution. But for those tasks, I question whether or not Python itself (as an interpreted, dynamically typed language) is suitable.
in “Let's Remove the Global Interpreter Lock” · View on HN
santiagobasultoJul 2023Recommends
There's a great talk by David Beazley (I don't remember which one) that he basically explains that removing the GIL isn't difficult at all. After all, it's just one lock. The issue is all the libraries and packages that were built around it. It's now been YEARS of building on top of the GIL. So anyways, PEP 703 is a a nice effort, but I doubt we (everyday mortals) can enjoy it. Meta, and big companies with a specialized team might be able to exploit it by making sure their entire stack is GIL-free. EDIT: My bad, the talk is from Larry Hastings: ▶ this video
in “Meta pledges Three-Year sponsorship for Python if GIL removal is accepted” · View on HN
zbentleyOct 2018Recommends
▶ this video Or search “gilectomy” on LWN. Or check out stackless etc.; it turns out removing the GIL technically is historically one of the easier parts of removing the GIL entirely. It sounds like your main problem is with python’s GC model. Reference counting doesn’t have to be thread-unsafe, but in scripting/interpreted-ish languages that want their threading story to involve seamless (I.e. no special syntax unless you want it, it’s on you not to blow off your foot) sharing of state between threads at will, like Python and Ruby, a GIL or equivalent is the norm. Sometimes it’s not as intrusive as python’s, but it does seem like a necessary (or at least very likely) implementation pattern for languages that want to provide that seamlessness. You can have thread-safe reference counted GC in a traditional scripting language, but that tends to come with a much less automatic threading/concurrency API. Perl 5 is an example of that category, and it is implemented in C.
in “Questions” · View on HN
pfranzMar 2017Recommends
Larry Hastings - Removing Python's GIL: The Gilectomy - PyCon 2016 ▶ this video (I'm pretty sure this is the video I'm thinking of) It's 30m, but worth it if you're interested. Not sure what progress has been made since then.
in “CPython internals: A ten-hour codewalk through the Python interpreter (2015)” · View on HN
tedmistonJul 2016Recommends
See Larry Hastings' talk "Removing Python's GIL: The Gilectomy" from PyCon 2016. ▶ this video As well as his earlier talk "Python's Infamous GIL". [link] The GIL is a simple design that provides serious benefit to Python (end user) developers. There are plenty of options for working around this for parallelism, such as the built-in module multiprocessing.
in “Why is Python slow” · View on HN
the_why_of_yNov 2019Cites it
In addition to the points other people raised, "simply" removing the GIL by changing the current reference-counting GC to do atomic reference updates will also substantially reduce single threaded performance; there was a PyCon talk about that. ▶ this video See also Swift for another language whose implementation has performance issues caused by ubiquitous atomic reference counting.
in “Python adopts a 12-month release cycle” · View on HN
Show 5 more comments
gshulegaardFeb 2017Cites it
Just to add more context for others... C extensions called from Python are still subject to the GIL of the calling interpreter[1]. In fact, this guaranteed thread safety has led to the creation of many non-thread safe Python C extensions which is now a critical blocker to efforts to removing the GIL[2], although there are also a number of other issues as well. As noted in the StackOverflow, you can release the GIL but there are some tricks to doing so safely[3]. But threads of a C extensions library are not necessarily pinned to a single core by the GIL like regular Python byte code. Although many performance oriented libraries like numpy[4] forsake multi-threading by default in many cases and rely on parallelization by the user (programmer)[5]. [1] [link] [2] ▶ this video [3] [link] [4] [link] [5] [link]
in “The multiprocessing module in Python” · View on HN
xapataJan 2017Cites it
Python has made some trade-offs that you dislike. You complain about the negative consequences without comparing those against the benefits. One of the major factors in speed is efficient memory layout. Contrast a Python list with a NumPy array. To achieve speedier loops and vectorized arithmetic [0], the array gives up dynamic typing and dynamic sizing. In most applications, I would gladly give up some compute speed to gain some programming productivity. I love duck-typing. Formal typing has some impressive examples, but in the projects I've worked on has reduced my productivity. Perhaps because data are so often serialized to simple formats or written to databases that discard the best tools of formal typing. Anecdotal evidence, for sure. I've never been bothered by the GIL, but I have benefitted from it [1]. [0] [link] [1] ▶ this video
in “Problems I Have with Python” · View on HN
gshulegaardJul 2016Cites it
The BDFL (Benevolent Dictator For Life) Guido Von Rossum himself put forth the idea that he would consider a patch removing the GIL in 2007 [1]: > "... I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease." Unfortunately, experiments thus far have not succeeded to meet these requirements. There is some work being done by the Gilectomy project to try and meet this bar as well as some other requirements currently though [2]. But it is currently grappling with the afore-discovered performance issues that come with removing the GIL. Also at PyCon 2016, Guido himself mentions the Gilectomy project and it's potential consideration (if it works) for Python 3.6+ [3]. So when you say Python leaders declared the GIL a "non-issue", I think you are oversimplifying the actual reality of what removing the GIL means and why leaders (like Guido) have been reluctant to invest resources pursuing. [1] [link] [2] ▶ this video [3] [link]
in “Why is Python slow” · View on HN
gshulegaardJul 2016Cites it
To be fair, the GIL wasn't included because it was a simple threading model (AFAIK). It was included because it was simple to implement and it was/is fast(er) (than removing it)[1][2]. If the Gilectomy [2] project succeeds, Guido has mentioned he would consider it for Python3.6+ [3]. [1] [link] [2] ▶ this video [3] [link]
in “Why is Python slow” · View on HN
gshulegaardJul 2016Cites it
To be fair, the GIL was included (AFAIK) because it was simple to implement. Also, Guido has said (2007) [1] that he would welcome a patch to remove the GIL if...: > "... I'd welcome a set of patches into Py3k only if the performance for a single-threaded program (and for a multi-threaded but I/O-bound program) does not decrease." Unfortunately, experiments thus far have not succeeded to meet these requirements. There is some work being done by the Gilectomy project to try and meet this bar as well as some other requirements currently though [2]. But it is currently grappling with the afore-discovered performance issues that come with removing the GIL. Also at PyCon 2016, Guido himself mentions the Gilectomy project and it's potential consideration (if it works) for Python 3.6+ [3]. [1] [link] [2] ▶ this video [3] [link]
in “Why is Python slow” · View on HN