Closures, Lexical Binding & Cell Objects

Functions in Python are first-class objects (PyFunctionObject). They can be passed as arguments, assigned to variables, and returned from other functions.

When an inner function references a variable defined in an enclosing outer function, CPython constructs a lexical closure. Even after the outer function finishes executing and its frame is removed, the inner function continues to access and mutate the enclosed variable.

Understanding how CPython achieves this without causing memory corruption requires examining the PyCellObject heap wrapper mechanism.

1. The Closure Challenge: Lifetime Mismatch#

In standard stack-based execution architectures, local variables live on the stack frame of the function that created them:

If the inner function executes later, how can it read or mutate count when the outer frame that created it no longer exists?

CPython solves this dilemma by promoting captured variables from the stack to the heap using PyCellObject.

2. The PyCellObject Architecture#

When the CPython compiler detects that an inner function references an outer local variable:

1. In the outer function, that variable is marked as a Cell variable (co_cellvars).
2. In the inner function, that variable is marked as a Free variable (co_freevars).

Instead of storing the variable directly in the outer frame's fastlocals array, CPython allocates a PyCellObject on the heap:

Both the outer function and the returned inner function hold pointers to the same PyCellObject. When the outer frame exits, only its pointer to the cell is decremented. The inner function's __closure__ tuple retains its own pointer to the cell, keeping it alive on the heap!

3. Bytecode Analysis: LOAD_DEREF & STORE_DEREF#

Inspect how the Python Virtual Machine reads and mutates closure variables:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import dis

def make_counter(start: int = 0):
    count = start

    def increment():
        nonlocal count
        count += 1
        return count

    return increment

counter = make_counter(10)
print(f"Count: {counter()}")  # 11
print(f"Count: {counter()}")  # 12

# Inspect closure metadata
print(f"\nClosure cells: {counter.__closure__}")
print(f"Cell contents: {counter.__closure__[0].cell_contents}")
print(f"Free variables: {counter.__code__.co_freevars}")

print("\n=== Inner Function Bytecode ===")
dis.dis(counter)

Operational Breakdown#

• LOAD_DEREF 0: Fetches the cell at index 0 from the function's __closure__ tuple and extracts cell_contents.
• STORE_DEREF 0: Stores the updated object pointer into cell_contents inside that exact same PyCellObject struct!

4. The nonlocal vs global Binding Rules#

Python provides explicit keywords to direct how variable bindings are scoped:

5. Concept Verification#

Test your understanding of Python closures and cell objects: