CPython C-API & Reference Ownership Rules

The CPython C-API allows developers to write native C and C++ extension modules, enabling CPU-intensive algorithms (like NumPy, PyTorch, and cryptography) to run at bare-metal speeds while interacting directly with Python object structs.

However, writing C extensions requires strict, manual adherence to Reference Ownership Formalisms. A single missing Py_DECREF in a C extension produces a permanent memory leak that the garbage collector can never detect.

C Reference Leak InvisibilityC-API

Zero GC Tracking

Reference leaks in native C code keep ob_refcnt >= 1 forever, hiding them from the cyclic GC.

1. Reference Ownership Taxonomies#

Every pointer returned or passed across C-API functions belongs to one of three ownership categories:

\text{Reference Ownership} \in \{\mathbf{Owned\_Reference}, \mathbf{Borrowed\_Reference}, \mathbf{Stolen\_Reference}\}

Architecture Diagram

Rendering diagram...

1. Owned References#

When a C-API function returns an Owned Reference, it has already incremented the object's ob_refcnt. The caller assumes legal responsibility for the reference:

If you store it in a long-lived C struct, keep it.
If you are done using it, you must call Py_DECREF(ptr); otherwise, the memory leaks forever.

2. Borrowed References#

A Borrowed Reference gives you read-only inspection access without transferring refcount ownership. The object's refcount is not incremented:

You must never call Py_DECREF on a borrowed reference! Doing so causes premature deallocation and fatal double-free segmentation faults.
If you need to keep a borrowed pointer alive across function boundaries, you must explicitly convert it to an owned reference using Py_INCREF(ptr).

3. Stolen References#

Rare functions (notably PyList_SetItem and PyTuple_SetItem) steal reference ownership from the caller:

The receiving container assumes responsibility for calling Py_DECREF when the container is destroyed.
The caller must not call Py_DECREF on an item after passing it into PyList_SetItem!

2. Anatomy of a Minimal C Extension#

Inspect a production-grade native C extension module implementing a fast integer squaring function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#define PY_SSIZE_T_CLEAN
#include <Python.h>

// C implementation of the function
static PyObject* fast_square(PyObject* self, PyObject* args) {
    long input_val;

    // Parse Python argument tuple into C long
    if (!PyArg_ParseTuple(args, "l", &input_val)) {
        return NULL; // Propagate TypeError exception
    }

    long result = input_val * input_val;

    // PyLong_FromLong returns an OWNED REFERENCE
    return PyLong_FromLong(result);
}

// Method specification table
static PyMethodDef FastMathMethods[] = {
    {"fast_square", fast_square, METH_VARARGS, "Calculates square of an integer at C speed."},
    {NULL, NULL, 0, NULL} // Sentinel terminating the table
};

// Module definition struct
static struct PyModuleDef fastmathmodule = {
    PyModuleDef_HEAD_INIT,
    "fastmath",               // Module name
    "High-speed C math module", // Docstring
    -1,                       // Global state size (-1 = stateless)
    FastMathMethods           // Method table pointer
};

// Module initialization entry point
PyMODINIT_FUNC PyInit_fastmath(void) {
    return PyModule_Create(&fastmathmodule);
}

3. The Dangerous Borrowed Reference Trap#

A classic source of segmentation faults in C extensions is holding a borrowed reference across operations that mutate the parent container:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// DANGEROUS C EXTENSION CODE
void bug_demonstration(PyObject *my_list) {
    // PyList_GetItem returns a BORROWED reference
    PyObject *item = PyList_GetItem(my_list, 0);

    // Call external Python code that might mutate or clear my_list!
    PyObject *result = PyObject_CallMethod(external_handler, "process", NULL);

    // If external_handler executed 'my_list.clear()',
    // the original item was DEALLOCATED!
    // 'item' is now a dangling pointer into freed heap memory!
    PyObject_Print(item, stdout, 0); // CRASH: Segmentation fault!
}

// CORRECT PRODUCTION PATTERN
void safe_pattern(PyObject *my_list) {
    PyObject *item = PyList_GetItem(my_list, 0);
    Py_INCREF(item); // Convert borrowed reference to OWNED reference!

    PyObject *result = PyObject_CallMethod(external_handler, "process", NULL);

    PyObject_Print(item, stdout, 0); // Completely safe!
    Py_DECREF(item); // Release ownership when done
}

4. Concept Verification#

Test your understanding of the C-API and reference ownership:

Concept check

What happens if a developer forgets to call Py_DECREF on an Owned Reference returned by PyLong_FromLong in a C extension?

Concept check

Why must you call Py_INCREF on a pointer returned by PyList_GetItem if you intend to use it after operations that might mutate the list?