Multiprocessing Architecture & Shared Memory

To achieve true multi-core parallel processing for CPU-bound computation, Python applications leverage the multiprocessing module.

By spawning independent operating system processes, each child process runs its own independent CPython interpreter instance, its own private heap memory, and its own Global Interpreter Lock, eliminating GIL contention entirely.

1. Process Start Methods: Fork vs Spawn#

How operating systems create child processes deeply influences performance, memory usage, and stability:

The Three Start Methods#

1. spawn (Default on Windows and macOS): Spawns a completely new Python process. Only the necessary resources are passed via command-line arguments and IPC. Avoids inheriting parent thread locks, making it immune to deadlocks.
2. fork (Legacy POSIX default): Clones the parent process using the OS fork() system call. Extremely fast startup utilizing Copy-On-Write (COW) memory pages.
3. forkserver: Starts a clean, single-threaded companion server process at startup. Whenever a new worker is needed, the clean server forks it, combining fast startup with lock safety.

2. IPC: Pipes & Queues vs Serialization Cost#

Because processes have completely isolated virtual memory spaces ($\mathcal{M}(P_1) \cap \mathcal{M}(P_2) = \emptyset$), transferring data between processes using standard multiprocessing.Queue requires serializing objects into bytes via pickle:

For large datasets (such as a 4 GB NumPy array), serializing, copying through the kernel pipe, and deserializing adds massive latency and memory duplication overhead.

3. High-Performance Zero-Copy Shared Memory (PEP 574)#

Python 3.8 introduced multiprocessing.shared_memory, which maps a shared physical memory block directly into the virtual address spaces of multiple processes:

Processes can read and mutate the shared buffer simultaneously with strictly zero memory copying and zero serialization overhead!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
from multiprocessing import shared_memory
import numpy as np

# In the Master Process: Allocate 100 MB shared memory block
shm_master = shared_memory.SharedMemory(create=True, size=100 * 1024 * 1024, name="market_data")

# Construct a NumPy array on top of the shared buffer
shared_array = np.ndarray((10_000, 10_000), dtype=np.int8, buffer=shm_master.buf)
shared_array.fill(42)

print(f"Master array element: {shared_array[0, 0]}")

# --- In a Worker Process: Attach to the existing shared memory ---
shm_worker = shared_memory.SharedMemory(name="market_data")
worker_array = np.ndarray((10_000, 10_000), dtype=np.int8, buffer=shm_worker.buf)

# Instantly read without copying 100 MB over IPC!
print(f"Worker array element: {worker_array[0, 0]}")  # 42

# Mutate from worker
worker_array[0, 0] = 99

# Clean up
shm_worker.close()
shm_master.close()
shm_master.unlink()  # Deletes shared memory segment from OS

4. Concept Verification#

Test your understanding of multiprocessing: