Threading and multiprocessing — PythonCert 5.0 documentation

Threading and Multiprocessing in Python

• Core Concepts: Python provides threading and multiprocessing modules for concurrency. While they share similar APIs, they serve different purposes.
• Threading Limitations:
  • Python threads are "true" OS-level threads, but there is no Python-specific thread scheduler.
  • I/O Bound: Excellent for I/O-bound tasks (can handle thousands of threads).
  • CPU Bound: Avoid using threads for CPU-bound tasks in Python, as the Global Interpreter Lock (GIL) can make them slower than sequential execution. Use C extensions that release the GIL for CPU-intensive work.
• Synchronization Challenges:
  • Race Conditions: Occur when multiple threads access and modify shared data concurrently, leading to unpredictable results.
  • Deadlocks: Occur when threads wait indefinitely for each other to release resources.
  • Livelocks: Occur when threads repeatedly change state in response to each other without making progress.
• Synchronization Tools:
  • Locks (threading.Lock): Implements mutual exclusion (mutex). Use context managers for safety.
  • RLock (Reentrant Lock): Allows the same thread to acquire the lock multiple times; useful for recursive algorithms.
  • Semaphores: Maintains a counter to limit the number of threads accessing a resource simultaneously.
  • Events: Used for signaling between threads (e.g., waiting for a specific state).
  • Conditions: Combines locking and signaling for complex producer/consumer patterns.
• Thread Management:
  • Daemon Threads: Set via thread.daemon = True. These are terminated abruptly at program exit without cleanup; useful for background tasks like garbage collection.
  • Joining: Use thread.join() to block the main program until a specific thread completes.
  • Subclassing: You can create custom threads by subclassing threading.Thread and implementing the run() method.
• Communication:
  • Queues (queue module): The preferred, thread-safe way to exchange data between threads.
  • Queue Types: Includes standard FIFO, LifoQueue (Last-In, First-Out), and PriorityQueue (retrieves lowest-valued entries first).
• Scheduling:
  • threading.Timer: Executes a function after a specified delay.
  • sched module: A low-level module for scheduling events based on absolute time or delays.