Parallel Processing in Python
Our computers (or laptops or virtual machines in the cloud) have multiple processors (CPUs). While running a code, only some of the CPUs may be in use. In some cases, we can use more than more CPUs and reduce the runtimes. For example, consider a “for” loop where the outcome of one iteration doesn’t depend on another. In such a case, instead of running the loop in series, one can run the loop in parallel on different available CPUs. Examples of such instances in which calculations are done sequentially using a single CPU are as follows:
- You run several Monte Carlo simulations in which you toss a coin (generating a random number for a stochastic process) in a loop (series) and then take an average.
- Forecasting several independent time series one after the other.
Introduction to Multiprocessing
Multiprocessing module in Python allows one to use multiple processors on a given machine.
Consider the following simple function that takes a value, a, and waits 2 seconds before returning the result “a” multiplied by 2. The wait tie of 2 seconds is just for demonstration purposes. In practice, these 2 seconds could be a complex calculation.
%%time
import time
def double(a):
time.sleep(2) # assume this is a complex calculation
return a * 2
double(3)## This code will take about 2 seconds
Series Function
If the above function is run a sequential manner (one after the other), then the runtime will scale proportionately.
def series_func():
for i in range(0,4): # a loop for 4 sequential runs
print('\t', double(i))
returnseries_func()
# This code will take about 2 * 4 seconds = 8 seconds
# The output of the code is
0
2
4
6
Here is the visual representation:
Parallel Function
Let’s assume that you want to get the same results in 2 seconds (not in 8 seconds). Let’s assume that your machine has 4 CPUs. Instead of the running the 4 jobs sequentially, we can run the function in parallel, and get the results in 2 seconds. Let’s try to make use of all 4 available CPUs as shown in the following figure:
Consider the following code which takes the same time for running processes as the time to take 1 process. Simply by changing the PROCESSES parameter and adding CPUs (if you are running on the cloud), you can get the results in the same time (2 seconds in the example):
%%time
import multiprocessdef parallel_func():
# List of numbers
numbers = [0, 1, 2, 3]# Create a Pool of worker processes
pool = multiprocess.Pool()# Apply the function to each number in parallel
results = pool.map(double, numbers)# Close the pool to release resources
pool.close()
pool.join()# Print the results
for r in results:
print('\t', r)
parallel_func()# This code will take about 2 (not 2*4 = 8 seconds)
# The output of the code is
0
2
4
6
