Summary: multiprocessing.dummy.Pool().map() offers a really simple solution to the “do N independent tasks with M threads” problem in Python:
import multiprocessing.dummy, urllib2
pool = multiprocessing.dummy.Pool(4) # thread pool size; defaults to number of CPUs
pool.map(urllib2.urlopen, urls)
pool.close()
pool.join()
No real discussion of why you’d use multiprocessing.dummy instead of just multiprocessing, but the answer is (I think) that you can’t transparently pass e.g. open HTTP transactions (what urlopen returns) back from a subprocess to the parent in Python.
This is simple enough that I may start using it tomorrow.
Summary:
multiprocessing.dummy.Pool().map()offers a really simple solution to the “do N independent tasks with M threads” problem in Python:No real discussion of why you’d use
multiprocessing.dummyinstead of justmultiprocessing, but the answer is (I think) that you can’t transparently pass e.g. open HTTP transactions (what urlopen returns) back from a subprocess to the parent in Python.This is simple enough that I may start using it tomorrow.