<div id="readability-page-1" class="page"><div>
<p>When you want to shutdown multiprocessor pool, I find this way the best. I tried cancel_futures, and so on, but they all suffers some weird behaviors due to processes terminated forcefully.</p>



<p>Instead, we let all workers finish gracefully</p>



<p>First, define a stop latch:</p>


<div><pre title="">from threading import ValueProxy

class StopLatch:
    def __init__(self, value: ValueProxy):
        self._value = value

    def stop(self):
        self._value.set(1)

    def should_stop(self) -&gt; bool:
        return self._value.get() == 1
</pre></div>


<p>When submitting task, pass it:</p>


<div><pre title="">from multiprocessing import Manager
from concurrent.futures import ProcessPoolExecutor

  with Manager() as manager:
    value = manager.Value("i", 0)
     stop_latch = StopLatch(value)
     with ProcessPoolExecutor(
       max_workers=max_workers, initializer=get_process_initializer()
     ) as executor:
    ...
      executor.submit(..., stop_latch=stop_latch)
    ...
</pre></div>


<p>In your busy worker function, check the stop latch:</p>


<div><pre title="">for ....
   if stop_latch.should_stop():
        raise RuntimeError()
</pre></div>


<p>That’s it!</p>
</div></div>

Passion is like genius; a miracle.

When you want to shutdown multiprocessor pool, I find this way the best. I tried cancel_futures, and so on, but they all suffers some weird behaviors due to processes terminated forcefully. Instead, we let all workers finish gracefully First, define a stop latch: When submitting task, pass it: In your busy worker function, check the […]

Shutting down multi processing in python

댓글