evokit.core.accelerator package

Submodules

evokit.core.accelerator.parallelisers module

evokit.core.accelerator.parallelisers.parallelise_task(fn: Callable[[S, A], B], self: S, iterable: Sequence, processes: int | ProcessPoolExecutor | Pool | None, share_self: bool) → Sequence

Parallelise tasks such as variation and evaluation.

Default implementations in Variator.vary_population() and Evaluator.evaluate_population() use this method.

Warning

Consider installing multiprocess (by the author of dill) if you run into any trouble with multiprocessing.

Do not give multiprocessing.pool.Pool in processes. Instead, consider using either an integer, or a multiprocess.pool.Pool, or a concurrent.futures.ProcessPoolExecutor.

The multiprocessing module, which is part of the standard library, uses pickle. This means a self, which contains lambdas, complex objects, and most complex objects created in a Jupyter notebook, would crash worker processes. This is impossible to detect and correct in multiprocessing.pool.Pool because it would attempt to restart crashed workers, which leads to indefinite hanging.

Correcting this would require a significant investment in time and diving quite deep into the Python source code. This is beyond the Developer’s capabilities.

Parameters:

• fn – Task to be parallelised.

• self – The caller, which might be shared by worker processes if share_self is True.

• iterable – Data to be processed in parallel.

• processes –

  Option that decides how may processes to use. Can be an int, a ProcessPoolExecutor, or None.

  • If processes is an int: create a new ProcessPoolExecutor with processes workers, then use it to execute the task. On Windows, it must be at most 61.

  • If processes is a ProcessPoolExecutor: use it to execute the task.

  • If (by default) processes==None: Do not parallelise.

  To use all available processors, set processes to os.process_cpu_count().

• share_self –

  If True, share a deep copy of self to each worker process. Non-serialisable attributes are replaced with None instead.

  If processes is None, then this argument has no effect.

evokit.core.accelerator.parallelisers.__getstate__(self: object) → dict[str, Any]

Machinery.

Ensure that when this object is pickled, its process pool, if any is defined (see Variator.processes() and Evaluator.processes), is not pickled.

evokit.core.accelerator.parallelisers.__deepcopy__(self: object, memo: dict[int, Any])

Machinery.

Ensure that when this object is shared by processes, its non-serialisable members are not copied.

Module contents

This module contains utility functions that parallelise the standard Variator and the standard Evaluator.

Module name is coincidentally French.

evokit.core.accelerator.parallelise_task(fn: Callable[[S, A], B], self: S, iterable: Sequence, processes: int | ProcessPoolExecutor | Pool | None, share_self: bool) → Sequence

Parallelise tasks such as variation and evaluation.

Default implementations in Variator.vary_population() and Evaluator.evaluate_population() use this method.

Warning

Consider installing multiprocess (by the author of dill) if you run into any trouble with multiprocessing.

Do not give multiprocessing.pool.Pool in processes. Instead, consider using either an integer, or a multiprocess.pool.Pool, or a concurrent.futures.ProcessPoolExecutor.

The multiprocessing module, which is part of the standard library, uses pickle. This means a self, which contains lambdas, complex objects, and most complex objects created in a Jupyter notebook, would crash worker processes. This is impossible to detect and correct in multiprocessing.pool.Pool because it would attempt to restart crashed workers, which leads to indefinite hanging.

Correcting this would require a significant investment in time and diving quite deep into the Python source code. This is beyond the Developer’s capabilities.

Parameters:

• fn – Task to be parallelised.

• self – The caller, which might be shared by worker processes if share_self is True.

• iterable – Data to be processed in parallel.

• processes –

  Option that decides how may processes to use. Can be an int, a ProcessPoolExecutor, or None.

  • If processes is an int: create a new ProcessPoolExecutor with processes workers, then use it to execute the task. On Windows, it must be at most 61.

  • If processes is a ProcessPoolExecutor: use it to execute the task.

  • If (by default) processes==None: Do not parallelise.

  To use all available processors, set processes to os.process_cpu_count().

• share_self –

  If True, share a deep copy of self to each worker process. Non-serialisable attributes are replaced with None instead.

  If processes is None, then this argument has no effect.