Using generators in Python

Dr. Rahul Remanan

CEO, Moad Computer

​Python generator is a stateful function that can return an intermediate result ("the next value”) to its caller. Since a Python generator maintains the function’s local state, this allows resuming the function from right where it is left off. 

Here is a simple, readable example; modified from the Python Enhancement Proposals (PEP) website. To improve the understanding of how Python generators work; it will be a fun, but, worthwhile effort to play the role of a human Python interpreter and try to execute this function.

Reading DICOM data using a generator

Since the generators perform lazy evaluation, where the value of the item is computed only when the caller requests the item, it is a really powerful mechanism of making programs more memory efficient.

For example, instead of loading all the images in a group in a single step, with the help of generators, each image can be loaded when its needed, thereby saving the need for storing the entire group of images in-memory.

In the example below, a DICOM file reader generator function is created to reduce the memory foot-print of the program while reading DICOM images.

Import libraries

Helper function to read DICOM data

DICOM reader generator function

Extending Python generators using Sequence API in Tensroflow Keras

Sequence is an improvement on the Python generators. They are a safer way to do multiprocessing.

​The Sequence structure guarantees that the network will only train once on each sample per epoch, which is not the case with generators.

In the example below,  a DICOM file reader object is created using Tensorflow Keras Seqeuence API.

References:
  1. Python Enhancement Proposals #255 - Simple generators; N. Schemenauer, T. Peters, M. L. Hetland
  2. Tensorflow documentation on Sequence