I'm trying to run multiple repetitive tasks in parallel under python. I'm pretty new to multiprocessing but as all the tasks are independant, I used the simple following piece of code :

    import numpy as np
    import sys
    import os
    import glob
    import matplotlib.pyplot as plt
    import concurrent.futures as Cfut
    
    def analize(simul, N_thread):
        path = os.getcwd()

        print('Analizing topo ...')
        Data = output of some calculations
    
        print('Writing Data ...')
        np.save('Data', Data)
    
        print('Data saved')
        print('Printing figures')

        plt.figure()
        plt.plot(Data[0])
        plt.savefig('figure1.pdf')

        plt.clf()
        plt.plot(Data[0])
        plt.savefig('figure1.pdf')

        plt.close('all')
        print('figures saved')
        os.chdir(path

if __name__ == '__main__':
    list_simul = sorted(glob.glob('Layer*'))
    executor = Cfut.ProcessPoolExecutor(5)
   #executor = Cfut.ThreadPoolExecutor(N_jobs)
    futures = [executor.submit(analize, item, 10) for item in list_simul]
    Cfut.wait(futures)

It ran very well during 3 months, and since since morning I get sometimes the following error :

Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x7fac6c187f98>>
    Traceback (most recent call last):
      File "/usr/lib/python3.5/tkinter/__init__.py", line 3359, in __del__
        self.tk.call('image', 'delete', self.name)
    RuntimeError: main thread is not in main loop

What is odd is that some jobs are done without any problem, and it happens not all the time. With a bit of research, I found a lot of posts with this problem, all of them were related to GUI. I understood the problem, but I guess I should be able to find a workaround since I'm not showing any figure, just creating the object and saving it, and since all the tasks are independent.

Any hint ?

matplotlib uses TkAgg for default. Backends like TkAgg, FltkAgg, GTK, GTKAgg, GTKCairo, Wx, and WxAgg are all GUI based. And most GUI backends require being run from the main thread. Thus, if you are running on an environment without GUI, it will throw RuntimeError: main thread is not in main loop.

Therefore, just switch to backends that do not use GUI: Agg, Cairo, PS, PDF, or SVG.

For example:

import matplotlib.pyplot as plt
plt.switch_backend('agg')

References: https://matplotlib.org/stable/faq/howto_faq.html#work-with-threads