When running Python codes in parallel using concurrent.futures.ProcessPoolExecutor, I noticed that for larger values of n, there was a significant increase in RAM usage. Upon further investigation, it seemed that storing futures set (or list) was contributing to this issue. In an attempt to reduce RAM consumption, I modified the code by not storing the futures set and performing operations directly in my_function like so:
with concurrent.futures.ProcessPollExecutor(max_workers=10) as executor:
for i in range(n):
executor.submit(my_function, i)
However, this approach still resulted in high RAM usage. After some research, I came across an issue on Python's bug tracker which explained that submitting all tasks at once can lead to tasks being stored in memory until execution completes.
To address this problem without introducing delays in the loop, I sought a better method to submit tasks gradually as previous ones are completed. I considered using the map method instead of submit, but was unsure about the purpose of the chunksize argument and how to determine its value.
Having previously used GNU Parallel without encountering similar RAM issues, I am looking for a Python-only solution that optimizes memory usage. Is there a more efficient or elegant way to achieve this parallel processing while minimizing RAM consumption?