Error Multiprocessing in RF

colinmford

I was trying to try out some parallelism in my code with RF. I'm running into a potential issue, I was wondering if it was a bug.

The following two simple examples for multiprocessing are found on this page: https://realpython.com/python-concurrency/

import multiprocessing
import time


def cpu_bound(number):
    return sum(i * i for i in range(number))


def find_sums(numbers):
    with multiprocessing.Pool() as pool:
        pool.map(cpu_bound, numbers)


if __name__ == "__main__":
    numbers = [5_000_000 + x for x in range(20)]

    start_time = time.time()
    find_sums(numbers)
    duration = time.time() - start_time
    print(f"Duration {duration} seconds")

import requests
import multiprocessing
import time

session = None


def set_global_session():
    global session
    if not session:
        session = requests.Session()


def download_site(url):
    with session.get(url) as response:
        name = multiprocessing.current_process().name
        print(f"{name}:Read {len(response.content)} from {url}")


def download_all_sites(sites):
    with multiprocessing.Pool(initializer=set_global_session) as pool:
        pool.map(download_site, sites)


if __name__ == "__main__":
    sites = [
        "https://www.jython.org",
        "http://olympus.realpython.org/dice",
    ] * 80
    start_time = time.time()
    download_all_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} in {duration} seconds")

For both examples, the method that is called by the multiprocessing pool (cpu_bound in the first and download_site in the second) are met with the same PicklingError:

Traceback (most recent call last):
  File "<untitled>", line 18, in <module>
  File "<untitled>", line 11, in find_sums
  File "multiprocessing/pool.pyc", line 268, in map
  File "multiprocessing/pool.pyc", line 657, in get
  File "multiprocessing/pool.pyc", line 431, in _handle_tasks
  File "multiprocessing/connection.pyc", line 206, in send
  File "multiprocessing/reduction.pyc", line 51, in dumps
_pickle.PicklingError: Can't pickle <function cpu_bound at 0x148c59320>: attribute lookup cpu_bound on __main__ failed

Both examples run in terminal and in my IDE A-OK.

I was wondering if either of you would know what was going on here. Is there a way I should edit the examples above to get them to run, or is it a bug in RF?

Thanks!

colinmford

@frederik Frederik, this is all brilliant!! Thanks so much for digging into it.

Importing the multiprocessing code is indeed the trick; the demos work now perfectly.

And I can't wait to try out the new async stuff in the next releases!

Thanks!!

frederik

DrawBot PR to improve asyncio: https://github.com/typemytype/drawbot/pull/392

this allows to create a task in a run loop:

# will not work in RF3.4 or DB1.126 but will work in the next releases!! 
import asyncio

async def count():
    print("One")
    await asyncio.sleep(10)
    print("Two")

async def main():
    await asyncio.gather(count(), count(), count())

if __name__ == "__main__":
    asyncio.create_task(main())

frederik

found it!!

in RF and DB the main module must be the py file starting the app!

import sys

print(sys.modules["__main__"])

while when you run your code in terminal the __main__ module is the file itself...

do an import of the your multiprocessing code instead.

frederik

why it worked in DrawBot, I dont know.. Could be the switch between py2 -> py3.6 -> py3.7... Do you have a DrawBot version number?

frederik

asyncio support can be improved, will try out some bit and pieces.

Threading, multiprocessing is harder with user interaction and with the bridge between python and cocoa...

FontGoggles uses https://github.com/alberthier/corefoundationasyncio

colinmford

@frederik And for what it's worth, the multiprocessing scripts don't seem to work in the standalone Drawbot app either, same error. So whatever the issue is in RF, it seems like it's the same in DrawBot.

I tried it in DrawBot because I seem to remember trying multiprocessing a year ago for a drawbot thing after seeing this post https://forum.drawbot.com/topic/150/drawbot-multiprocessing?_=1591994686928

and it worked back then! I wonder what changed?

colinmford

@frederik I see, so not much hope for this specific issue to be fixed?

Essentially where I was hoping to go with this is I think it would be great to somehow run a process in the background without RoboFont locking up the interface. Even this simple asyncio example, while it does execute in parallel, causes RoboFont to "beachball" while the process completes.

import asyncio

async def count():
    print("One")
    await asyncio.sleep(10)
    print("Two")

async def main():
    await asyncio.gather(count(), count(), count())

if __name__ == "__main__":
    import time
    s = time.perf_counter()
    asyncio.run(main())
    elapsed = time.perf_counter() - s
    print(f"{__file__} executed in {elapsed:0.2f} seconds.")

Is there any way to spawn a process or run a script, maybe on another core (I have 8 cores on this iMac, RoboFont only runs on 1), that runs in the background, and doesn't block RF's other processes?

frederik

I see...

asyncio examples does work... I assume, without doing to much research, the issue is related to the combo with pyObject and an already running runloop. RoboFont is one big runloop.

colinmford

@frederik No, it still gives the same error whether or not it is run from a file via the Scripts menu or through the Python Scripting Window :/

colinmford

@frederik Oh ok, I'll try that in a minute!

frederik

have to dive into how RF executes py code, but I guess this solved when you run th py from a file that is saved...

SOLVED Error Multiprocessing in RF