Error Multiprocessing in RF



  • Hi @frederik and @gferreira,

    I was trying to try out some parallelism in my code with RF. I'm running into a potential issue, I was wondering if it was a bug.

    The following two simple examples for multiprocessing are found on this page: https://realpython.com/python-concurrency/

    import multiprocessing
    import time
    
    
    def cpu_bound(number):
        return sum(i * i for i in range(number))
    
    
    def find_sums(numbers):
        with multiprocessing.Pool() as pool:
            pool.map(cpu_bound, numbers)
    
    
    if __name__ == "__main__":
        numbers = [5_000_000 + x for x in range(20)]
    
        start_time = time.time()
        find_sums(numbers)
        duration = time.time() - start_time
        print(f"Duration {duration} seconds")
    
    import requests
    import multiprocessing
    import time
    
    session = None
    
    
    def set_global_session():
        global session
        if not session:
            session = requests.Session()
    
    
    def download_site(url):
        with session.get(url) as response:
            name = multiprocessing.current_process().name
            print(f"{name}:Read {len(response.content)} from {url}")
    
    
    def download_all_sites(sites):
        with multiprocessing.Pool(initializer=set_global_session) as pool:
            pool.map(download_site, sites)
    
    
    if __name__ == "__main__":
        sites = [
            "https://www.jython.org",
            "http://olympus.realpython.org/dice",
        ] * 80
        start_time = time.time()
        download_all_sites(sites)
        duration = time.time() - start_time
        print(f"Downloaded {len(sites)} in {duration} seconds")
    

    For both examples, the method that is called by the multiprocessing pool (cpu_bound in the first and download_site in the second) are met with the same PicklingError:

    Traceback (most recent call last):
      File "<untitled>", line 18, in <module>
      File "<untitled>", line 11, in find_sums
      File "multiprocessing/pool.pyc", line 268, in map
      File "multiprocessing/pool.pyc", line 657, in get
      File "multiprocessing/pool.pyc", line 431, in _handle_tasks
      File "multiprocessing/connection.pyc", line 206, in send
      File "multiprocessing/reduction.pyc", line 51, in dumps
    _pickle.PicklingError: Can't pickle <function cpu_bound at 0x148c59320>: attribute lookup cpu_bound on __main__ failed
    

    Both examples run in terminal and in my IDE A-OK.

    I was wondering if either of you would know what was going on here. Is there a way I should edit the examples above to get them to run, or is it a bug in RF?

    Thanks!


  • admin

    have to dive into how RF executes py code, but I guess this solved when you run th py from a file that is saved...



  • @frederik Oh ok, I'll try that in a minute!



  • @frederik No, it still gives the same error whether or not it is run from a file via the Scripts menu or through the Python Scripting Window :/


  • admin

    I see...

    asyncio examples does work... I assume, without doing to much research, the issue is related to the combo with pyObject and an already running runloop. RoboFont is one big runloop.



  • @frederik I see, so not much hope for this specific issue to be fixed?

    Essentially where I was hoping to go with this is I think it would be great to somehow run a process in the background without RoboFont locking up the interface. Even this simple asyncio example, while it does execute in parallel, causes RoboFont to "beachball" while the process completes.

    import asyncio
    
    async def count():
        print("One")
        await asyncio.sleep(10)
        print("Two")
    
    async def main():
        await asyncio.gather(count(), count(), count())
    
    if __name__ == "__main__":
        import time
        s = time.perf_counter()
        asyncio.run(main())
        elapsed = time.perf_counter() - s
        print(f"{__file__} executed in {elapsed:0.2f} seconds.")
    

    Is there any way to spawn a process or run a script, maybe on another core (I have 8 cores on this iMac, RoboFont only runs on 1), that runs in the background, and doesn't block RF's other processes?



  • @frederik And for what it's worth, the multiprocessing scripts don't seem to work in the standalone Drawbot app either, same error. So whatever the issue is in RF, it seems like it's the same in DrawBot.

    I tried it in DrawBot because I seem to remember trying multiprocessing a year ago for a drawbot thing after seeing this post https://forum.drawbot.com/topic/150/drawbot-multiprocessing?_=1591994686928

    and it worked back then! I wonder what changed?


  • admin

    asyncio support can be improved, will try out some bit and pieces.

    Threading, multiprocessing is harder with user interaction and with the bridge between python and cocoa...

    FontGoggles uses https://github.com/alberthier/corefoundationasyncio


  • admin

    why it worked in DrawBot, I dont know.. Could be the switch between py2 -> py3.6 -> py3.7... Do you have a DrawBot version number?


  • admin

    found it!!

    in RF and DB the main module must be the py file starting the app!

    import sys
    
    print(sys.modules["__main__"])
    

    while when you run your code in terminal the __main__ module is the file itself...

    do an import of the your multiprocessing code instead.


  • admin

    DrawBot PR to improve asyncio: https://github.com/typemytype/drawbot/pull/392

    this allows to create a task in a run loop:

    # will not work in RF3.4 or DB1.126 but will work in the next releases!! 
    import asyncio
    
    async def count():
        print("One")
        await asyncio.sleep(10)
        print("Two")
    
    async def main():
        await asyncio.gather(count(), count(), count())
    
    if __name__ == "__main__":
        asyncio.create_task(main())
    


  • @frederik Frederik, this is all brilliant!! Thanks so much for digging into it.

    Importing the multiprocessing code is indeed the trick; the demos work now perfectly.

    And I can't wait to try out the new async stuff in the next releases!

    Thanks!!


Log in to reply