It might be worth trying to confirm that the problem is still present in the current version of macOS. One report suggests that it is an intermittent problem and expresses pessimism about whether the change to macOS that will be required to fix this problem will be implemented by Apple. I have not tried to reproduce the crash of the subprocess on my more recent version of macOS. But a preliminary look suggests that the key difference is that multiprocess has reverted to ‘fork’ as the default start method even on macOS. There is a report that the problem arises because of the limitations of ‘pickle’ and that because the multiprocess fork of the multiprocessing library uses ‘dill’ instead, this error can be avoided. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. On the other hand, if I change the start method to ‘fork’, this code works and so do the examples cited above.Īccording to the documentation for multiprocessing, this switch to ‘spawn’ was made because: I get the same error when I use ‘forkserver’ as the start method. Self = (from_parent)ĪttributeError: Can't get attribute 'subprocess_function' on When I run this using ‘spawn’ as the start method I get the following error: Traceback (most recent call last):įile "/Library/Frameworks/amework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_mainįile "/Library/Frameworks/amework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 126, in _main Ps = mp.Process(target=subprocess_function, args=((() / "mp_test.txt"), )) Relative to the code in this solution, I modified the code slightly to make it easy to try different start methods and to have the subprocess write to a file so I could rule out problems related to having two different output streams in the notebook: # Long running cell I tested the code I quoted using Python 3.10.7 and Jupyter Lab 3.4.8 on macOS 12.6 on both an M1 Mac and an x86 Mac. This may be caused by changes to the multiprocessing module that have taken effect since others tested it – in particular, the switch to the ‘spawn’ start method implemented in Python 3.8. I tried this code and am getting an error. Yet, while the first ‘long-running’ cell keeps running, you are able to run the other ‘normal’ cells. The output from the first cell stays isolated in the first cell as it continues to run and doesn’t pollute elsewhere. This is working in my tests in the classic notebook interface and JupyterLab.
Multiprocessing.Process(target=network_call).start() Better approach to try, running the long running cell in a multiprocessing process, see Python 3 Module of the Week: multiprocessing – Manage processes like threads.įollowing your posted example, you’d run: # Long running cell
0 kommentar(er)
