Subscribe 11 Mar
Re: Grid computing for Python
Matthieu talks about leveraging the unused desktop-computer power in his lab for performing calculation. I share his feeling that there is a huge potential here, and I think this is where the work on IPython1 by Fernando, Brian and Benjamin comes into play. Hopefully one day we will have a tool to transform prototyping code in Python into calculations scattered on a grid made of desktop computers. Off course this can only be easy for embarrassingly parallel problems, as in general parallel algorithmic is hard, but a lot of the problems I encounter are of this kind, for instance a not-at-all-parallel computation, that I need to run for a large number of different parameters.
Posted
on
Tuesday, March 11th, 2008 at 1:58 pm under
The Condor project http://www.cs.wisc.edu/condor does just this sort of
March 12th, 2008 at 4:14 pmdistributed computing and is being used by the NEOS server
http://neos.mcs.anl.gov which is very popular in optimization circles. The more
recent versions of NEOS now have Python interfaces. NEOS accepts submission
using XML-RPC and therefore, you can do that with Python. I have a (very simple
script) at http://www.gerad.ca/~orban/pyneos. Condor is released under the
Apache license. Maybe that’s close-ish to what you are looking for?!
Hum, not really. Condor seems to be a system based on jobs described by XML files. I hate these systems, I find them too restrictive. I want to be able to describe my problem in Python. Moreover I do not want to describe it with strings of Python code, but with Python code itself. What Ipython1 is trying to achieve is much closer to that. From what I last heard, there might even be a syntax of the type:
with engine1:
big_ugly_computation()
where big_ugly_computation gets executed on the engine1, which is located on a distant machine. Obviously you have a lot of problems on how to sync operations (I don’t even know if this operation would be blocking), and how to move the code and the data around. I am really not up to date on this, and I can’t talk about what these guys are trying to do, but I can assure you that they are trying to work with pure Python code (and hitting limitations).
By the way, I messed up with wordpres (clicked on the wrong button) and had to repost you comment. Doing so I messing up you URL, here is a correct link for your package
March 12th, 2008 at 4:23 pmhttp://www.gerad.ca/~orban/pyneos
All these solutions are pretty close, I think. IPython1 uses XMLRPC under the hoods to do his stuff, so perhaps one day there will be a module able to talk to pyneos servers.
My view was that there are a lot of raw CPU time not used in every lab. That means a lot of IPython engines that could run on these. Depending on the time, some may go down, other up. IPython is able to cope with this (at least to some level).
I’d like to see the implementation of a standard P2P layer with different services that could be used, one of them being IPython1 engine services. Then one peer could ask for some some peers with the IPython1 service (that being Java, .Net or of course C Python-based peers), depending of what iss available.
March 20th, 2008 at 12:15 pmOf course this is not possible with interactive computation (something that IPython1 does great), but it is for task farming, which is my main use in the end.
Cool Thanks for your article. I am a newbie at development and this will be a big help.
March 7th, 2010 at 12:29 am