Python Multi-processing

In this era of multiple cores everywhere, it kind of makes me nervous to see one of my computer cores siting idle as the other crunches away at some numerical simulation. Running heavy numerical simulations is the bread and butter of my work, so I am always on the lookout for ways to extract as much  computational juice as I can from my CPUs.

Over last couple of years I have played different approaches available to python programmers, from tools available in the standard library such as  forking processes and the threading module, to external packages such as Parallel Python and Ipython1. All of them have their pros and cons, and in many occasions, I found myself wasting valuable computable time trying to get my simulations to run under the parallelization models inherent to each of the solutions listed above.

I will not go into details about what I lked and disliked about each of them but rather I will focus  on the future of parallel processing in the Pythonsphere: the already available and soon to be part of the standard library, Processing module (renamed multiprocessing for the standard library).

It can be installed with a simple "easy_install processing".  For those who don't know yet, the processing/multiprocessing module is a multi-processing (duh!) module using the same API as the standard library's threading module.

The processing module  takes a lot of the pain out of the process of writing parallel code when compared to other methods. By using multiple processes, it saves you from having to deal with problems associated with having a shared memory between tasks. This means you can elegantly bypass the GIL, with the same code you would write for multithreaded application minus the boilerplate code you'd have to write to handle racing conditions and whatnot. This is the meaning of  sharing the same API with the threading module. Moreover, with processing, your code runs on Windows just as well as on Linux, which is something you couldn't do with fork.

Before processing, the (IMHO) best tool for "simple" multi-processing was Parallel-Python, but I found extremely painful having to manually declare global variables and modules which  each process would have to have access to. 

I must say that so far, my experience with processing is quite limited. However, I benefit from the point of view of having implemented the same exact (simple) code on all of the said platforms except for Ipython1, and I can attest  that for simple parallelizable problems, processing makes the task it is as simple as it can get.  

In conclusion,  if you can benefit from parallel processing in your application, I strongly suggest trying out the processing module.

AppEngine and Google Base

My experiments with AppEngine keep expanding. One thing they got right
was the incentive to use other google services through google data
APIs also known as GData.
The first step towards that was forcing us to use Authsub by default
in appEngine. Noticing that, I am thinking about taking advantage of
that to aleviate the usage of the tiny storage quota offered by Google
at AppEngine.
I'll post more about this as my experiments mature.

--
FLávio Codeço Coelho, PhD

Editra: a Great New Python Editor

Editra is a relatively new (at least for me) text editor written in Python/GTK.

I have been working extensively with it recently, and although I am not ready to leave behind my currently favorite Python IDE, Eric, Editra doesn't make me miss it too much. One thing that it does that Eric still does not, even though it could, is to provide a code browser for a source file which is not part of a project. I know, many editors do that, but Eric only enables the browser for files in a project, silly thing isn't it?.

Editra has an interesting plugin system, through which it provides most of its goodies, such as the code browser, comment browser, file browser, support for projects, a python shell, themes etc. There is also a plugin for an Ipython Shell that is apparently broken.

Its project support includes source control integration for Git, Bazaar, CVS and Subversion, Unfortunately notmy favorite: Mercurial.

One thing that annoys me is the lack of support for the middle-mouse-button paste, typical of Linux, which I use a lot (on other editors, naturally). I couldn't find the convenient run button as well... too bad.

Overall it is an excellent editor, and I wish it continues to develop well. One last wish: Good code completion, if it can do that it make an excellent alternative to commercial editors such as Komodo. Kudos to its developers!