Friday, October 22, 2010

Model-Builder: the road to 1.0

Model-Builder is a niche educational software and the first "largish" Python application I ever wrote, way back in 2001. Back then, WxPython seemed like the easiest way to develop a GUI application inPython, mainly due to the automation provided by the IDE Boa-constructor.

A lot has changed since then and for many years now, my favorite GUI Toolkit has been Qt via it's Python binding, PyQt. For a long time, WxPython turned me away from improving the GUI of Model-builder, but not anymore. Six months ago, on a lazy sunday afternoon, I decide to port Model-Builder to Qt.

Naturally that decision was made easier by long postponed desire to add features to Model-Builder. So with the port I got my first foot on the road to MB 1.0.

Porting and redesigning the GUI was the easiest part of it, The new features, both planned and invented on the way, are still keeping me from making the much anticipated stable 1.0 release.

In this post, and possibly on subsequent ones, I'll try to cover the new features implemented so far along with a brief commentary on the particular challenges associated with their implementation. First the list of the most important new features:
  1. New Qt4-based GUI
  2. Tab-based design
  3. Support to internationalization (i18n)
  4. Configuration persistence in a configuration file in the User's home directory.
  5. Splash Screen
  6. Uncertainty analysis feature based on methods from my BIP library.
  7. Report generation in Html format with Latex formatted Equations.
  8. Web application counterpart integrated with the desktop app: Model-Builder cloud community.
  9. Collection of usage statistics with user permission.
  10. Sympy based basic algebraic analysis.
  11. Social collaboration tools on Model-Builder Cloud Community.
  12. Model upload and download from the cloud.

Now let's talk a little about each of these steps:

  1. New Qt4-based GUI
    This was the best decision I could have made. Qt4 is a great GUI toolkit and together with QtWebkit and QScintilla, it allowed me to easily implement a lot of interesting features. QScintilla has builtin support for syntax highlighting, which I use for the equation editor. Current highlighting is still very simple but can be vastly improved in the future. QtWebkit allowed me to implement the display of html reports which I plan to make exportable soon. Besides that, In conjunction with Eric4, It's a breeze to handle the internationalization of the GUI, thanks to the integration with QtLinguist.
  2. Tab-based design.
    The tab-based design was meant to allow for the separation of various modes of operation of MB without the need to open separate windows all the time. Moreover it provides events which I use to sinchronize the various activities, for example every time a tab changes, MB check if model-definition has changed and updates the control panel in the simulator.
  3. Support to internationalization (i18n)
    Without Qt4 I would not have the time to do internationalization by more traditional methods. The first language supported, besides the default english, is my own native language, Brazilian Portuguese, Translators are welcome, a new translation can be completed in less than an hour.
  4. Configuration persistence in a configuration file in the User's home directory.
    That became necessary to facilitate interaction with the web app. Otherwise the user would have to enter his/her userid every time.
  5. Splash Screen.
    This is just a bit of eye candy. ;-)
  6. Uncertainty analysis feature based on methods from my BIP library.
    This feature consumed most of the implementation effort in this branch, and is still not complete, but pretty close. It is not as flexible a tool as using BIP directly as a library in a script, but does a good job to fit arbitrary models to data. Implementation of this feature required the implementation of loading and display of data-sets in csv format. Basically a simple spreadsheet-like widget, with support to opening multiples files in separate sheets.
  7. Report generation in Html format with Latex formatted Equations.
    This feature was needed for better formatting of the model's equations in mathematical notation. For the rendering part, I chose the MathJax javascript library which converts LaTeX expressions on the fly, with great quality. The LaTeX is generated by Sympy, which is now a required dependency. The Html is compose from Jinja2 templates. One unresolved issue with this solution is the size of the MathJax library, since QtWebkit has no caching support and packing it with MB would gretly increase the total size of the package.
  8. Web application counterpart integrated with the desktop app: Model-Builder cloud community.
    This is by far the biggest innovation for this release. The Model-Builder Cloud Community is meant to serve initially as an online model repository, but evolve to become a full-fledged online collaboration platform dedicated to the Model-Builder users. It already supports OpenId Authentication, basic social features such as inviting others to collaborate on models, visualization of models, etc. Sugestion for new features are welcome, and the source code for the web app is also open, so if you fell like collaborating, check it out!
  9. Collection of usage statistics with user permission.
    One of the most useful features for me. It will allow me to track the number of installations, and uses of MB. All with user permission, of course.
  10. Sympy based basic algebraic analysis.
    Since I had to add Sympy as a dependency for the formatting of the equations, I decided to use it to provide further algebraic manipulations of the models. The first thing I implemented was an automatic derivation of the Jacobian Matrix for the system of equations. I am considering embedding an Isympy console in a tab, to give the user the full power of Sympy to analyze the model.
  11. Social collaboration tools on Model-Builder Cloud Community.
    This is still very simple. Users can invite others to collaborate in a model and can also visualize their network of collaborators both in table format and as a graph on a map. The webapp captures the user's geo location (if permission is granted).
  12. Model upload and download from the cloud.
    This is the corner stone of making the website into a model repository. In the future it may interesting to add semantic descriptors which can be filled in after the model is uploaded. Currently there is no support for storing data on the web but graphical snapshots of the last run are planned.
That is it. Collaborators are very much welcome both for the desktop and the web application. My bandwith is definetly not enough to implement all this and still get food on the table. ;-)

Sunday, August 22, 2010

Python in the Clouds

Consegi 2010 is over. It was a great meeting! I was happily suprised to realize how far the adoption of free software has come in the last few years. Congratulations to the Free Software community for such a successful lobby!

Big thanks are due to Giuseppe (@gsromag) , Luiz Guilherme (@aldabalde) and their colleagues from SERPRO, who where responsible for the or ganization of this amazing meeting.

Python was also a big winner in this event. A large number of talks and workshops either focused or used python directly. I personally gave a workshop and a talk on Google App Engine in auditoriums filled to capacity with people with the most varied backgrounds, from students to government IT people.

Another interesting trend (sentiment?) that I could pick up in the event was a certain disconfort around JAVA, mainly due the its legal liabilities as illustrated by the recent legal action unleashed by Oracle against the usage of Java in the Android platform.

I was also glad to learn about the Free Cloud alliance, and some interesting free-software, cloud related tools such as NiftyName and Neopodd.

My big thanks to the organizers for inviting me, and for such a great event.




Thursday, July 15, 2010

Consegi 2010

I have been invited to give a talk on the 3rd International Congress on Free Software and E-Government -- Consegi, taking place in Brasilia on 18-20 of august. It is a massive event bringing together Leading professionals from all over the world. Last year's event had 6000 participants from 19 countries!

This year's theme is cloud computing and I'll bee talking about Google AppEngine (GAE). Since I am not a Google employee, I plan to make an unbiased presentation of the the plaform's potential in the context of the growing ecosystem of similar cloud computing platforms. I also plan to talk about GAE integration with other Google services such as Gadgets, Wave API, and OpenSocial but I hope to be able to also discuss the use of GAE as a front-end to other cloud-based systems which may require heavier computing and data storage features which are not offered by GAE. By this I mean scientific computing and data analysis platforms, which are subjects which I am deeply involved with.

The topics above are naturally of great importance to E-gov projects but I am afraid I wont have time to go into this subject, Hopefully in the q&a after my talk some one will give me an opportunity to talk about this.

I'd appreciate to hear any suggestions of other topics which might fit in such a talk.

ccp

Amazon