Data Visualisation with Gnuplot

Visualisation of data using charts and other types of plots is immensely helpful in getting a feel for it without carrying out detailed analyses. Gnuplot is a freely-available tool for data visualisation that is also very simple to use. The article "Visualize your data with gnuplot" is a nice introduction to this tool. Gnuplot proved to be quite handy for me recently.

I wanted to find out whether the Unit Price of a particular fund varies in line with the popular equity market indices in India, the NSE S&P CNX Nifty and the BSE Sensex. The current values of these indices are always readily available in the newspapers and on television channels, while I have to use the web-site of the fund to get its current Unit Price. If the Unit Price of the fund varied in line with the values of the equity market indices, it would save me some effort in determining its current worth.

The portfolio of the fund in question is almost entirely based on equities. It holds the shares of some of the biggest and the most stable companies across a variety of industry sectors. It was therefore reasonable to suspect that its Unit Price would vary in line with the values of the indices. However, it is not as diversified as the indices and it might not have invested across sectors in the same proportion as that represented by either of the indices.

It was easy to obtain the historical closing prices of the two indices and the Unit Prices of the fund. To keep things simple, I only considered the current month for making this comparison. To simplify things further and to improve the visualisation, I normalised the first value in each series to "100" by scaling all the values appropriately. (This is a technique that I have often seen put to good use in The Economist.)

Using Gnuplot, I obtained the following chart:



This gave me the desired answer right away!

In case you're curious, here are the Gnuplot commands I used for creating the chart above:

# We want PNG output.
set terminal png
set output "plot.png"

# Specify how and where the key (legend) for the chart should
# appear.
set key bottom right
set key width 2 box

# Tweak the look of the chart.
set title "Fluctuations in Unit Prices Relative to Market Indices"
set xlabel "June 2007"
set ylabel "Normalised Value"
set grid

# The data on the X axis represent time values.
set format x "%d"
set xtics "01-Jun-2007", 3600*24
set xdata time
set timefmt "%d-%b-%y"
set xrange ["01-Jun-2007":"22-Jun-2007"]
set yrange [95:100]

# Plot the chart using data files normalising the values in
# each case.
plot \
  'nifty.dat' using 1:($5)/42.9705 \
    title 'NIFTY' with lines linewidth 2, \
  'sensex.dat' using 1:($7)/145.7075 \
    title 'SENSEX' with lines linewidth 2, \
  'fund.dat' using 1:($2)*100.00/57.0337 \
    title 'FUND' with lines linewidth 2

By the way "Junk Charts" seems to be a blog devoted to criticising charts that appear in various magazines and web-sites in general and in The Economist in particular.

iTunes and QuickTime

iTunes and QuickTime would probably be the applications by Apple that most non-Mac-OS-X users get exposed to and it is a shame that these applications are so frustratingly unintuitive, not to mention bloated and slow. Are these really made by the same company that is famous for putting a lot of emphasis on the usability of its hardware and software products?

Indic Scripts and Linux

If you have the fonts for Indic scripts (for example, the Lohit fonts), Firefox on Linux is able to display the Devanagari text on sites like BBC Hindi and Google News in Hindi. (Devanagari is the primary writing system for languages like Hindi, Sanskrit, etc.) However, if you are using the builds released by mozilla.com, you would notice that the matras (diacritics) are not applied properly to form the correct ligatures. For example, the word "हिन्दी" ("Hindi") itself is not rendered properly. Konqueror does not suffer from such problems.

It turns out that Firefox does not support complex text layout because it doesn't use Pango in the officially-released builds (Firefox 3 will support it by default). You have to either compile it yourself from the source and enable the support for Pango by using --enable-pango, or use a build that has Pango enabled - for example, the builds provided by the Fedora Project. (Setting the environment variable MOZ_ENABLE_PANGO to "1" had no effect for me with Firefox 2.0.0.3.)

On Fedora Core 6 (FC6), it is very simple to get this working:

1. Install the fonts for the Indic scripts you are interested in. For example, "sudo yum install fonts-hindi" , "sudo yum install fonts-malayalam", "sudo yum install fonts-kannada", etc.
2. Install a Firefox build for Fedora using "sudo yum install firefox". Note that FC6 installs Firefox 1.5 by default - if you prefer Firefox 2.0 instead, you can install it using "sudo yum --enablerepo=development install firefox".

By the way, I recently came across Omniglot, a site about the writing systems of almost all known human languages, existing or extinct, naturally-evolved or artificially-created. I found it extremely fascinating and insightful. For example, I did not know that Devanagari was not considered to be an "alphabet" but an "abiguda". Check out the International Phonetic Alphabet (IPA) that can represent almost all spoken languages. How about Loglan (and its freer derivative, Lojban) that claims to be a "logical" language? (I first came across the IPA on Wikipedia, where it is used to provide the pronunciation for some terms. xkcd is where I first read about Lojban.)

ICFP Contest 2007

The 10th ICFP Contest is scheduled for the weekend of 20^th July 2007. I hope it turns out to be as fun as the one held last year.

There is already a blog written by one of the organisers that contains some teaser puzzles. (Do these images use some form of steganography or can we simply work out the graphical transformations applied to the original image and apply them in reverse to obtain the desired image? I wonder.)

"Superstition Reigns"

"Superstition Reigns" by Rahul Singh, published in The Times of India today:

"Amitabh Bachchan, an icon for tens of millions of Indians, makes his daughter-in-law [Aishwarya Rai] perform outlandish ceremonies because she is supposedly under the evil influence of Mars. Politicians routinely consult astrologers before taking important decisions, despite abundant proof that astrology is no science at all, just quackery. Horoscopes continue to be cast in most families and palmists consulted. A newly-inducted cabinet minister insists that her bungalow be completely redesigned because it does not follow vaastu principles, a system nobody had heard of till only a few years ago."

Superstition in all its ugly forms is sickeningly pervasive in India, even among educated people who ought to know better. We waste a lot of time and money and unnecessarily make life difficult for ourselves as well as others, all in the name of something that doesn't withstand rational scrutiny.

Firefox 3 and Linux

Mike Connor blogs about the proposed requirements for Firefox 3 to run on Linux. A nasty surprise for me was the inclusion of GNOME as a dependency. While the GTK/Pango/Cairo/etc. requirements are quite understandable, I don't understand why it should need GNOME. Many of us are happy with KDE or Xfce and would like to avoid the bloat and the dependency hell of GNOME for the sake of running a browser.

As an aside, Firefox on Linux also seems to behave quite differently from Firefox on Windows. For example, on Linux Firefox seems to consistently consume more CPU time and memory than on Windows. Some pages are rendered differently on Windows and Linux (perhaps due to the availability, or otherwise, of the fonts requested by the page designer and the rendering infrastructure). I have personally also noticed bug-337093 on Windows but not on Linux.

VU3RDD Gets $2.56 From Donald Knuth

VU3RDD (a.k.a. Ramakrishnan Muthukrishnan) recently received a cheque for $2.56 from Donald Knuth as a reward for spotting a mistake in TAoCP Volume 2. Among the people I have met, he is the first such person. Congratulations!

Mohan Embar

I met Mohan Embar this weekend. He used to maintain the Windows port of GCJ. It was nice to finally be able to associate a face and a voice with the name, since our interaction so far had only been over email. He turned out to be much thinner, more soft-spoken and more boyish than I had imagined.

I think I ended up asking him a bit too much about how he managed to remain a freelance programmer for so long, that too in Milwaukee, since it is something that interests (as well as scares) me.

Wayanad

The Labour Day holiday last Tuesday, combined with a day's leave off work on Monday, offered us a four day long weekend that Anusha and I used for a mini vacation in Wayanad in Kerala.

A couple of Anusha's friends, along with their spouses, also joined us on this trip. We drove from Bangalore to Wayanad via Mysore, Nanjangud, Gundlupet and Sultan Battery, taking SH-17 and NH-212 and passing through the Bandipur wildlife sanctuary. The roads were quite good in general and the road from Bangalore to Mysore was excellent in particular. We covered the distance in about 5.5 hours, including a couple of short breaks. We stayed in Edakkal Hermitage, a resort quite close to the Edakkal Caves.

Pre-historic Carvings of the Edakkal Caves

Kerala is one of the most beautiful states in India. Even for someone from a generally green city like Bangalore, the lush and pervasive greenery of Kerala is an exhilarating change. As with The Tall Trees Resorts in Munnar that we had visited earlier, the cottages of Edakkal Hermitage were located on the quiet slopes of a mountain in a way that allows one to soak in the beauty of nature in relative privacy while affording a fantastic view of the valley below. The two resorts were also very similar in the amazing service provided by the respective staff and the delicious food that were served by them.

Our Cottage

On the first day, after having refreshed ourselves and having had lunch at the resort, we drove down to the Pookote Lake. This lake turned out to be a disappointment. It was small, filthy and full of tourists. We didn't stay there long, moving on to a view-point and then returning to our resort. Later in the evening, we had our dinner in a cave in the resort that was beautifully lit by more than a hundred candles.

The next morning, we visited the Edakkal Caves and admired the pre-historic carvings on the walls of the caves. The climb to the caves is a bit difficult and is not everyone's cup of tea. Three of our party, including me, wanted to climb further up the mountain and on to the summit, while the others preferred to stay back on a landing waiting for us to come back. We kept climbing up till we reached a rock-face that was a bit steep. There was a single rope for support and not many footholds. Not being experienced climbers and only having our ordinary shoes for support, we chickened out. We tried to find an alternate route to the summit and turned back on not finding any. I regret this now and wish I had mustered the courage.

Our only consolation was spotting a huge butterfly on the way back. It was the biggest butterfly I had ever seen and must have been about 20 to 25 centimetres (8 to 10 inches) across. We were able to get very close to the butterfly and even touch it - it just moved its wings and continued to sit on its tree.

The Butterfly

Tired from our trek, Anusha and I chose to relax in our cottage that afternoon while the rest of the party drove to the Suchipara waterfalls. In the evening we had our dinner in an amphitheatre in the resort.

The next morning we went for a safari through the Muthanga wildlife sanctuary. The safari was utterly disappointing for the most part with not many animals in sight (as has been our luck on all such occasions), when it suddenly turned rather interesting towards the end. Our way was blocked by a herd of at least seven elephants, one of which was a baby elephant. The driver of the Jeep was evidently quite scared and was ready to scoot at the slightest hint of trouble. Retreating was a bit difficult since the path was rather narrow (you had to either retreat in reverse gear or find a clearing large enough to turn the Jeep around) and there were other Jeeps behind us. We had to wait for about half an hour before the herd moved away and we could proceed. Other than the elephants, we were able to spot a Malabar Squirrel, different types of deer, langurs and peacocks.

The Elephants

That afternoon we headed back to Bangalore, regretting that we could not stay longer and dreading the plunge back into the daily grind of our lives.

Running Java Applets in Internet Explorer on Linux

One of the unfortunate things about the current state of the internet is that some web sites refuse to work with anything other than Internet Explorer (IE). Some of these also require you to run Adobe Flash Player and/or Sun's JVM within IE. Most of these sites can be happily avoided, but some of them just can't, especially when they run important applications within a corporate intranet. This can seriously dampen the enthusiasm of people willing to try out Linux as their primary desktop.

WINE allows you to run many a Windows application natively on Linux, including IE (albeit with a few tweaks). IEs4Linux makes it really simple to install one or more versions of IE on your Linux system, something that is very difficult on Windows itself, if not impossible! You can also view Flash content and run Java applets within such an IE. The latter requires a bit of tweaking with the current release of WINE (0.9.34), if you want to use Sun's JVM instead of that provided by Microsoft, as explained below.

Install a version of IE using IEs4Linux into, say, $HOME/.ies4linux (the default). Assuming that you choose to install only IE 6.0 SP1, IEs4Linux will create a WINE "bottle" named "ie6" within "$HOME/.ies4linux", separate from your regular WINE bottle (which is present by default in "$HOME/.wine"). IEs4Linux can also automatically install Adobe Flash Player along with IE. Run IE at least once to verify that it is working.

Now install the Java Runtime Environment (JRE) making sure that you correctly specify the WINEPREFIX environment variable by pointing it to the IE WINE bottle. For example:

export WINEPREFIX=$HOME/.ies4linux/ie6
wine jre-1_5_0_11-windows-i586-p.exe

For some reason, RegUtils.dll is not correctly copied during the installation of the JRE and therefore you must copy this file from a Windows machine that has exactly the same version of the JRE. This file is usually found in the "bin" sub-folder of the JRE installation folder. Without this file, the Java Control Panel applet will not be able to register Sun's JRE with IE.

Now run the "javacpl" programme found in the "bin" sub-folder of the JRE installation folder. With the current WINE release, this would cause your display to flicker or black out since it does not yet fully support DirectX-based acceleration (but where the maximum development effort currently seems to be directed). To avoid this, you can also invoke the Java Control Panel applet alternatively like this in the "lib" sub-folder of the JRE installation folder:

java -classpath deploy.jar \
  -Dsun.java2d.noddraw com.sun.deploy.panel.ControlPanel

Go to the "Advanced" tab and uncheck the check-box for "<APPLET> tag support" for "Internet Explorer", apply your changes and close the applet. Restart the applet once again and this time check the check-box, apply your changes and close the applet. You should now be able to see Java applets within IE using Sun's JRE.

If you wish to avoid the flicker/blacking-out of the display whenever you run Java GUI applications, you can either pass the JVM option -Dsun.java2d.noddraw to Java applications and applets or disable DirectX-based acceleration for Java 2D completely by looking for a registry key like:

HKEY_CURRENT_USER\Software\JavaSoft\Java2D\1.5.0_11

and setting the value of "DXAcceleration" to "0". (WINE includes a "regedit" programme just like its Windows counterpart.)

With IE, Flash and Java Applets with you, you are now ready to savour the worst of the internet first-hand on your Linux desktop instead of hearing about it from your friends who use Windows.

(The method outlined here seems to work with WINE 0.9.34 on Fedora Core 6, IE 6.0 SP1 as installed by IEs4Linux 2.0.5 and Sun's JRE 1.5.0_11 - your mileage might vary.)

Update (2007-07-03): With WINE 0.9.40 on Fedora 7 and Sun's JRE 1.5.0_12, I don't see the problem with "RegUtils.dll" and the JRE installs just fine. Another way of avoiding the blackening of the entire desktop while using Java Swing applications (and for getting a much more accurate display) is to enable a "virtual desktop" that will hold your Windows applications. To do this, invoke "winecfg", select "Enable a virtual desktop" under the "Graphics" tab and provide a size for the virtual desktop (say, 800 by 600 or 1024 by 768).

Update (2007-10-31): The default location used by IEs4Linux is $HOME/.ies4linux (notice the period in front of the directory name). Changed the post to use this location instead as people were getting confused by the location used earlier.