Goodbye 2007

It's been a crazy year...

Homesick in Australia

I'm in Sydney for a couple of weeks on work...

So here are some of the pictures of family and friends I put together in my erm..copious free time!

Family and Friends

Picasa and Blogger

Something seems to have completely broken down with Picasa and Blogger and I can't post pictures as easily as I used to...

Seems like a lot of people are having this problem... here and here and here are some similarly disgruntled people..

Arrgh..

So the switch to my own WordPress server may be required after all..

A Church Blog???

I've volunteered to help set up a website for the church I go to (Richmond Town Methodist Church), and it struck me that most of what they wanted could actually be done with a blog.

Off to google-land, and I came across this wonderful blog entry that talked about setting up our own blog server...

Hmm...maybe this will work...

"Can't Touch This"

Very interesting video and article I caught this morning - thanks Greg!

Fascinating what the human mind comes up with!

Enjoy

Coorg Trip

All pictures taken with the Nokia N70 camera-phone

Coorg is a wonderful, picturesque place on the western side of Karnataka. Known for lush vegetation, brisk weather, fantastic arabica and robusta coffee & friendly, cheerful people, Coorg is definitely one of the best places to be for a relaxing weekend getaway.

We stayed at a holiday resort near Kakabe, from where a daredevil 4x4 Jeep ride took us to the top of a local hill. The scenery around was stunning as we watched the clouds try to crest the peak in vain...

Into the Wild Blue XML

So in the previous entry, I posited a hypothetical template-based, metadata-driven, code generation system with one shortcoming - there was no concrete way to perform the application of the template onto the data.

In this article, I'm going to describe an approach we have developed to perform this operation.

The template we ended up with last time was:

template :=
 [PATTERN_BEGIN(property : property_metadata)]
                 do
                 {
                     try
                         {
                         cmd.Parameters.Add("@p_[property.name]", [property.type]);
  [IF property.is_mandatory]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
  [ELSE]
   [IF property.is_reference]
                         if (this.[property.name] == null)
                         {
                             cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;
                         }
                         else
                         {
                             cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
                         }
   [ELSE]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
   [ENDIF]
                     }
  [ENDIF]
                     catch (Exception ex)
                     {
  [IF property.is_mandatory]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Bailing.", ex.Message));
                         throw ex;
  [ELSE]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                         cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;

                         break;
  [ENDIF]
                     }
                 }
                 while(false);
 [PATTERN_END]

Also, the data used to parametrize the template was:

data := {
  {"ID",       "System.Data.SqlDbType.UniqueIdentifier", true,  false},
  {"IsActive", "System.Data.SqlDbType.Bit",              false, false},
  {"Name",     "System.Data.SqlDbType.NVarChar",         false, true}
 }

One way of looking at this snippet is to realise is that the invariant portion of the code (the non-italicised bits) is actually the output we want. It follows that if the non-italicised bits are the output, then the italicised portion must be the program, in some hitherto undescribed language, which when processed by some hitherto unspecified compiler, will generate the output.

If we want to take this approach, and we don't want to be saddled with the task of creating a new language and writing a full-fledged compiler, then it behooves us to look at what languages and tools are already out there to be able to acheive what we want without going through all the motions.

Epiphany #1

As a side note, it is very interesting to note that the output of an XSLT transformation of an XML document can be formatted text! All you have to do to achieve this is to appropriately set up the <xsl:output/> tag within the stylesheet.

So, if somehow, we could represent our data in an XML document, and have the template as an XSLT transform, we could then apply the transform to the data and emit text.

The reality, of course, is that it's fairly straightforward to represent both the data and the metadata in XML. In fact, we already have a mechanism to describe the metadata using XSD, which is itself an XML document. We won't appreciate the ramifications of this observation just yet, but all you need to do for now is to keep in mind that both the data and its metadata can be stored in XML format.

Application

Consider the following XML snippet which would be a reasonable XML representation of the data:

<data>
 <property name="ID" type="System.Data.SqlDbType.UniqueIdentifier" is_mandatory="true" is_reference="false"/>
 <property name="IsActive" type="System.Data.SqlDbType.Bit" is_mandatory="false" is_reference="false"/>
 <property name="Name" type="System.Data.SqlDbType.NVarChar" is_mandatory="false" is_reference="true"/>
</data>

And consider the following XSLT transform:

<xsl:transform match="data/property">
 <xsl:value-of select="concat('if (this.', @Name, ' == null')"/>
 <xsl:value-of select="'{'"/>
 <xsl:value-of select="'}'"/>
 <xsl:value-of select="'else'"/>
 <xsl:value-of select="'{'"/>
 <xsl:value-of select="'}'"/>
</xsl:transform>

If we apply this transform to the data xml, we would generate the following snippet:

if (this.ID == null)
{
}
else
{
}
if (this.IsActive == null)
{
}
else
{
}
if (this.Name == null)
{
}
else
{
}

Interesting.

Three observations can be immediately made now:

1. We have something which seems to give us a first approximation of what we want! This positive observation is very important!
   
2. XSLT has control structures like <xsl:for-each> and <xsl:if> which means we seem to have a control language and its processing tools. This is also a positive observation.
   
3. It's a right pain-in-the-patootie to write the XSLT by hand. This negative observation is critically important.

In fact, if we wanted to write the whole XSLT for the whole class by hand, it would be a laborious and time-consuming procedure. We wrote all of the v1 patterns of Designer this way, and believe us - it was frustrating and difficult work. Emitting apostrophes, quotes, ampersands and getting the formatting right are all singularly difficult, as is debugging any one of a dozen kind of problems that may crop up. Besides, all those <'s and >'s were starting to look a lot like distorted Lisp parens!

(The similarity with Lisp is a very interesting coincidence. Lisp programmers think nothing of generating Lisp functions at runtime and executing them in-situ. In fact, a functional programming language like Lisp would make generation of heirarchical data structures like XML snippets very easy. More on this later.)

I'm lazy, remember? Catch me trying to write XSLT by hand longer than I need to!

Watch out for the next article where I'll discuss an approach to generate the XSLT itself!

In the meanwhile, recap what we discovered here. We can use XML and XSLT as the basis of our code generation approach, as long as we don't have to write the XSLT by hand.

Laziness Rocks!

The Code-Generation Mindset

I'm a lazy guy.

I hate doing the same kind of thing over and over again. That's what computers are good at. So the cool thing to do is to try and get the computer to do the grunge work of writing a program, leaving me to do interesting things like designing the application properly...

There is nothing new about Code-Generation. Compilers have been doing it for years, consuming simpler syntax and higher-level abstractions and churning out instructions native to the execution platform. What's more, programming languages and their compilers have become more intelligent and automatically perform significant optimizations so that an expert assembly programmer (a dying, if not extinct, breed) would find it hard to improve upon the code generated at the instruction-set level.

And this was old news in 1998...

However, almost ten years on, we're still writing code...

Granted, a lot of the code written in the industry now is in scripting languages, which can be argued to be at a higher level of abstraction than is system or even application programming. However, the same principle holds even now - as is evidenced by the kind of program required to connect to a database table and generate a table of its contents to be viewed on a web page.

"Hang on", you say, "that was a big deal in 2004 - before tools like Visual Studio.NET 2005 came along with their point-and-click wizards which generated all the code for that kind of thing"...

...and you'd be right - that's exactly what I'm saying too. In fact, by the time I'm done with this topic, we'll build the exact pattern (down to tabs, semicolons and spelling mistakes) that Visual Studio.NET 2005 uses in its TypedDataset generator tool. (If that, by itself, isn't remarkable, consider the fact that it took me less than 2 days to write my pattern!)

What would be nice is if I could apply the code-generation paradigm to any repetitive, semi-mindless pattern of code. The way we currently deal with this now is either:

1. Cut/Paste/Edit the pattern to fit the requirement. Pray and compile/deploy hoping that the editing was comprehensively correct. Too many problems to list with this approach.
   
2. Make the pattern into a soulless library which boils everything to the lowest common denominator and adds a pattern to calling it. Generally not a huge gain there, plus added complexity.
   
3. Use a functional programming language with functions as arguments to the variable bits of the pattern. Maps, Filters and Folds work really well for some kinds of patterns that iterate over a set of items. More on this much later.
   
4. Code-Generate. If the pattern is repeated a lot, this is the most sensible approach if you have the tools to do this. I'm going to try and expound on this in a little detail now.

By the way, let me plug my company here - I founded, and currently work in, BrightSword Technologies Private Limited, and I am based in Bangalore, India. We recognised the value of this approach a long time ago, and built a code-generation framework which we have successfully used in a variety of real-world applications. We also took the data-driven web-application as a pattern, and wrote and sold a standalone code-generation tool called BrightSword Designer way back in 2002. We even had the point-and-click wizards for ASP, JSP, PHP and ASP.NET long before Visual Studio.NET 2005, and the ASP pattern implemented inheritance and polymorphism using explicit delegation even though VBScript has neither concept fully implemented!

Anyway, back to the mindset...

To successfully apply the Code-Generation approach, two prerequisites must be fulfilled:

1. There must be a recognizable pattern. A pattern of code is basically a chunk of code with a few parts that are invariant, and some variants which change between instances of the pattern.
   
2. We must be able to describe the variant portion meaningfully with respect to the pattern.

Let me try to clarify this with an example. Consider this snippet of code you might write when calling a stored procedure using ADO.NET:

                do
                {
                    try
                    {
                        cmd.Parameters.Add("@p_ID", System.Data.SqlDbType.UniqueIdentifier);

                        cmd.Parameters["@p_ID"].Value = this.ID;
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.ID threw Exception [ {0} ]. Bailing.", ex.Message));
                        throw ex;
                    }
                }
                while(false);

                do
                {
                    try
                    {
                        cmd.Parameters.Add("@p_IsActive", System.Data.SqlDbType.Bit);

                        cmd.Parameters["@p_IsActive"].Value = this.IsActive;
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.IsActive threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                        cmd.Parameters["@p_IsActive"].Value = System.DBNull.Value;

                        break;
                    }
                }
                while(false);

                do
                {
                    try
                    {
                        cmd.Parameters.Add("@p_Name", System.Data.SqlDbType.NVarChar);

                        if (this.Name == null)
                        {
                            cmd.Parameters["@p_Name"].Value = System.DBNull.Value;
                        }
                        else
                        {
                            cmd.Parameters["@p_Name"].Value = this.Name;
                        }
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.Name threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                        cmd.Parameters["@p_Name"].Value = System.DBNull.Value;

                        break;
                    }
                }
                while(false);

  // mumble the incantation to execute the cmd on the connection

As a digression, notice that this kind of code contains the kind of bullet-proofing and optimization that you would likely want to put into your code, but can't or won't because it's too tedious. It's far simpler to write the usual code given in all the demos, and leave things to chance and the gods:

                cmd.Parameters.Add("@p_ID", this.ID);
                cmd.Parameters.Add("@p_IsActive", this.IsActive);
                cmd.Parameters.Add("@p_Name", this.Name);

  // mumble the incantation to execute the cmd on the connection

Anyway - let's identify the pattern in the code.

It's basically the piece of code between the "do" and the "while(false);". (In reality, the entire class is the pattern, but let's keep things simple for the example.)

The variant portion of the pattern takes a little recognition. We recognise the obvious:

• the name of the parameter
  
• the type of the parameter

Not so obvious are the following:

• whether the parameter is mandatory or optional
  
• whether the property of the class is a reference or value type

So in a formal way, we can say that the metadata (or "type") of the variant is:

property_metadata ::= {name : string, type : string, is_mandatory : bool, is_reference : bool}  

If we assume a kind of pseudocode notation, we could write

template :=
 [PATTERN_BEGIN(property : property_metadata)]
                 do
                 {
                     try
                         {
                         cmd.Parameters.Add("@p_[property.name]", [property.type]);
  [IF property.is_mandatory]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
  [ELSE]
   [IF property.is_reference]
                         if (this.[property.name] == null)
                         {
                             cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;
                         }
                         else
                         {
                             cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
                         }
   [ELSE]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
   [ENDIF]
                     }
  [ENDIF]
                     catch (Exception ex)
                     {
  [IF property.is_mandatory]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Bailing.", ex.Message));
                         throw ex;
  [ELSE]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                         cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;

                         break;
  [ENDIF]
                     }
                 }
                 while(false);
 [PATTERN_END]

Now, we can enumerate the parametrizing entities:

data := {
  {"ID",       "System.Data.SqlDbType.UniqueIdentifier", true,  false},
  {"IsActive", "System.Data.SqlDbType.Bit",              false, false},
  {"Name",     "System.Data.SqlDbType.NVarChar",         false, true}
 }

It's pretty obvious that with this kind of a setup, we could make the assertion that we could generate the desired block of by somehow applying the data onto the template. We simply need some magic wand to perform the desired apply operation. In pseudo-functional notation,

code := apply(template, data)

My primary assertion in this article is that recognizing patterns and following the metadata-driven approach will lead to cleaner and better code. Thinking this way allows us to worry about bullet-proofing and optimizing our code, because the time invested in strengthening the template of the pattern is miniscule compared to the time required to scan and change all occurrences of the pattern. We could extend and strengthen the code by simply modifying the template, or more radically by extending the metadata and template together. One bug fixed or one optimization baked into the template gets multiplied into the generated code. Big Hammer!

Of course, there is the small matter of performing the apply operation to somehow take both template and data to give us code - but you'll have to stay tuned for that!

Food for thought, no?

Well! Bon Appetit!!

Technology Indroduction: My Early Years...

I've been involved in Computer Science and Technology for my entire career, and for a good portion of my education before...

I was interested in electronics and tinkering around with Radios and Tape Players from the time I was in 5th grade. My mum, who worked in Indian Institute of Science, helped me build a 555-based flashing LED for the school science fair when I was in 6th grade. My interest in electronics remains until this day...

The first computer I wrote any software for was a Sinclair ZX Spectrum - while I was still in high school. A couple of friends and I then went on to build our own Zilog Z-80 computer. We wrote programs for these machines in Basic, and using PEEK and POKE to write stuff into memory was considered cutting-edge!

The first computer I owned was a Macintosh Plus - which was the standard-issue computer for the class of '93 at Dartmouth College. The whole concept of event-driven programming and visually appealing user interfaces was a sea-change from the low-level, command-line based, DOS programs I had written until then.

I managed to find a balance between the powerful GUI-driven paradigms of the Mac and the NeXT boxes found in the computing clusters at Dartmouth with the command-line and shell-script heavy environments of the unix accounts given to us at the computer science department, so to this day, it's nice to feel at home in both worlds.

It was here that I made my first contribution to global software - a fix in Emacs, submitted and accepted!

It was at this time I was offered an internship at Microsoft Corporation, and I had the good fortune to work on Excel 4.5 as a developer in the Charting Group. This introduced me to visual debuggers, x86 assembly code, and production-grade software engineering methodology. Very useful exposure...

I then spent a short stint at Oracle Corporation after graduating, in the Oracle Forms development team. This was my first experience using industry-strength VMS and Unix tools, and working in an environment where portability between dozens of environments was concern #1. I must say that I really didn't appreciate the experience I got there until much much later...

My big break was in 1994, when I joined Microsoft Corporation as an employee in the Microsoft Project group. Microsoft Project was one of the coolest, albeit little-known, applications in the Office group, and the team was an extremely talented and hard-working group. In a lot of ways, this was my baptism in fire, and I don't think I had a bigger influence in my career than my team lead and colleagues in those days at Microsoft.

One of the tasks I was assigned was to help sever the connections between the front and back ends of Project, so that the back end could be replaced with a powerful new custom database. This experience had me touching almost all of the grungiest parts of a version 5 application and gave me a real appreciation for solid software architecture - one of my driving passions to this day. The importance of writing clean, maintainable code within a clean, extensible architecture was driven home so forcefully that I cannot imagine working any other way. It is indeed a valuable skill to keep a balance between performance, future extensibility, maintainability and coding efficiency.

In addition to this, this was also where I learned the importance of building and maintaining a proper development environment - I became intimately acquainted with the versioning and build processes, identifying and applying software tools and hardware knowledge.

My first new software project was a project that eventually served as the prototype for SharePoint Portal. The small team that worked on this was on fire to use the collective knowledge to set the foundation for a solid product with a long future, so architectural purity became a driving design guideline. It also introduced me to XML, and gave me a serious primer on database design and usage. Again, I would only appreciate the nuances of what I learned much later...

I then moved back to India for personal reasons, and joined the fledgling Microsoft India R&D Private Limited, where I worked on what was to become Visual J#. Although some of the work I did here resulted in my first US Patent application, it was becoming clear to me that the time had come for me to move on...

In March 2000, I resigned from my post at Microsoft, bringing to a close more than 5 years with the software company. By all accounts, these were enjoyable years, where both skills and friendships were forged, and which set me up for the next part of my career...

More Flora..er..Pictures

It's that time of year again...

January 26 is the Republic Day of India, and in Bangalore, where I live, the local botanical garden puts up a massive flower show.

I've not been to the show since I was a kid - and it's mayhem out there on the 26th...so Mum, Smitha and I went out there on the first weekend following...

Smitha wanted to learn how to use the 30D, and I, not having much of a clue myself, thought I'd try a few shots as well.

We shot almost exclusively with the 75-300 lens. The camera was mostly on High-speed Multi-shot mode, and the lens got a chance to show off the ability to take technically difficult shots of flying bees and beetles...in focus, natch!

The gardens were still crowded, and the amount of litter everywhere was appalling - yet another indictment of what people who keep their houses spotless will do in public...

I've uploaded the pictures with reduced resolution to save space. Track back to me if you want any picture in full resolution and I'll do my best to e-mail it to you :)

Enjoy!!

Flora

Flora, Fauna and other Wildlife

Here are some of the pictures taken on the Ooty trip with the 30D.

The picture quality in some cases has been compromised because I aggressively compressed them on the camera. I need to get a 4GB card (or two) to allow me to take the pictures RAW.

The kit as it stands now is the 30D body, Canon EFS 18-55 and Canon EFS 75-300 IS.

The pictures have been uploaded without the corrections proffered by Picasa. So there! :)

Enjoy

Flora and Fauna

The Ooty Trip: Jan 15-17 2007

Smitha!

Smitha, my wife, is my favourite photography subject. Usually she acquiesces to my insane demands to "stand here...look there...there...THERE!!!" by gritting her teeth - which makes her smile look wonderful on camera!
She was sitting around while I was fumbling with the 30D the moment I got it, and I managed to squeeze off this spontaneous portrait.

Canon 30D, 70-300mm IS, flash

November 7-10 2006: Sydney, Australia

Sydney Opera House

All these pictures were taken with Smitha's handy Creative DiViCam. This was taken from a boat as we took a tour of the Sydney Harbour.

The Coat Hanger

Smitha and I in front of the famous Sydney Harbour Bridge. Someday we'll go on the Bridge Climb!

Pipe Organ, Sydney Opera House

A friend works in the Sydney Opera House so we got a bit of a tour of the inside. This is the pipe organ in the main hall.

Gimme..Gimme..Gimme..

So...

I feel more and more like a Luddite these days...

I still have a laptop which weighs about half a ton, is made of pig-iron and runs on coal and steam...

I keep thinking that someday I need to get a new one, if only to retire the mule that hauls it around for me, and I keep looking for that new feature that will finally make me give it all up and buy a new one...

It's a Dell Inspiron 8200 more or less gifted to me by a good friend - and against all odds, it's survived around 3 years with me. I love its 15" screen with 1600 x 1200 resolution - makes it easy to see a reasonable block of code in Emacs or Visual Studio.

So I lied - it was a cutting edge machine in its day...

The real problem is that while it's bulky and really does give my arms a workout carrying it, it is perfectly sufficient for my use - and I do actually write code and stuff on it :)

What I actually noticed is that over the last few months, my travel kit has grown to prodigious proportions with power supplies and adapters of all shapes and sizes, so my next purchases will all try to minimize the number of wall-warts and bricks I need to carry around to stay powered up...

• One laptop from the mesazoic era with its brick...
• Pair of simple headphones and mic for Skype...
• Mobile phone and its wart...
• Bluetooth headset - ok..so I'm not a Luddite after all - and its extension. The extension plugs into the phone's wart so it's not so bad...
• A small point-and-shoot digital camera (for undercover paparazzi work :)) and its charger
• USB Multi-card reader...
• assorted cables...
• assorted adapters to deal with international power socket form factors...
• A small USB web-cam..

Now this is after I got rid of

• The PDA and its charger - my phone does a reasonable approximation thereof
• The CD-player and its charger - my phone plays music in Offline mode, and I can always boot up the laptop...

And I didn't buy an iPod for roughly the same reasons :)

What would be lovely is if the laptop:

• got a lot smaller (yes..that's within reach) and had battery power to last an 8 hour flight (that's OK too)
• could play music without having to have it open and turned on and taking up my tray-table on a flight (the newer HPs do just this - score! :)) and
• have a fast-enough startup time to actually use as a PDA to check flight schedules and things (this should be easy...my antediluvian Dell comes out of "Hibernate" in 12 seconds flat...)

Now...

...if only the laptop could also act as my mobile and use/charge my headset, I could get rid of having to carry my phone on my business trips - my laptop could just be in my backpack as always and I could take calls on the wired or the bluetooth headsets...or heck - just get rid of the wired headset and buy a bluetooth noise-cancelling headband with mic. (Yep - I really was lying about being a Luddite!)

This shouldn't be too difficult to do - the newer cars already have the ability to accept SIM cards and behave like telephones - muting the music and routing the call to the audio system of the car - so why not on a laptop?

So that's what its going to take for me to buy a new laptop...

Goodbye Old Friend...

I just upgraded to a Canon 30D from a Minolta 800si film SLR.

I've shot a lot of stuff with the 800si - bought it before a trip to Bryce Canyon in the US - but time and technology move on, and after a lot of compromising (mostly in the form of Digital Point and Shoots), and after a trip to beautiful, picturesque New Zealand with nothing but my Nokia N70 camera-phone, I finally chanced upon an opportunity to get a Canon 30D.

We went to the Himalayan foothills for the New Year and I shot a couple of rolls with the 800si before finally switching. I think the film roll is finally going the way of the polaroid...

Took the 30D on a trip to Ootcamund - a hill station close-by - and shot a bunch of water-birds in a bird sanctuary. The machine feels good. I have to still get used to the spot-metering and the AF, but it was a good first outing.

I'll post a few of the pictures as I get the time...

Arrrghhhhhhhh...phewwww

Well..I'm finally here.

What I really want to do is have my other blog's posts moved to this one so I can use my johnazariah.blogspot.com address as my main blog...

Can't figure out if it can be done...

Oh well..let me start with this address then!