Homesick in Australia

I'm in Sydney for a couple of weeks on work...

So here are some of the pictures of family and friends I put together in my erm..copious free time!

Family and Friends

Picasa and Blogger

Something seems to have completely broken down with Picasa and Blogger and I can't post pictures as easily as I used to...

Seems like a lot of people are having this problem... here and here and here are some similarly disgruntled people..

Arrgh..

So the switch to my own WordPress server may be required after all..

A Church Blog???

I've volunteered to help set up a website for the church I go to (Richmond Town Methodist Church), and it struck me that most of what they wanted could actually be done with a blog.

Off to google-land, and I came across this wonderful blog entry that talked about setting up our own blog server...

Hmm...maybe this will work...

"Can't Touch This"

Very interesting video and article I caught this morning - thanks Greg!

Fascinating what the human mind comes up with!

Enjoy

Coorg Trip

All pictures taken with the Nokia N70 camera-phone

Coorg is a wonderful, picturesque place on the western side of Karnataka. Known for lush vegetation, brisk weather, fantastic arabica and robusta coffee & friendly, cheerful people, Coorg is definitely one of the best places to be for a relaxing weekend getaway.

We stayed at a holiday resort near Kakabe, from where a daredevil 4x4 Jeep ride took us to the top of a local hill. The scenery around was stunning as we watched the clouds try to crest the peak in vain...

Into the Wild Blue XML

So in the previous entry, I posited a hypothetical template-based, metadata-driven, code generation system with one shortcoming - there was no concrete way to perform the application of the template onto the data.

In this article, I'm going to describe an approach we have developed to perform this operation.

The template we ended up with last time was:

template :=
 [PATTERN_BEGIN(property : property_metadata)]
                 do
                 {
                     try
                         {
                         cmd.Parameters.Add("@p_[property.name]", [property.type]);
  [IF property.is_mandatory]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
  [ELSE]
   [IF property.is_reference]
                         if (this.[property.name] == null)
                         {
                             cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;
                         }
                         else
                         {
                             cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
                         }
   [ELSE]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
   [ENDIF]
                     }
  [ENDIF]
                     catch (Exception ex)
                     {
  [IF property.is_mandatory]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Bailing.", ex.Message));
                         throw ex;
  [ELSE]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                         cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;

                         break;
  [ENDIF]
                     }
                 }
                 while(false);
 [PATTERN_END]

Also, the data used to parametrize the template was:

data := {
  {"ID",       "System.Data.SqlDbType.UniqueIdentifier", true,  false},
  {"IsActive", "System.Data.SqlDbType.Bit",              false, false},
  {"Name",     "System.Data.SqlDbType.NVarChar",         false, true}
 }

One way of looking at this snippet is to realise is that the invariant portion of the code (the non-italicised bits) is actually the output we want. It follows that if the non-italicised bits are the output, then the italicised portion must be the program, in some hitherto undescribed language, which when processed by some hitherto unspecified compiler, will generate the output.

If we want to take this approach, and we don't want to be saddled with the task of creating a new language and writing a full-fledged compiler, then it behooves us to look at what languages and tools are already out there to be able to acheive what we want without going through all the motions.

Epiphany #1

As a side note, it is very interesting to note that the output of an XSLT transformation of an XML document can be formatted text! All you have to do to achieve this is to appropriately set up the <xsl:output/> tag within the stylesheet.

So, if somehow, we could represent our data in an XML document, and have the template as an XSLT transform, we could then apply the transform to the data and emit text.

The reality, of course, is that it's fairly straightforward to represent both the data and the metadata in XML. In fact, we already have a mechanism to describe the metadata using XSD, which is itself an XML document. We won't appreciate the ramifications of this observation just yet, but all you need to do for now is to keep in mind that both the data and its metadata can be stored in XML format.

Application

Consider the following XML snippet which would be a reasonable XML representation of the data:

<data>
 <property name="ID" type="System.Data.SqlDbType.UniqueIdentifier" is_mandatory="true" is_reference="false"/>
 <property name="IsActive" type="System.Data.SqlDbType.Bit" is_mandatory="false" is_reference="false"/>
 <property name="Name" type="System.Data.SqlDbType.NVarChar" is_mandatory="false" is_reference="true"/>
</data>

And consider the following XSLT transform:

<xsl:transform match="data/property">
 <xsl:value-of select="concat('if (this.', @Name, ' == null')"/>
 <xsl:value-of select="'{'"/>
 <xsl:value-of select="'}'"/>
 <xsl:value-of select="'else'"/>
 <xsl:value-of select="'{'"/>
 <xsl:value-of select="'}'"/>
</xsl:transform>

If we apply this transform to the data xml, we would generate the following snippet:

if (this.ID == null)
{
}
else
{
}
if (this.IsActive == null)
{
}
else
{
}
if (this.Name == null)
{
}
else
{
}

Interesting.

Three observations can be immediately made now:

1. We have something which seems to give us a first approximation of what we want! This positive observation is very important!
   
2. XSLT has control structures like <xsl:for-each> and <xsl:if> which means we seem to have a control language and its processing tools. This is also a positive observation.
   
3. It's a right pain-in-the-patootie to write the XSLT by hand. This negative observation is critically important.

In fact, if we wanted to write the whole XSLT for the whole class by hand, it would be a laborious and time-consuming procedure. We wrote all of the v1 patterns of Designer this way, and believe us - it was frustrating and difficult work. Emitting apostrophes, quotes, ampersands and getting the formatting right are all singularly difficult, as is debugging any one of a dozen kind of problems that may crop up. Besides, all those <'s and >'s were starting to look a lot like distorted Lisp parens!

(The similarity with Lisp is a very interesting coincidence. Lisp programmers think nothing of generating Lisp functions at runtime and executing them in-situ. In fact, a functional programming language like Lisp would make generation of heirarchical data structures like XML snippets very easy. More on this later.)

I'm lazy, remember? Catch me trying to write XSLT by hand longer than I need to!

Watch out for the next article where I'll discuss an approach to generate the XSLT itself!

In the meanwhile, recap what we discovered here. We can use XML and XSLT as the basis of our code generation approach, as long as we don't have to write the XSLT by hand.

Laziness Rocks!

The Code-Generation Mindset

I'm a lazy guy.

I hate doing the same kind of thing over and over again. That's what computers are good at. So the cool thing to do is to try and get the computer to do the grunge work of writing a program, leaving me to do interesting things like designing the application properly...

There is nothing new about Code-Generation. Compilers have been doing it for years, consuming simpler syntax and higher-level abstractions and churning out instructions native to the execution platform. What's more, programming languages and their compilers have become more intelligent and automatically perform significant optimizations so that an expert assembly programmer (a dying, if not extinct, breed) would find it hard to improve upon the code generated at the instruction-set level.

And this was old news in 1998...

However, almost ten years on, we're still writing code...

Granted, a lot of the code written in the industry now is in scripting languages, which can be argued to be at a higher level of abstraction than is system or even application programming. However, the same principle holds even now - as is evidenced by the kind of program required to connect to a database table and generate a table of its contents to be viewed on a web page.

"Hang on", you say, "that was a big deal in 2004 - before tools like Visual Studio.NET 2005 came along with their point-and-click wizards which generated all the code for that kind of thing"...

...and you'd be right - that's exactly what I'm saying too. In fact, by the time I'm done with this topic, we'll build the exact pattern (down to tabs, semicolons and spelling mistakes) that Visual Studio.NET 2005 uses in its TypedDataset generator tool. (If that, by itself, isn't remarkable, consider the fact that it took me less than 2 days to write my pattern!)

What would be nice is if I could apply the code-generation paradigm to any repetitive, semi-mindless pattern of code. The way we currently deal with this now is either:

1. Cut/Paste/Edit the pattern to fit the requirement. Pray and compile/deploy hoping that the editing was comprehensively correct. Too many problems to list with this approach.
   
2. Make the pattern into a soulless library which boils everything to the lowest common denominator and adds a pattern to calling it. Generally not a huge gain there, plus added complexity.
   
3. Use a functional programming language with functions as arguments to the variable bits of the pattern. Maps, Filters and Folds work really well for some kinds of patterns that iterate over a set of items. More on this much later.
   
4. Code-Generate. If the pattern is repeated a lot, this is the most sensible approach if you have the tools to do this. I'm going to try and expound on this in a little detail now.

By the way, let me plug my company here - I founded, and currently work in, BrightSword Technologies Private Limited, and I am based in Bangalore, India. We recognised the value of this approach a long time ago, and built a code-generation framework which we have successfully used in a variety of real-world applications. We also took the data-driven web-application as a pattern, and wrote and sold a standalone code-generation tool called BrightSword Designer way back in 2002. We even had the point-and-click wizards for ASP, JSP, PHP and ASP.NET long before Visual Studio.NET 2005, and the ASP pattern implemented inheritance and polymorphism using explicit delegation even though VBScript has neither concept fully implemented!

Anyway, back to the mindset...

To successfully apply the Code-Generation approach, two prerequisites must be fulfilled:

1. There must be a recognizable pattern. A pattern of code is basically a chunk of code with a few parts that are invariant, and some variants which change between instances of the pattern.
   
2. We must be able to describe the variant portion meaningfully with respect to the pattern.

Let me try to clarify this with an example. Consider this snippet of code you might write when calling a stored procedure using ADO.NET:

                do
                {
                    try
                    {
                        cmd.Parameters.Add("@p_ID", System.Data.SqlDbType.UniqueIdentifier);

                        cmd.Parameters["@p_ID"].Value = this.ID;
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.ID threw Exception [ {0} ]. Bailing.", ex.Message));
                        throw ex;
                    }
                }
                while(false);

                do
                {
                    try
                    {
                        cmd.Parameters.Add("@p_IsActive", System.Data.SqlDbType.Bit);

                        cmd.Parameters["@p_IsActive"].Value = this.IsActive;
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.IsActive threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                        cmd.Parameters["@p_IsActive"].Value = System.DBNull.Value;

                        break;
                    }
                }
                while(false);

                do
                {
                    try
                    {
                        cmd.Parameters.Add("@p_Name", System.Data.SqlDbType.NVarChar);

                        if (this.Name == null)
                        {
                            cmd.Parameters["@p_Name"].Value = System.DBNull.Value;
                        }
                        else
                        {
                            cmd.Parameters["@p_Name"].Value = this.Name;
                        }
                    }
                    catch (Exception ex)
                    {
                        System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.Name threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                        cmd.Parameters["@p_Name"].Value = System.DBNull.Value;

                        break;
                    }
                }
                while(false);

  // mumble the incantation to execute the cmd on the connection

As a digression, notice that this kind of code contains the kind of bullet-proofing and optimization that you would likely want to put into your code, but can't or won't because it's too tedious. It's far simpler to write the usual code given in all the demos, and leave things to chance and the gods:

                cmd.Parameters.Add("@p_ID", this.ID);
                cmd.Parameters.Add("@p_IsActive", this.IsActive);
                cmd.Parameters.Add("@p_Name", this.Name);

  // mumble the incantation to execute the cmd on the connection

Anyway - let's identify the pattern in the code.

It's basically the piece of code between the "do" and the "while(false);". (In reality, the entire class is the pattern, but let's keep things simple for the example.)

The variant portion of the pattern takes a little recognition. We recognise the obvious:

• the name of the parameter
  
• the type of the parameter

Not so obvious are the following:

• whether the parameter is mandatory or optional
  
• whether the property of the class is a reference or value type

So in a formal way, we can say that the metadata (or "type") of the variant is:

property_metadata ::= {name : string, type : string, is_mandatory : bool, is_reference : bool}  

If we assume a kind of pseudocode notation, we could write

template :=
 [PATTERN_BEGIN(property : property_metadata)]
                 do
                 {
                     try
                         {
                         cmd.Parameters.Add("@p_[property.name]", [property.type]);
  [IF property.is_mandatory]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
  [ELSE]
   [IF property.is_reference]
                         if (this.[property.name] == null)
                         {
                             cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;
                         }
                         else
                         {
                             cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
                         }
   [ELSE]
                         cmd.Parameters["@p_[property.name]"].Value = this.[property.name];
   [ENDIF]
                     }
  [ENDIF]
                     catch (Exception ex)
                     {
  [IF property.is_mandatory]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Bailing.", ex.Message));
                         throw ex;
  [ELSE]
                         System.Diagnostics.Debug.WriteLine(String.Format("Customer.Persist - Persisting Customer.[property.name] threw Exception [ {0} ]. Filling Null Value.", ex.Message));
                         cmd.Parameters["@p_[property.name]"].Value = System.DBNull.Value;

                         break;
  [ENDIF]
                     }
                 }
                 while(false);
 [PATTERN_END]

Now, we can enumerate the parametrizing entities:

data := {
  {"ID",       "System.Data.SqlDbType.UniqueIdentifier", true,  false},
  {"IsActive", "System.Data.SqlDbType.Bit",              false, false},
  {"Name",     "System.Data.SqlDbType.NVarChar",         false, true}
 }

It's pretty obvious that with this kind of a setup, we could make the assertion that we could generate the desired block of by somehow applying the data onto the template. We simply need some magic wand to perform the desired apply operation. In pseudo-functional notation,

code := apply(template, data)

My primary assertion in this article is that recognizing patterns and following the metadata-driven approach will lead to cleaner and better code. Thinking this way allows us to worry about bullet-proofing and optimizing our code, because the time invested in strengthening the template of the pattern is miniscule compared to the time required to scan and change all occurrences of the pattern. We could extend and strengthen the code by simply modifying the template, or more radically by extending the metadata and template together. One bug fixed or one optimization baked into the template gets multiplied into the generated code. Big Hammer!

Of course, there is the small matter of performing the apply operation to somehow take both template and data to give us code - but you'll have to stay tuned for that!

Food for thought, no?

Well! Bon Appetit!!