In the Trenches -
Software Engineering in the Real World

The Myth of Drag-and-Drop and Template-Based Flexibility

I once had a boss who decided to purchase some EDI (electronic data interchange) software to hopefully speed up EDI data conversion development. The company had hundreds of potential EDI-enabled suppliers. I am sure the salesperson talked up a storm. I told him that I did not think dedicated EDI software would help much.

The problem with EDI conversions is that it is not the portion of the project that fits the EDI standard, but the portion that is outside the standard. For example, our company tended to put bill-related comments in the invoice detail lines. It would give a dummy product that was zero dollars where the description was really a comment, or part of a comment. (This was the only way known to have comments near related line-item detail in some systems.)

Although this is a little bit hokey, there is nothing in the official EDI specification that says all prices have to be greater than zero. Thus, we did not violate the EDI specification.

However, one company that received our EDI invoices by chance could not handle zero amounts for line items. So, we looked at the EDI specification and found that a comment field was defined, but that there was only one comment field per invoice. (Actually we were working with a subset of the EDI spec, but that is another story.)

So, I made a little utility to concatenate (combine) all the comment detail lines and make them into one big comment. We then sent over a test invoice.

Well, the test invoice was rejected because their system could not handle comments beyond a certain size. The EDI specification did not give any size limit; it was an internal limit on their part. And, they had no intention of enlarging their internal limit just to accommodate us.

After kicking around ideas for a few days with the other party, it was finally agreed that we would filter out certain comments to make the total comment be a size acceptable to their system.

This filtering was very specific to their system and their business. It is very unlikely that another trading partner would have exactly the same size-limit and filtering rules.

It involved not only filtering out a subset of classification codes, but also looking for certain sub-strings (text patterns). Rough example:

  classif = lineItem.classif
  includeIt = false    // initialize
  if classif in (33,37,42,58) _
         and lineItem.price = 0 then
     // we want only *some* of classif. 37
     if classif = 37 then
        if contains(lineItem.descript, "service")
           includeIt = true
        end if
     else
        includeIt = true
     end if
  end if
  if includeIt then
     appendToComment(lineItem.descript)
  end if

Perhaps some of the EDI stuff can be handled using a drag-and-drop icon interface or a template management system, but the more complex the logic gets, the more a programming language is better suited to the task. The simple, predictable stuff is easy to program anyhow and the hard stuff requires a programming language. In other words, the bottleneck of using a programming language for EDI is not the stuff that the new EDI widget has built-in.

Mouse-intensive tools tend to only make only graphical or visual tasks easier in my opinion; but, EDI is not graphical. It is ugly text and/or binary bytes. Thus, the new tool is not likely to be any better than a regular business programming language. A programming language is more general-purpose than a dedicated EDI tool because the programmer can make his/her program do almost anything he/she wants, whereas the EDI tool is built with pre-conceived notions about what buyers will encounter. If the vendor's crystal ball is not perfect, then the buyer takes a fall.

Such tools may make handling the known, typical patterns a little bit easier (often depending on the skills of the programmer being compared), but the unpredictable stuff they often make much harder. In other words, the tasks that they simplify do not make up for the tasks that they make harder.

I roughly estimate that such a tool would make the normal stuff take one-forth less time (0.75), but make the tricky, unanticipated stuff about 3 times as hard. For example, suppose a typical EDI conversion involved 5 hours dealing with run-of-the mill conversions and 5 hours dealing with tricky, messy conversion rules using a regular programming language. Using our rough numbers, the vendor's tool would then require almost 19 hours (3.75 + 15) instead of 10. In short they make the easy things a little bit easier, but the harder things much harder.

And, if the tool does have a built-in programming language to help with the tricky stuff, then why add yet another language to the IT shop? That just makes more languages for new or replacement programmers to learn.

Another thing to note is that much of the time spent is communicating with, negotiating with, and waiting on the other party to make decisions. This time will be consumed even with an EDI tool. Management did not seem to fully realize this, instead viewing the delays as mostly a lack of our knowledge about EDI itself.

Keep this in mind whenever a vendor tries to sell you or promote the ultimate development automation tool. His/her fancy demo only demonstrates stuff that the tool builders planned for, and of course, a salesperson is going to avoid an unplanned demo. (You may want to ask him/her to demo a few trickier ones from your own past.)

XML in some ways seems to be falling victim to the same kind of hype. The idea that applications can be built from static templates is a very old and very appealing idea. The problem is that dealing with the stuff that a static template like XML cannot handle is going to take more effort than any savings on the typical, foreseen stuff.

XML may offer some incremental productivity value because it can represent certain data patterns a bit easier than earlier data formats could, but it is nothing revolutionary. It is simply another format among many, such as SDF, comma-delimited, DIF, etc. Nested layouts are probably where XML will shine the most.

For example, I once was building a web-page where the color did not seem to match the colors chosen by the graphics department. I verified that it matched using a screen "print" and some color tools. However, it was noticed that the color area in question was next to a very dark area, and thus appeared lighter, even though technically it wasn't. I decided to add a custom fudge that roughly resembled:

  if pageID=5 and section=6 then
     factor = 0.95     // darkening factor
     red   = red   * factor  
     green = green * factor
     blue  = blue  * factor
     // or like "red *= factor" in some langs.
  end if
  ....

One may argue that one should be able to privide a custom template for that one page or one section. However, even that has problems. If the whole color scheme changes, then the slightly-modified copy will reflect the outdated color scheme. Not only does the above approach have a finer granularity, but it will also scale the new color regardless of what that color is. (It is not perfect, but still "smarter" than a template.)

One place where a template or mouse-centric approach may perhaps be worth it is a shop that is too small to have programmers or where there is no budget for more programmers and there is a backlog of needed changes. Templates are easier for say a system administrator or knowledgeable computer operator to deal with. They may not have the experience to use a full-blown programming language appropriately. Thus, templates and drag-and-drop interfaces may be the best way to go there because the ability to control what the templates allow is better than no control at all.

However, templates are not more flexible than programming. Nor are they going to be a significant time-saver in the hands of a skilled programmer. A skilled programmer will know how to "factor" information such that it is not un-necessarily replicated, for example.

Another place where this pattern is trying to push itself again is in ASP's (Application Service Providers, not to be confused with Microsoft's Web scripting system called Active Server Pages.)

ASP vendors promise an instant system that can be "customized" for your needs via filling in some anticipated settings. Perhaps ASP's might be a nice idea for standardized tasks that don't vary much from business-to-business, such as human resources, payroll, public accounting, office supply procurement, etc.

Some large accounting and manufacturing systems use "gobs" of templates in an attempt to provide the needed control. However, managing the settings and templates is so complex that it requires dedicated experts in those particular systems. The jury is still out on whether this approach is better than custom programming or purchased code (below). Some companies swear by them and some swear at them.

1. You can make any desired changes to the source (programming) code. Thus, you don't have to wait for a vendor to make you a custom copy, which they usually won't because it is hard to merge general changes and additions with custom changes down the road.

2. If the vendor goes bankrupt, you still have the source so that you can make changes to handle things like new operating systems, and so forth.

Purchased source code may be a good compromise between a packaged solution and a built-from-scratch solution. You get something that works from day one (or at least early), yet your company can change whatever it wants when it wants without waiting for a vendor.

Still, there is no magic shortcut to full flexibility. Beware of boxed or template-based solution vendors. Custom flexibility does not yet come in a box. Such tools have their place, but it is probably not as large a piece of land as the vendor claims.