ShelterIt - My digital think-tank

I just read Robert Barta's hilarious post on some of Java's (the programming language) antics, which, for those who's following from home, I agree with wholeheartedly. His post triggered a monologue in my head as I was going out for coffee today, and I felt the need to jot down what I think is my thoughts on languages, semantics and semiotics from that inner tirade.

No, this won't be on programming languages, at least not only. Language permutes every aspect of our human evolvement, and has been one of the main factors for our success for the last 100 000 years or more, and lays down the foundations for that one thing which everything else spins around; communication.

We communicate. We all do it, all the time, every day. In fact, apart from physically mundane things such as eating, resting and procreating, it is all we do.

But as much as we do it all the time, and as such you would expect us to get better at it, we need more than just communication, but the right kind of communication. Anything can communicate, but anything can't always communicate well. A dog barks at me, but I'm not always sure what that bark means (although the drool and glistening fangs provide valuable hints). Good communication is how to get the message across without ambiguity.

Developing software

Let's take a rather common problem; how do we develop software? Well, it's actually quite easy. Just open whatever tool you need, start writing code, hit "compile", and voila! The trick is of course not to develop software, but to develop software well, to remove the ambiguity between the tools we use and the customer who asked for it. And just like with communication, we expect that if we do this often and repeatedly, we learn something along the way, and get better at it.

But developing software still sucks. I've been in this business for almost 20 years working in more than 7 different countries, and every place had the exact same problem; how do we get the communication right?

There's an absolutely stunning focus amongst people in my line of work on the technological parts of the solutions we create. Of course we need those technological bits to make stuff happen, but most of the time they drive the solution more than the other way around. There's a sensible saying that says "choose the right tool for the job", but it seems to me that very few actually stop and ponder that very good piece of advice. What does it mean to choose the "right tool for the job"?

The job

There's a few indicators to the problem above. First of all, what is "the job"?

Our ideal target is a working functionally-complete system that solves some loosely defined problem. As professionals we put a lot of methodology and effort into translating that napkin sketch into documentation fit for enterprise development, which in turn will be translated into programming code, which in turn a GUI will translate to the users of the system. We can say with great confidence that "the job" is pretty much a translation job.

How many levels of translation can we find in our daily work? Let's see, first there's Bob who has an idea (level 1) who talks to an agreeing Pete (level 2), they talk to John (level 3) who agrees (level 4) who writes a mail (level 5) to the marketing manager (level 6) who brings that to a meeting (level 7) to get agreement (level 8). Then John takes the revised document (level 9) to the CEO (level 10) who agrees (level 11) and gives a go, to which the document is sent to Sam who will write a spec (level 12) which Bob, Pete, John, CEO, Marketing boss and Sam all agree on matches the original idea (level 13). Then the spec is sent to the development project manager Jim (level 14) for scheduling, and then to the development team (level 15) for estimation and thoughts (level 16). they come up with estimations and suggestions (level 17), there's a bit back and forth between them and marketing (level 18 through 22), before Jill the functional designer is given the roughly agreed upon documents (level 23) which she alters (level 24) and the leadership team agrees with (level 25). The document now goes back to the development team approved (level 26), and entered their schedule. Development starts with visual design sketches by Pop (level 27), which needs approval (level 28 through 34), which Jim further plans (level 35), and then each developer gets a slice of the new plan (level 36 through 40). A couple of iterations go by (level 41 through 46), and we're ready for development release 1 (level 47) which needs to be agreed upon by all bosses involved (level 48), with tweaks and continued development iterations ongoing (level 49 through 52), we're getting to first test version (level 53), which needs agreement from bosses (level 54 through 57), and finally the first production release that is to match Bob's idea (level 58), to which Bob says "WTF?" (level 59) and to which customers respond "WTF?" (level 60 through 65)

You get the picture. And it gets worse; not only must we translate from person to person, but from person to tool to person to tool to tool to tool to person to tool to person to person to tool to tool ... .

The tools

It's hence a given that our tools need to be good at translations between humans and computers, no? The whole idea of programming languages is to do just this, so choose your language wisely, young padawan. Some choose low-level languages to get closer to how the computer sees things, other choose high-level languages hoping those who built the underlying layers have all the semantics down pat. Or up.

Back to Robert's post; in Java people use a lot of syntax to express something that in other languages takes less. But is language about semantics? Is the difference between French and German the words they use? Of course not. The same goes for programming languages.

How many levels of translation do I find in my development stack these days? I'm doing a pretty straight-forward webapp wizard thing. We could start with the customer having an old webapp which sucks, and they want a new and improved one. So they tell us what they want (level 1) which we jot down on paper (level 2) while showing us the old one (level 3), of which we agree roughly on what needs to be done (level 4). Our functional designer draws up a few sketches (level 5), which the customer representative agrees to (level 6) which she takes up the ladder to get more agreement on (level 7), and we're handed responses (level 8) to which we update the sketches (level 9) ... .

But let's look at it strictly from a technical aspect. Let's pretend we have a joint human understanding of what the needs and solutions should be. First, there's legacy to look at, and here we've got Java all the way (level 1), running WebSphere app servers (level 2), using enterprise given code guides shaped with Java 1.6 / 6 (level 3), running in three environments (level 4), with Maven to build (level 5), Hibernate for db (level 6), Struts 2.0 for MVC (level 7) and some JSP semantics (level 8), FileTags for more JSP "fun" (level 9), and of course JSP itself with mixed in Java code (level 10), running Jetty in development (level 11), with proxied Maven repositories due to no internet connection in devel (!!) (level 12), our own JavaBean naming conventions (level 13), legacy-based property names (level 14), about 10 different Java tools that comes bundled (but not used) with the platform (level 15, just in case namespaces gets muddled) ... . And that's just development. I could add a cut-down version of Eclipse as IDE (levels 16 through 20), running through WM Ware (level 21) and Citrix for intranet access (level 22), servers running Linux (level 23 through 25) while we develop on Windows XP (level 26 through 28) ...

APIAPIAPIAPI?

You know, at this point we gotta mentions application programming interface (API), because, frankly, they drive me friggin' nuts! Basically they are lists of words (which are objects or functions) that convey semantics of whatever module they represent. Sorry, that was the interpretive version. I mean, they are "a set of declarations of the functions (or procedures) that an operating system, library or service provides to support requests made by computer programs." It's what we use - all the time, all over the place - to have bits of software talk to other bits. All the semantics (i.e. meaning) is left in the hands of those write the docs for those APIs, which must match the behaviour of the code. Without this perfect match of perfect symmetry, everything can quite quickly turn to batshit. And it does, because this is what programmers really do all day; try to use APIs the way they were intended. Communication, programmer to programmer. But the focus again is on technology, rarely on the meaning of the thing.

Did I mention that I hate automated documentation with a passion? The way an API works is left in the hands of programmers slapping keywords with short explanations in code? All the hundreds of layers of fragile communication is autogenerated from code?

I refer often to something I call API thinking, where the semantics of a problem space is defined through keywords. It's a bit like trying to explain the complex nature of the Iraq War (the current one) in terms of War, Bush, 1991, 2001, USA, Iraq, Oil, WMD, and few getters and setters on top. Let's call it the iWreck API ;

Dependency oil = new Dependency ( Dependency::RESOURCE_OIL ) ;
int startDate = new Date ( "2001" ) ;
War war = new War ( startDate, War::TIME_UNKNOWN ) ;
war.author = "Bush" ;
war.history = new HistoryFactory ( new HistoryContext ( new Context ( new Identifier ( ... ) ) ) ) ) ) ;

There, done. Easy-peasy. Now we know the meaning of the Iraq War (current). What else is there to say? Come to think of it, I reckon Bush would make an excellent programmer.

Stacking it up

How many levels of translating our communication from one unit to the other have we got when we add them all up, all the human ones, the tools and the systems? Bucketloads! So many it is quite crazy to think that any of the original ideas come through this enormous filter monster we have created!

Doing communication right is damn hard. How can I explain? Maybe through example. I had some JSP code similar to this ;

When someone was going through the code for this, they found out that the JSP for the

tag didn't add any HTML nor did any logic in it, but simply told the compiler to travel up the stack. They hence removed

tags, because they didn't have any functionality attached. We now got;

I'm not sure you've noticed, but you just lost the semantics between the and a . What is a sub-section without a section? It's API hell, that's what. It may not look like much of a deal to most people, but remember here what we're trying to do, remember that very thing, the most important thing in all that we do; communication! And when you're removing a section in which a sub-section lives, you get a sub-section out of context. The reason for the sub-sections' life is lost, and the technologists killed it thinking they cleared things up by removing redundant code.

My soul weeps. I see this sort of innocent but utterly devestating loss of semantics through every level of interpretation, from customer to developer to user. We remove little bits here and there, add a bit there, and the original idea comes out all warped and garbled, and we think it is a good thing!

Folks, if I have one request if you were to listen to anything, then please be careful about semantics and meaning, careful about adding it in, state purpose and meaning, write more about what the hell the point of what you're trying to achive, in your APIs, in code, in thinking, in documenting, in specing, in talking to eachother, in anything you do, and - perhaps and hopefully poignantly - in reading my little plea for semantic sanity in software development.