Electronic resource by any other name

An opinionated lament by Alexander Johannesen, National Library of Australia

Sinfonetta

Since we're at the beginning of this article, like in any good opera we turn to the overture, or, if it's a really good one, the sinfonetta. This introductory piece of music was meant to set the scene, to put you at ease of the things to come and a chance for the composer to get a little crazy, so we'll do the same, and take some huge steps backwards and start at the beginning of what the library is meant to be all about, with a little help from WikiPedia. At the core sits 'collection of books and periodicals', and on top of that, with the advent of newer technologies, ' libraries are now also repositories and access points for maps, prints or other artwork, microfilm, microfiche, audio tapes, CDs, LPs, video tapes and DVDs, and provide public facilities to access CD-ROM databases and the Internet.' And with 'recent thinking' thrown into the mix, 'libraries are understood as extending beyond the physical walls of a building, providing assistance in navigating and analyzing tremendous amounts of knowledge with a variety of digital tools.'

To match those goals, most libraries have a set of guiding principles or library values to which we're to keep in mind through our daily drudge. Here's some core library values stolen from a randomly selected national library ;

1. 
   
2. We are to ensure that a significant record of our country and our countrymen is collected and safeguarded
   
3. We shall meet the needs of our users for rapid and easy access to our collections and other resources
   
4. We shall demonstrate our prominence in our nation's cultural, intellectual and social life and to foster an understanding and enjoyment of the library and its collections
   
5. We are to ensure that our nation's people have access to vibrant and relevant information services
   
6. We are to ensure our relevance in a rapidly changing world, participate in new online communities and enhance our visibility

Those are good words; the library as the all-embracing custodians of culture and human knowledge. Well, I don't know about you, but here's my story. First of all, I was attracted to the library as a moth to light a few years ago due to the vast amount of metadata and hidden knowledge within, and the above library values that seemed to be smack on compatible with my own thoughts and values. My big passion back then as now was Topic Maps which is a standard for better control and navigation through vast amounts of metadata. A few of my personal interests also drove that metadata passion, like my obsession with renaissance-turned-baroque super-star Claudio Monteverdi; I wanted to create a portal for all things Monteverdi, his life, contemporaries and otherwise.

So what does a moth like me think about that shining light that draws me? You would think that by watching my fellow moths flying about that I would get the hint, but I'm not smart enough to admit that the light can be rather deadly; I'd rather just go there, never to distract my furious flapping of wings with silly things like contemplation. At least, up until recently.

I'm afraid that I must ask why librarians and their supporters (and I count myself as one of those) are closing their eyes to the impending doom which is the shift in our modern cultures from physical to virtual knowledge?

Over the last little bit I've had a growing concern, a nagging rash on my patient elbows, causing me to re-evaluate the value of the services we're giving. I'm of course not referring to the underlying principle of giving access to books and information about them to the wide public, although I do brush upon it. I don't mean the basic concept of free information, although I'm brushing on that too. I'm not really talking about the cultural implications of a civilized society losing its foothold on those titles ... oh, hang on, I am talking about that. Culture.

People are people; libraries are not libraries

That's the thing, really; culture is not books, nor is it resources, but people. Let's go back and take another hard look at epistemology which for me links culture and people to our perceived knowledge. I believe firmly in representationalism, or otherwise known as proxy thinking; that all things are perceived by proxy, that we see things not as they are, but as they are seen through our tools such as our eyes and brain. Given the nature of tools, as passive but interacting objects at our disposal, we all perceive things differently dependent on what our tool is like. Even two seemingly same objects can be seen differently through the same tool depending on contextual differences. The shear amount of neurons and couplings that are different from person to person give us a good hint that all perceived knowledge cannot be constant, nor singular, nor perceived or contextualised the same.

'Culture' is one word that tries somewhat haphazardly to join perceived experiences from many people into one fuzzy blob we can talk about and understand. I personally feel that the lack of this focus when we librarians (or as the case is with myself, a perceived wannabe librarian) talk about our world is the very thing that will be our demise!

The book will change, and unless the library changes with it, the library will disappear. Should we not focus more on the culture we are made from than on the objects in our collection? Our collection is one more proxied perception away from the real thing; a curator for a collection is yet another hop away; an exhibition from a curator about objects in a collection that belongs on a shelf in section three of a building in Dixon is so many hops away from the real thing that it becomes nothing more than a museum.

So, is that what we are? Or perhaps more importantly; is that what we want to be? A while back I wrote a letter to Mr. Library, our patron saint and all librarians incarnate, and it went something like this ;

Libraries sure are a strange mix of conservative and slow, and radical and fast. And, I've noticed, they divide the world in two; resources, and e-resources.

What resource?

Since the age of dawn humans have looked to tokens of wisdom to gather insight and guidance, such as tablets, scrolls and books. These things have always been easy to do; you walk over to the darn thing, and if you can read it, you read it. Access was controlled by mere physical obstacles, such as on top of a mountain, inside a holy temple, behind big and menacing guards, or deep in a dungeon with hidden traps. Easy.

What is a resource like? In the broad and all-embracing definition used by librarians everywhere, it could be anything our patrons wants to get their inquisitive hands on, anything from books, manuscripts and journals to pictures and music. This was reasonably easy to manage as they all came wrapped in some physical package, where "this bit of knowledge comes from this physical thing." Easy.

Then came the computer where the knowledge of man became unstuck from its physical prison, but business models that were guarding the knowledge through physical means did not like this, and hence created virtual obstacles to guard this now virtual knowledge. It went from a tooth for a tooth, to a bit for a bite.

So what exactly is an e-resource? When we give them a computer and we talk about electronic resources, there are as many electronic resources as there are webpages, eBooks, applications, games, music and multi-media and database entries put together, and bucket loads more. Just to try to explain all these new resources, we produced about 5 exabytes of new information in 2002 alone ;


I trust the 2007 figures to be substantially higher, and the arrow exponentially pointing up. There's going to be more and more stuff out there, so much that a multi-billion dollar company with some of the smartest people in the world with massive amounts of computing power such as Google can't find it all, little less any library with their puny budgets.

Compare that to the 32 million unique titles in the entire OCLC "WorldCat", which represents their best estimate of the number of titles held in all US libraries. That is about 256 terabytes, which, when you play with the numbers is about 0.00005 of 5 exabytes. Since 2002 there's been about 40 exabytes added. Yes, I know that amongst those many exabytes there is a lot of stuff that we don't want, that's bad, that's of poor quality, unrefined, unchecked, boring, uninteresting, but with all things near infinite, there is Shakespeare in there as well, - and Shakespeare that's not authorised, controlled, checked or known about to boot. This truly is the age of info-glut ; the volume of information that crossed our brains in one week at the end of the 20th century is more than a person received in a lifetime at the beginning of it.

I understand that e-resources traditionally have been treated as special and unique, a rare treat indeed for that privileged person who has his workplace pay big bucks for it. But the world has changed, and is forever changing. The specialness of a resource could possibly still be validated, but unique? Don't count on it, nor should you count on access being a rare treat. But most of all, count on the fact that whatever model we have today will be quite different soon enough. In fact, I bet that in the time it has taken you to read this whole article, the model of how we deal with e-resources should have changed at least twice. There are billions of people on this planet, each - statistically speaking - creating millions of resources.

In short, a resource is something that a user wants to have a closer look at. An e-resource is something that the user needs to analyze to find out if she really want to have a closer look at it.

Humongous amounts of resources, small tiny budgets

I think it's fair to say that neither we nor our patrons know all that is out there to know. There is just so much stuff out there! So much, in fact, that librarians mostly don't deal with it (and, admittedly, it's hard!), and leave patrons to their idle surfing of free and scruffy resources as much as possible, telling them "it might be out there", and tell them about Google.com and a few governmental sites.

"Oh, and by the way", says the librarian, "there are these e-resources that might be what you're after."
"E-resources?" the patron ask, "Aren't they all electronic resources?"
The librarian, with a sigh of relief, replies "Ah, but some e-resources are more special than others!"

Yes, some of the e-resources out there are indeed special; they cost money, often quite lots of money, and - if they're really special! -sometimes so much money that a lot of libraries (with their before-mentioned puny budgets) never have them. This content is special in that you must pay money upfront to see them, and libraries step in as your Samaritan uncle and offers to pay that fee for you and your friends, with a few inconveniences (you must physically come to the library, or you need to be a VIP patron, etc).

So how are librarians choosing what special electronic resource the library should buy with their scarce budgets? There are mainly two questions in this process; what e-resources do we know about that we could buy, and how much do they cost? The first part is sometimes decided in cooperation with patrons, other times just by our librarian knowledge, but mostly it's a carefully planned mystery mingled with tons of assumption as to what gets selected for patron infliction. The second part is easy; either we can afford it, or we can't.

What, then, is an electronic resource, in terms of how libraries currently deal with them? We all understand the basics; a licensed or free place on the net in which our users can get more information about various subjects. But the good stuff is so hard to find, and the world is so big.

I've been wanting the world to be different for as long as I can remember, so of course it's easy for me to sit here (or flap, as it may be, like a moth) wanting the world to be different in terms of our world as well. As such, most businesses want money. Not because they're greedy, but because creating and maintaining resources can be very expensive, that's how the world works, and as long as libraries interact with that crazy notion which is "the real world", we need to come up with the best way to balance their business with ours. And ours in this article's context, is the business of free access to licensed electronic resources such as databases and journal articles online.

Most e-resources comes with a limited set of licensing rules; not too many users, not all at once, and only from one spot, and libraries are supposed to bend over backwards in not breaking these licensing rules. But our bending backwards are heavily constrained by how libraries often get their funding.

Money, money, money

It comes down to a few principles of generic funding models; "we give you X if you can prove yourself in Y and Z, with a return of G, a deficit of H and turn-over of J. If all goes well, we can repeat this process next year." The trouble is of course that libraries are not easy to classify; we land in the shady areas of most organisational models, as we're collectors, preservists, archivists, public access granter, lender, national institution with constitutional tasks (collecting, archiving for example), academia, special interest research, broad interest supplier, service provider, intellectual property aggregator, funds granter, practitioners of copyright and rights, educationalists, conference and concert and presentation organisers and holders, and heaps more. We do so many things that we're stuck in a political minefield, caught between tasks, and have goals reaching far, far beyond bureaucratic understanding.

Within this minefield we create business models that are to serve the patron, avoid the scorn of politics and protect the business interest of our vendors. The focus of these business models become the license that is offered to us, and these license models are mostly interesting in the sense that no matter what we are doing now, it will change as rapidly as the internet evolves. These models are interesting because we knowingly know they are sub-par and will have to change, we know we're putting in a lot of effort for their protective pleasure, we know this cost us a lot of time and effort, and we know our results are temporary and will be changed shortly. This is a funky challenge for libraries who traditionally move rather slowly when it comes to more politically sensitive issues, and by golly, this is one of those issues.

How much time and effort does your library spend on juggling assumed access and licenses? For any library of a reasonable size, this can amount to quite a substantial chunk of what the library do. Libraries are faced with the challenge of balancing access to all and keeping the business owners of electronic resources happy; our users are, well, everyone, mostly. If anyone has free access through us, how can these businesses sell licenses to them?

Why don't we flip the problem around, and look at micropayments for a minute. Micropayments are basically us paying a tiny fee each time any of our patrons use their resource. What models of library interaction actually works in terms of micropayments? If we look to the traditional literature model, authors are paid by a library lending model where libraries gather statistics and pay from that. Micropayments work best when governed by a trusted source, and who is more trusted than the library world? This model already works in the literate world, so why not adopt it in the e-resources world?

If I was an e-resource business owner and I was offered to have a national library pay me for every time a patron used access through them, I'd jump on their offer. They should know better than most about the long tail of legacy content; People want metadata, even metadata from a long, long time ago, and they want it right now, so if the price to "metadata now" is small enough, we can balance better the resources and cost. It is access on demand instead of assumed future access. This means smaller upfront costs, yet with greater long-term payments and income stability, which leads to lots of little but concrete cents as opposed to one large dollar upfront for something assumed.

Currently, our various library systems have ways of dealing with e-resources, most of them made by larger libraries themselves, and modeled after the current hodgepodge license model. Can we make things better?

Library systems

We need to give careful consideration to how these library systems are measured in terms of success and failure. Make no mistake about it; we're in the library world here, and there are no normal rules here like "project success is determined by resources and time spent creating it, budget for maintenance, and return on investment." Currently libraries fake these factors more than anything; if something isn't delivered on time, no one gets fired. If a project doesn't satisfy a given set of requirements, no one gets shafted. If a project is functional but completely useless, no one gets kicked up the backside. That's just the way of the library world, mostly funded through academic and / or government channels. And of course, there is no return on investment, as it is measured in patron-positivism, a magical force that comes and goes with the feelings of the librarians that run the project.

Now, I'm not writing this to give librarians the blame for things; it is of course hard to claim return on investment when the return is measured in something as flaky as "patron feelings." It's just not doable. In fact it is symptomatic for rules and measures brought upon the library world by government systems that has no clue to what it is that we're really doing. It is terribly frustrating indeed.

How do we know that we're delivering systems that are supposed to help them in whatever quest they have? Right now I feel we're second-guessing on every level on that question; we design systems with a specific set of features in the hopes that we help at least a given percentage of users. I'd stress that we really need to work it the other way around, as usability has shown us time and time again that guessing what the user wants will always fail; we need completely open systems where the user narrows the features until the goal is reached! This is what we humans are about, isn't it? First we read the table of contents or the index (both from which you gain a sense of overview), then jump to the right chapter for the details, and from there make decisions on where to go next.

Library systems are traditionally super-focused on functionality; there are some core functionality we simply must have, and everything else is a feature, such as a clear understanding of the user requirement (as opposed to the librarian business requirements) or a usable interface (as opposed to a pretty one). Seriously, we must stop developing software applications; we must design humane systems. We need to think user-centred, otherwise our systems become a little bit more like weird Uncle John who smells funny and tell jokes no one gets.

Clay Shirky points out that we design software in such a way that we force people into patterns of behaviour that in real-life would be classified as sociopathic. It's a fun exercise, so let's try it out ;


But we can try out this experiment with our own systems, too ;


E-resource systems are often interesting anthropologistic exercises. Why haven't librarians and catalogers gone mad yet? Does it require years of library school to condition you to a life with library systems? Don't they know that there are better things out there? I think they do, and there is going to be some changes around here, very soon.

Winds of change

There is a glaring gap between library budgets and what society wants us to do, at least in light of our library values that are governed by politicians and our society. Librarians are expected to perform costly and complex things on a shoestring budget. If there is anything the library world has become really good at, it is being pragmatic and good at accomplishing amazing things within these ridiculous budgets constraints. I'm not sure why society at large doesn't see the link between the cost of e-resource operations and what they're giving us to do the job, but I suspect, as with all things in life that happen seemingly at random and without purpose, society doesn't have a clue to what we have set out to do.

It's hard not to see parallels between the way the library world currently see electronic resources and the way electronic content is heading, and there are some questions we need to work out about that future content ; Who owns the electronic bits and bytes that gets transferred over the net? What, in terms of copyright, does Digital Rights Management really mean? Who's rights? Are we providers of access and / or content? Does our vendors have unique content not available elsewhere? Should we treat them as special? Unique? How do we judge the importance of their content? I fear that the answers aren't readily available to us - certainly not given through this article - and that we need to sit down and be tough on ourselves on these issues, to think about our core values more than pleasing business models.

Will e-resources be the beginning of the end of the library world as we know it? Our current e-resource practices certainly highlights the sensitive issue of trying to balance the world of economics and the world of freedom. In our distant past there was a significant gap between the rich and poor, those who could read and those who could not, between factory owner and the worker. That gap is ever closing in terms of access to knowledge and information. The very fabric of the library world, our library values and the fight for knowledge everywhere have been very successful, perhaps so successful that the library, perhaps, if it doesn't stop thinking in terms of physical constraints will lose some of its core reason for being.

In fact, are we too afraid that the answer might be something that alters the purpose and meaning of the library as we know it? Is that what it all amounts to? Being alive means to be changing. We all know this; the brand of butter we had as kids are probably called something different today, probably don't look much like the old stuff, nor does it probably contain real butter anymore. The library is like that butter, but unless we too change with the time, I fear - so to speak - our time has come. Our values are still solid and good, but our implementation and practice is moving further and further away from the real world.

Being alive means to be changing, and we simply need to hurry up; we've got values!

Building a better XSLT for our common future!

A little while ago I was asked to look at some XSLT for someone who had done some changes to a template they didn't originally create, but who now couldn't figure out what the hell was going on. You know, the usual stuff.

So I told the person to send me the stuff, and I'll have a look and see what I can do to help. The files appeared; one input XML file and one input XSLT template, and one HTML file as an example of what it should look like when working properly. The XML file was basically a repository of various guides that has been made, and the XSLT was for creating an index of these guides, sorted by either subject or format, and grouped alphabetically.

This article will deal with a number of things, from badly created schemas to Muenchian grouping, giving some advice along the way to some real examples. Yes, I will inflict you with my opinionated advice on things such as filtering, XSLT best-practice and sorting and grouping. The example codes used are real data, but with the disclaimer that its somewhat old, the author doesn't work here anymore, some
content is scrambled, and I have seen lots of good code here as well. Ahem.

I skimmed through the XML and knew already then that the XSLT - in the virtue of having to deal with this XML file - would be a real charmer indeed. Let's start with a snippet from the very top of the page;



Guides
[snipped]
Contents

Aged
aged


Asia
asia


Australia
aust


Australian Studies
elec


Biography
biog



Ignoring the fetish for all capitals for all your markup needs, you have to ask why there are so many topic lists all having apparently one entry each? Of course what they're trying to portray is something more like this;




Aged
aged


Australia
aust




And so forth. Basically we're having a semantic problem right there; the XML uses language to portray lists, while in reality they are all one list. Confusing. Let's look a bit further down our XML file, after about 100 items;



Miscellaneous
misc


Bibliographies
bib


Collection Guides
collection



All of a sudden the turns into , and note how the name of the item in question also changes from to . It boggles the mind why people feel that the semantics of your XML structure must be given in your element names when the structure is, um, quite apparently there for all to see. And why are we mixing up two lists like this? Maybe there is some reason for this that will be apparent once we dig into the XSLT?

XML, episode 1 : Very real Menace

Before we dig into those existential questions, let's cut to the real meat of our XML, the data in which our XML was conceptually designed to handle;






[snip]
../../oh/links.html
Oral History
Internet Guide
Multi-Disciplinary






[snip]
../../find/pacific.html
Pacific
Manuscripts
Find...
Multi-Disciplinary



[snip]
../../oz/pacsites.html
Pacific
Internet Guide
Multi-Disciplinary




There is about 300 of these records, all grouped by a first letter. The first thing to note here is of course the general structure of the thing;









There is a fun little thing to notice here; can you spot in the records starting with 'P' the record that doesn't start with 'P'? Now some may say that that record should be sorted and displayed in the 'P' section because it is a granular of the "Pacific" keyword, which is a big assumption, especially since the markup doesn't indicate any such thing. In fact, 'P' in the context of the XML is what group it is to be sorted by, not title. I know this from lots of research talking to the stakeholders. There is no way to know that by looking at the XML nor the result page, not even by the comments in the XML. In other words, this XML does not tell us - through semantics nor structure - why the data is structured the way it is.

Further, if you're going to all the trouble of sorting your records by something, wouldn't it help things if you gave an indication anywhere in the XML data itself about this wonderful fact and not just in the remarks? I don't know, I would certainly feel that helpful, because if you don't, then how on earth can you act upon that given data with XSLT when XSLT can't get it? How are we supposed to write XSLT to handle this? Ok, let's have a tiny look at the original XSLT.

XSLT, episode 2 : Attack of the cloning

First of all, the template has two matches; 'ROOT', and going from that, the match 'GUIDES'. Some framework HTML is created in 'ROOT' with a in it, and all the content is generated in the 'GUIDES' rule, of course being the root element of our XML file. Let's have a look at one of the hundreds of similar sections of that rule;

So basically there is a loop for every darn BIBREC with the conditional put on its SUBJECT sub-element. What do we have, then? A 1107 lines long active and hand-maintained piece of XSLT. And remember those and elements in the beginning of the XML file I didn't quite understand what did? At the top of this XSLT we've got some code to generate a list of all our topics;

#

Isn't it funny how some people want to do things the hard way? All of the following code;



#




Could be written as;





No wonder people complain about XSLT being a mess when people write such messy code. Anyway, a slight digression from the real matter, and the crux of what I want to talk about in this article;

The list of the topics and the topics themselves are both maintained in the one XML file. But where are the items used? Ah, well you see, there is actually two XSLT files; one for listing the records by topic and one for listing them by format. So, the end result as of now is that we have one XML file that is maintained by hand clocking in at 1837 lines, and two XSLT templates maintained by hand at 1107 and 418 lines of code respectively. Every time you add or change any record in the XML file, you need to update both XSLT templates with data specific rules. Every time. The mind boggles.

One can ponder about how anyone came to such a setup, but I won't go into too much detail here. But these are real examples that's out there, and I see them all the time. For some context, a Google-search for "xslt tutorial" gives about 5,770 specific hits. Searches for "DTD tutorial" and "Schema tutorial" gave 610 and 757 hits respectively. That's a ratio of normative 4.2:1 which to me really implies why we're in this sorry mess, and I'll let off some steam now before we get to an alternative version of the XML and XSLT we've just seen.

XML, episode III : Vindictive schemes

First off, creating schemas - be it a DTD, an XML Schema or RELAX-NG - is fun. I like it. It makes me think of my data in interesting ways. I love to make my schemas as semantically rich as possible without losing sight of simplicity and elegance. If I create a schema that does the job but doesn't feel elegant, I scrap it and start over. Let me exemplify;





[n]




This is a typical schema structure and naming of something called a topic. But I would not be happy with this at all.

First of all, we've got structure embedded in the naming of the elements, which is a big no-no in my world because you may want to reuse elements, restructure them and otherwise mangle the schema to fit more data than what your puny test-cases imply. A element is contextualized by its place in the structure, not by its name. Many things can have a name, but only topics can have a . We want to reuse in as many places as we can, because it gives us more semantically rich markup, so that at some later stage we can ask for any element with a sub-element instead of group many name variants together, clogging down the system.

Second, the use of plurals to indicate groups and lists can quickly become visually difficult to differentiate between, especially with lots of in and out traffic between them. Of course, good editors and validating tools help out here, but we're kind of assuming that no person will ever look at our XML for clues to what we're doing.

Thirdly, an element is used for something that is a direct PCDATA reference point to the element, in this case a unique identifier. I prefer these attached as attributes instead, cutting down on the ambiguity. Here's what I would prefer;









Here we got a clear indication of what the root level is (it is a list of topics, not just a bunch of topics), that each topic has a unique id (and not as many id's you like), and it has a name (and not a topic-name, which may or may not be the same as a name). All this semantically rich data without even filling it with data. That's how I like it. And it looks tidier too.

The next issue at this point is to talk a bit about what exactly is in the schema. In our XML file dissected a the beginning of the article we had presentation data mixed with actually data, where the data also relied on the order in which your XML elements came. Mixing presentation and data is something that should be avoided in all aspects of your development, including the schema level. Also given the declarative nature of XSLT, relying on the order of elements in your XML is a big no because order isn't guaranteed. Both of these are stumbling blocks in the land of not just XSLT processing, but any XML related development. Clear separations and no ambiguous assumptions about the processing of your XML is essential to happy development.

Let's get back to our XSLT problem. I fiddled with the original code a few minutes, and decided that not only did I want to rewrite the XSLT from scratch, but also change the XML file in question as well. Since these were both hand-maintained, there were no problems with doing it this way.

XML, episode IV : A new hope

First, I changed the XML;



Guides
[snip]


Gone are the all-capital elements. This isn't a major issue, but it can be significant if your XML in some way needs to resemble classes and instances of things. In Java, a class might be 'SomeBeaver' while an instance might be 'dFormBeaver'. You never know. Next;




Maps
Manuscripts
Bibliographies
Collection Guides
Discover Guides



Ok, ok, I admit it; I could have put these presentation related items in a separate XML file, but I decided that clearly marked and semantically prepped up, it would not be too evil to have them at the top of the XML file. So I'm a pragmatist. Sue me.

Next, all records converted into a similar but richer format, and not grouped by anything;




[snip]
../../map/aerialphoto.html
Maps
Collection Guide



First, let me point out the link between /guides/index-presentation/format/@name 'CollectionGuide' and the /guides/record/format 'CollectionGuide'; every record has one or more
and elements which is an indicator of what group that record belongs to and should be sorted by. Remember that we had two XSLT templates, one for each of these groupings. One of the key points to the rewriting was to combine them so that ordering is automatic.

Next, none of the records are sorted in the XML in any way. They can be entered willy-nilly, and hence we don't care if the maintainer of the XML puts them in some kind of order or not. That is the second point to the rewrite; we want the sorting to be automatic.

Here is a reminder of one of the items;



Miscellaneous
misc



Notice the element, now gone in the new XML schema; it was used to create anchors within the index page to the group it belongs to. Since we want to do this all automagically, it is removed and will be replaced with some filtering to make it HTML friendly instead. That means that to add a new group, you just ... um, add a new group to the records you feel fit and the system deals with it.

A quick check tells us that our XML file has gone from 1837 lines to 890 lines. That's a good start, considering the same data has been retained. But let's see how we can deal with the XSLT.

First, the very parameter that will drive the template;






The parameter 'sort' is, as the XML comment says, either 'subjects' or 'formats', which more or less speak for itself, and the default value is 'subjects'. That means that we can now apply templates based on this parameter;
















We call the template 'html.create.bookmarks' with either the parameter '$group.subjects' or '$group.formats'. Let's have a closer look at how we set '$group.subjects' up at the beginning of our template, because within lies the secret to it all!

The dread pirate Muenchian!

Since we're dealing with grouping, we need to take a deep breath, and invoke our skills in the Muenchian Method. This fact alone is possibly one of the biggest reasons why so much XSLT is unnecessary complex. Here's the crunch; learn Muenchian Method grouping Right Now(TM)! Let's look at how this is done;



select="/guides/record/subject[generate-id(.)=generate-id(key('group.subjects.key', .))]" />


Look at this code very carefully. First we create a key-set that holds all of all our records. That means all of them, including duplicates, but to do proper grouping we need to get rid of all the duplicates. The way to do that is to invoke the black magic of Muenchian grouping (named so after Steve Muench, who came upon it) in our variable 'group.subjects', using the function 'generate-id()';

We take our list of subjects, including the duplicates, and we select the same node set again with the conditional that the element's unique id (as provided with the function 'generate-id()') in the node set is the same as the one found for that node in the key set. The explanation is that generate-id() gives us the unique id of the element in the node tree. Every node, every little scrap of info has an id that we can get, and we use this id marker to make sure that all duplicates are taken out of the final result.

Let's create a sequence of imaginative subjects; 'ABCCDB', and let's assign some unique id's to them: 'A(1) B(2) C(3) C(4) D(5) B(6)'. To create a new list, we look up each of these elements in our before created key table, and compare the id of them, like this; "For element A, is the current id '1' the same as the id for the element found in our key table?" In fact, we're literary asking if the current id is the same as the first instance of that same element's id in the table. Is the element we're looking at the same as the first element of the keys? If it is, it is in our result; we want this element to be part of our result. If not, meaning it is not in the first position - hence meaning that it is already included in our result earlier - then it's not what we want in our result. Makes sense, doesn't it?

So, first we set up a lookup table of all subject elements, and then we pick out only the ones that has that first position id. Now we have a list of non-duplicated subjects. We're now free to write code that sort anything by subject. Let's see how that is done;







Here we invoke the template 'html.create.records' with the parameter 'sort.by' which IS our non-duplicated subject list. This, as with 'html.create.bookmarks', can be switched alternated depending on the '$sort' parameter. So let's have a look at what we find in the invoked template 'html.create.records';




For each of the elements in our input - here meaning for each of the on-duplicated subjects - sorted by the current node's content (meaning the name of the subject), do the following;



select="//record[format = $this or subject = $this]" />


We create one variable with reference to the current subject node because we'll reference it later on, and another variable that contains all records found in our XML that has a sub-element named 'subject' or 'format' with content matching the content of the current subject node. That means all records with a subject or format the same as our list of subjects.

Let's look at creating an anchor to this group automatically;








See, it isn't so hard; we use the 'translate()' function to translate all spaces to underlines. We can add more rules here as other characters need translations, but for the intent of this XSLT, this was sufficient. We put all our bookmarks in the left column, and all our grouped records in the right column. Let's start off by getting the name of the group. This can be either one of two; either just the name of the subject itself (since we're demostrating this through the subjects list), or taken from the XML file if it has a different name for it in the table. Earlier, at the top of our XSLT, we created;





We then can use this key set to look up names if they exist as so;













Basically, if the subjects (or formats) name is found in the 'naming.key' table, use the new name from the table. Otherwise, just use the subject name. One might wonder why I did this, and the answer is to provide an alternative display name for the subject. Very often you'll find that the subject is 'boat' when you want to display a more flamboyant title like 'The Boats of the Blue Pacific'. Options are good.

Now that we have the subject group sorted (pardon the pun), we want to list all the records within this group. This is easy now with this reminder from our XSLT above;


select="//record[format = $this or subject = $this]" />


So we simply loop through them;

And then wrap it all up;






What have we possibly hopefully learned?

I hope that this little article has given you some fraction of real-world hands-on examples of problems you might find and how to solve them in XSLT in a more true-to-the-XSLT-spirit way. I know this only scratches the surface, and alternatives to what is presented here also exist. I've however tried to focus on the real issues, giving examples of solutions that are somewhat easy to understand.

I hope I've given some good example to follow when you encounter XML and XSLT files like this. First of all, make sure that the XSLT doesn't need to be altered if your XML adds data; we want our XSLT templates to be totally data driven. Secondly, learn Muenchian and don't be afraid to use it. Third, don't let poor schemas get in the way of good XSLT. Fourth, separate your data from the presentation as much as possible. And fifth, size does matter; the smaller the better.

There is a sixth lesson learned, but since it involves princesses, pirates and pain, I've omitted it for now, in your best interest.

Regards,

The Dread pirate Alex

Introduction

The purpose of this article is to a) give people - like project managers, sales, customers and other developers - an insight into what a web user interface developer does and doesn't, shoulds and shouldn'ts, and b) give web user interface developers insight into methods, best-practices and decisions me and my (former) colleagues at Bekk Consulting (Norway) do every day. The Web User Interface Developer - WUID for short - is not someone who "knows a bit of HTML"; there is quite a bit more, and some of that in there that might just save the day.

Web

By "web" we usually refer to the World Wide Web of the Internet. But it really doesn't stop there. Our daily web can also be our local intranet, a customers closed network or our own local machine. In the normal sense, we talk about communication done through the HTTP protocol. To narrow it, we work with HTML-related technologies, and to broaden it, we work with anything that trod down a TCP/IP stream. Knowing what technologies and options available in this environment is important, and so, in the broader sense, it means that the first hat we wear, is that one of the limiter;

So, a customer has come up with a cool feature they want on their home page that includes whistles and bells beyond the scopes of what the web offers, and needs maybe to be told "Yay, that looks impressive and cool, but, nay, those features are beyond the capabilities of the web. Maybe limit the scope of the feature?"

As developers for the Web, we must know about HTTP (and siblings), TCP/IP, Flash, Java, JavaScript, browsers (and their individual differences), web servers, tools and programs that deal with web communications.

User

The users are our primary concern. It is they we develop for, no matter how cool a feature a user wants, how nifty the function is programmed, or how pressed for time we are. Every technology and technique available to us reflect that the end-user is our primary customer; everything else is a secondary concern. This means that our second hat to wear, is that one of usability;

So, a graphic designer that comes up with a nifty cool way of entering data needs to be told "Yay, that is truly a cool fancy thing you've got there, but, nay, the users will find it cumbersome and awkward. Maybe something different?"

This doesn't mean we are usability experts. Rather, we're usability professionals, able to recognize future problems and deal with them before they become a part of the specifications. We're the "early warning system" of web sites.

As developers for Users, we must know about human cognition, basic psychology, logic sorting and good common sense.

Interface

People and computers have struggled from the beginning with how the interaction between the two works. Many methods have been tried, and the set we are most familiar with are the ones that - as far as we can tell - work the best. But most agree that there must be something better than what we have now, because we certainly is not totally happy with it.

Enter the interface. Now, an interface is basically a means of one entity to communicate with another. In our scenario, we're talking about the entities (H)uman and (C)omputer. Chuck on the (I) for interaction, and you get the old abbreviation HCI, which has many references on the net. WUID's need to know a great deal about this topic, although the term HCI is very broad in scope. To us, the usability and accessibility aspects are clearly strong paths to travel, but visual profiling, limits of technology, limits of human cognition and implementational on-the-fly estimation is essential, too. Our next hat to wear is that of interaction focuser;

So, your governmental site has a youth section, and wants a Flash intro to "appeal to the young", but hold on; "Yay, that is good thinking about the appeal, but nay, Flash accessibility is low, and it breaks 'best-practice' of information sites. You do have disabled youth too, and they want info foremost."

Making a good interface is not an easy task. There are many, many considerations to make, and you are bound not to get them all right. Our techniques and methods are only there to make sure that at least the broadest scope can be achieved.

As developers of Interfaces, we must know about all technologies offering of limited in accessibility and usability, interaction principles, MVC (model/view/controller) separation, languages for creating interfaces, and generally tool kits, tools and programs that deal with the interface.

Developer

We're all developers, but the term 'developer' is so broad that it brings little meaning to computer professionals these days, hence the title of this article; you'll find developers in every field. That assembler hacker you've got locked away in the basement is a developer. The nice guy with glasses in the cubicle next to you doing pattern analysis is a developer. Your boss tinkering with the VB-scripts in the CMS system is a developer. Even the boss' snotty teenage son is a developer of 'scripts on daddy's machine that makes all our lives eventful.'

We mostly do things on defined work, known as "projects." At the top sits the project manager that tries to fill in reports and estimates so that the project can deliver on time. There are testers. There are core developers. There are sometimes information architects (although this is quite often - and not always correctly - a project manager role). And in the middle of all these people sits the WUID, trying to convince all people that the output to the end-user - the HTML, CSS and JavaScript - needs to be according to a certain standard, following certain rules and generally follow certain "best-practices" of the web. And that is no easy task when everybody "knows" that if you have some info, you chuck it in a table, or you slap a FONT-tag around it, or you swear to Front Page to draw it, and then copy and paste it into the application. Um, no, our next hat to wear is that of web-soup experts;

So, your core developer outputs HTML in his classes, but "Yay, your output looks alright in Internet Explorer 6.01, but nay, it looks awful in IE 4 and 5, every Opera browser in existance, most Mozilla (and of course Netscape 6+) and Gecko browsers, and insults old ladies using Netscape 4 series. You need to validate your output to a DTD, make the data accessible, fix code for bugs in certain browsers, adjust the CSS files, and tweak the JS file for the added functionality. And you can't use that JS function; it isn't supported in browser Y. And don't put the header in a P-tag following a DIV-tag; it will make browser Z crash. And don't give this CSS to NS 4; it will crash. Oh, and don't use FONT-tags; they are deprecated.

Or better yet, out source it to a library and let a WUID do the job, so that all the above gets done right, and you can focus on your actual job?

The end layer of a web application can be a very complicated mess. Core developers, who wants to create classes and API's, dig into storeprocs and likes to build obscene Perl scripts to remove hashes from local identifiers on the fly, do not want to - and should not - create HTML, CSS and JavaScript code. Don't fall into the pit of "HTML is easy", because that is only a brief look at the surface of the water. Any captain of a successful ship knows to leave the iceberg know-how to the iceberg experts. Follow good Model View Controller separation methods to make sure the success of our project.

As developers, we need to know quite a lot about other general developers areas; XML (and related), SGML (and derivatives, such as XML and HTML), Java (J2EE and EJB especially), C/C++, C#, VB, JS, CORBA, COM+, SOAP, Microsoft .NET, web services, Python, Perl, Rebol, Linux, *BSD, Windows, AIX, Apache, WebLogic, WebSphere, Tomcat, JBoss, xDoclet, Middlegen ... aaahhh, and quite a lot more, far too many here to list.

Coming together

Where does it end? How many things must we learn until we are a really good Web User Interface Developer? In truth; it doesn't end. We must always know the old stuff, learn new stuff, learn new stuff about old stuff, discard wrong stuff and generally try to be strict and pragmatic at the same time. Why? Because web developers are often the glue in projects.

Glue

WUID deliver HTML, CSS and JavaScript to Core Developers (and all the "best-practice" advice possible), pushes Model View Controller down on the Systems Architects (and implementation details of metadata, accessibility and validation), giving Information Architects the rounds of metadata (and how their CMS can handle it), implementing every bit of it, and telling the Project Manager all about how smooth things are going.

The role of glue in projects are often thought to be the project manager, and as much as that is true in the sense of pushing deadlines, threats and paper around, when it comes to making sure the quality of the output - the user interface, the interaction scripts and the HTML/CSS code that builds it all - is top-notch, the WUID is the centre of all things. There are several real reasons why this is so, and it is best defined what others don't usually care about;

• Not all care about validation, and found especially true amongst core developers. This is a strange notion given that they most of the time work with compilers that require correct syntax. HTML is really not any different, but the reason for this state is that there are certain browsers that are quite forgiving on bad code, teaching people bad habits about writing their HTML code. "It works" doesn't mean it really works, and especially not for everyone.
• Not all care about accessibility, which is understandable; most developers are seeing, talking, hand-waving and dancing individuals. Accessibility is easy to forget, but is a pain when missing for those who need it.
• Not all care about semantics of HTML, using HTML for what the intended standard was not intended to do; layout. If you need to put emphasis on a word, who are you to determine that it must be bolded? Emphasis can mean many things, bold being but one of many visual ways.
• Not all care about CSS, thinking that FONT-tags cover whatever layout they need. This may be true for a very early alpha version of your site, but it stops there.
• Not all care about usability; functionality in the classical sense is what defines most user interfaces. This is not always fortunate, as a good usable interaction might define program control and implementation differently. We are making these programs to cater for users, so why is the core developers in focus?
• Not all care about alternatives to given methods. Some surf without the capabilities you assume they have, like JavaScript, cookies, browser-specific methods and other assumed ways of going about things. Some have security issues, some surf through proxies, some browsers identify themselves as others (or some you never heard of, but are fully capable of doing what you think it doesn't) and so forth.

In closing; Who cares?

It feels often that most people - apart from those who work with web user interfaces on a daily basis, and even then not always - don't take their user interfaces seriously, even though the end result of any web site is exactly that. Sure, you've hired some designer to draw some pretty layouts for you, and you got some core hackers building you a rather complicated database, and all of this can be rendered useless in a very few lines of bad HTML code. In brief;

• Project Managers should care about enforcing "best-practice" HTML/CSS/JS guidelines to ensure that the projects lands on its feet and within time. Debugging sloppy code is always a major factor of projects landing face-down. They should also care about the accessibility to the site, making sure both that and the usability of the site maximizes the users intended for the site. And they need to remember that assumptions on who their users are are most often wrong.
• Core developers should care to make as little HTML as possible; it is not their job, it is (usually) not in their interest, and it isn't what they (usually) are hired for. They should also make sure that the little HTML/CSS/JS they do produce, holds a quality that do not muck up the end result, using semantic markup, validation, and - if possible - wrap their result in a API a web user interface developer can access instead.
• Information Architects should care about metadata embedded in their CMS and in the end-resulting web site, and what options are available to us after they are there. Are there better uses of metadata that KEYWORDS and DESCRIPTION? The WUID can tell you all about Dublin Core, RDF, Topic maps and other goodies.
• Developers in general should care about keeping the interaction and profile design to a level which a web user interface developer agrees on. A widget of a certain kind can take weeks to implement, but a similar design - once we agree to make compromises - can take days, even hours.
• Developers in general should care that their efforts are measured not in the functionality and the methods within, but through the quality of the interface and the content.

So you should definitely care about things that a web user interface developer cares about. It just might save the day.