REST and SOA as a process for application design
I'm going to stray a bit from the library theme, and talk about design of RESTful SOA. It's a topic close to my heart, as most SOA talk these days are full of vendors claiming money can buy you not only love, but immortality. With SOAP? Hah!
No, I think reinventing what the Web does really well already is a) a waste of time, b) doomed to make a bad copy (as the web is constantly moving, while the SOAP / WS-* stack is immersed in slow-moving standards), and c) over complicating things (I like elegant simplicity such as the innards of the Web).
REST
Roy Fieldings' REST dissertation has swooped upon the middle and higher layers of the IT world lately, making a lot of them admit that, perhaps, this whole deal about using HTTP and loose XML (often XHTML) to create scalable, fast, simple and dynamic applications (well, as an architectural style, to be specific) might have something going for it. REST has been around for a long while, being the very fabric of what the internet is based on, slowly extended and refined over the last years 15 years (even though a lot of these concepts are again based on earlier technology).
Service Oriented Architecture (SOA) is a little bit tricker to define, especially these days when big corporations have discovered and use it as a buzzword, but basically it is technical architecture creating loosely coupled (meaning; the items in question knows very little of each other) services, and where a service is a piece of software that some other piece of software might use (as opposed to direct human usage). Now, a lot of people already talk about this stuff, so I'm not going to add to that. I'd rather talk about what I think when I do this stuff, to talk about actual implementation.
Working in both these two worlds, putting them together to design and create applications, is quite different from the normal software development processes that's so popular these days. The most striking difference is that during application design you think in terms of resource orientation (as opposed to object orientation, or functional design) and how to represent services (as opposed to a program, or a module).
You can either plan a big-bang approach to this (standard waterfall models) per service, or you timebox a more agile approach of creating one or several services that does the simplest thing needed to service your proposed application. The world spins around the axis of identifying application to solve problems; let's turn things around (and this is a big part of SOA) and see if we can come up with services that solves problems instead.
Typically you have a sleigh of applications that all have common functionality, such as user management, database storage, configuration, session handling, search and a few other bits and pieces depending on the business you're in. There's many ways to deal with reuse of these "things", and I deliberately call them "things" at this stage, because as soon as you call them "modules", or "libraries", or "reusable code" you're setting the scene for quite implementation specific stuff, such as what language you're going to use, or what platform it runs on. I don't want to deal with "libraries" for example, because if some library is written in Java then I need to make my other solutions in Java, too. If I have a "module" that does X in Windows using C#, the chance that "module" is linked to that technology is quite high.
Things
No, I want to talk about "things". For example, let's talk about users. A lot of applications deal with users in some way or another, whether it's displaying information about them, for them, authenticating them, create properties on them, or otherwise work with their user data. How can we create a service that applications might have good use for?
Since we meddle here in all things REST, the first thing we do is to think of the service in terms of resources (as being resource oriented is extremely important; expose URIs for every resource, as small / atomic as need be). I usually create two sub domains to hold services, one for internal behind the firewall services (soa.domain) and one external (ws.domain; 'ws' for web services), and I also try to have a trim set of basic elements that express generic functionality (search, user, session, database, properties, etc) wrapped in an even smaller and more generic set of domains (x, y, z, a, o, a, etc.). Through this, the first part of my design process is to play around with URIs and hierachial taxonomical ideas to see what feels right ;
http://soa.example.com/identity/user[/{userid}]
http://soa.example.com/user[/{userid}]
http://soa.example.com/user/id/[/{userid}]
http://users.soa.example.com/{userid}
Balance this with ideas on premature optimization (what, you thought that was axiomatically bad? It's allowed to think about these things, you know :) in terms of request times for a domain (the more domains involved in a series of calls, the longer the overall response time, generally speaking) and what feels right.
In my case, the first one seemed the most right. I've developed a small set of root categories in which I "place" my services, such as /search, /publishing, /identity, and so on. These categories are not canon; they are placeholders for loose ideas and thoughts, bound to change in the future as your SOA evolves.
Evolution
Evolution in your SOA is very important, so you should design for it in mind. For example, what about version control of services? Some talk about versioning being part of the XML schemas that services deliver, others talk about content negotiation (crazies :). I take a rather pragmatic and somewhat naughty approach (in the sense that you shouldn't put semantics in your URL's which humans will look at and try to pry apart and use / misuse) and put versioning into the URL at the base of the service defined. For example ;
http://soa.example.com/identity/user/v1[/{userid}]
I also set a rule to service development ; maintain backwards compatibility as far as you can. There's no need for an ever update to the version number if you design your XML schemas that pass through them in smart ways, and this reduce the overhead of deployment, introspection and dependency. Another rule to service digestion is to only react to what you understand, and ignore all that you don't; this again enables backwards compatibility as you, say, add a new (but non-critical) element to your metadata which older service users don't understand and simply ignore.
For proper development of a RESTful SOA, though, I'd suggest two things as a minimum ;
Time management of this development is also important. Because services must be allowed to break, be allowed to screw up, we must also allocate time for these screw-ups to happen. Trust me, it's a good thing ; a smaller failure now ensure we don't screw up big time later. (And this very point is probably the cause of so much bad management and so many failed [enterprise] projects as it's very easy to overlook or not taken seriously enough. I can write a whole book on this topic alone!)
And people who have some sort of ownership of a service (as developers, or analysts, or whatever) must be given time for short iterative development, for little updates, modifications and tweaks. Services won't be successful if you treat them as small bangs (meaning; gather requirements, write spec, make it, sign it off), and probably only can work through continuous tinkering. Such tinkering doesn't have to be time-consuming nor difficult to manage, but it does require you to plan for it. When Bob goes on to his next project, remember that he's also needs a half-day per week to tweak and fiddle with his service.
Introspection
One feature that I can't emphasize enough is service introspection, an area that most writers I've seen gloss over. And sure, you don't need it in order to create a SOA or a web service. But I'll assert that you need one if you're a) smart and b) want to create a healthy SOA that can stand the test of time.
Introspection in my world does three important things ;
http://soa.example.com/identity/user/v1?introspection
or, if you want to split the three up ;
http://soa.example.com/identity/user/v1?introspection=state
http://soa.example.com/identity/user/v1?introspection=docs
http://soa.example.com/identity/user/v1?introspection=tests
1. Handling state of a client through hyperlinks is a somewhat forgotten part of REST, which is easy to miss when your design is at an early stage (and it usually stays that way because you don't think you need it by the stage you're made aware of it). It basically comes down to either URL-driven or FORM-driven hyperlinks that takes you from whatever state the current URL gave you to the next one. For example, a resource soa.domain/search?q=fish might give back a list of URL's to pages of results, or a form to do a sub-search, all documented through hyperlinks. I personally think the use of XHTML is good for this, but a bit more formal and equally elegant is the use of the Atom Publishing Protocol (not to be confused with the Atom Syndication Format).
2. Documentation is important, and could be as easy as just returning an XHTML page with some text about what it is, how to use it, and so forth. However, I see a major part of documentation as to what dependencies the service has got, so I've got a section that looks a bit like this ;
Notice that this is perfect XHTML. All that's required to understand this list is understanding the identifier for the list, the "SOA-dependencies", which I can locate easily through DOM or XPath. Through this mechanism in services you can now map the whole dang thing, plot in your dependencies, check it against your test suite (talked about earlier) for ultimate coolness and power.
In this section I might add that I often incorporate a ping parameter which testing and monitoring systems can use to check the health of a running SOA, something like ;
http://soa.example.com/identity/user/v1?ping
or, if you've got the RESTful chutzpah required, use the HTTP method OPTIONS instead of a GET on a URL. I actually do both. The HTTP response code hence talks about the generic health of the service as far as it knows, and you can use this info not only for monitoring and testing, but also for automatic systems and smart clients.
3. It may seem a bit strange to ask a service to give you a test-suite, but it actually is a very encapsulating and clever thing to do, making sure that tests are all handled at the same place where development takes place. I can do ;
http://soa.example.com/identity/user/v1?introspection=tests
and I'll get back something like this ;
href="http://soa.example.com/identity/user/v1/2456325786234985"
xpath="/response/item[@name='user']/id"
is-true="2456325786234985" />
... [more tests here]
Basic test-case skills are probably a plus at this point to understand what this is about, but basically we assert that the XML/XHTML that the URL returns will give the result "2456325786234985" when the XPath expression "/response/item[@name='user']/id" is run.
Your testing framework for the SOA simply collects these test files at intervals to build a larger test-suite that stands as the controller for the whole system.
Finally
Just a few finishing thoughts about rigidity, complexity and management of a RESTful SOA ;
If you don't have dedicated SOA people, then don't do it. If your people (developers, analysts, managers) aren't very flexible, then don't do it. If you don't understand REST, either really learn it (this book is the best there is on this subject!), or don't do it. If you think you need complex systems, don't do it. If you can't wrap your head around resource-orientation, then don't do it.
The thing is, you can perfectly well live without it, create SOA or some other well-meaning version of that concept with SOAP/WS-*/BPEL/ESB or whatever big vendors are more than happy to help you with. You can create POX services just fine. You won't be RESTful, but you will probably survive without it. You don't need it in as much as you can live on only water and bread for years and years, but of course I wouldn't recommend it. :)
Anyways, a few thoughts there on RESTful SOA design and implementation. I haven't digged into the semantics of modeling a full SOA yet, nor talked much about pipeline XML schemas (although the APP protocol is a good hint), system introspection through things like WADL, or even the hidden benefits of ROA (resource-oriented architectures). So. More to come, then. Until then, happy hacking.
No, I think reinventing what the Web does really well already is a) a waste of time, b) doomed to make a bad copy (as the web is constantly moving, while the SOAP / WS-* stack is immersed in slow-moving standards), and c) over complicating things (I like elegant simplicity such as the innards of the Web).
REST
Roy Fieldings' REST dissertation has swooped upon the middle and higher layers of the IT world lately, making a lot of them admit that, perhaps, this whole deal about using HTTP and loose XML (often XHTML) to create scalable, fast, simple and dynamic applications (well, as an architectural style, to be specific) might have something going for it. REST has been around for a long while, being the very fabric of what the internet is based on, slowly extended and refined over the last years 15 years (even though a lot of these concepts are again based on earlier technology).
Service Oriented Architecture (SOA) is a little bit tricker to define, especially these days when big corporations have discovered and use it as a buzzword, but basically it is technical architecture creating loosely coupled (meaning; the items in question knows very little of each other) services, and where a service is a piece of software that some other piece of software might use (as opposed to direct human usage). Now, a lot of people already talk about this stuff, so I'm not going to add to that. I'd rather talk about what I think when I do this stuff, to talk about actual implementation.
Working in both these two worlds, putting them together to design and create applications, is quite different from the normal software development processes that's so popular these days. The most striking difference is that during application design you think in terms of resource orientation (as opposed to object orientation, or functional design) and how to represent services (as opposed to a program, or a module).
You can either plan a big-bang approach to this (standard waterfall models) per service, or you timebox a more agile approach of creating one or several services that does the simplest thing needed to service your proposed application. The world spins around the axis of identifying application to solve problems; let's turn things around (and this is a big part of SOA) and see if we can come up with services that solves problems instead.
Typically you have a sleigh of applications that all have common functionality, such as user management, database storage, configuration, session handling, search and a few other bits and pieces depending on the business you're in. There's many ways to deal with reuse of these "things", and I deliberately call them "things" at this stage, because as soon as you call them "modules", or "libraries", or "reusable code" you're setting the scene for quite implementation specific stuff, such as what language you're going to use, or what platform it runs on. I don't want to deal with "libraries" for example, because if some library is written in Java then I need to make my other solutions in Java, too. If I have a "module" that does X in Windows using C#, the chance that "module" is linked to that technology is quite high.
Things
No, I want to talk about "things". For example, let's talk about users. A lot of applications deal with users in some way or another, whether it's displaying information about them, for them, authenticating them, create properties on them, or otherwise work with their user data. How can we create a service that applications might have good use for?
Since we meddle here in all things REST, the first thing we do is to think of the service in terms of resources (as being resource oriented is extremely important; expose URIs for every resource, as small / atomic as need be). I usually create two sub domains to hold services, one for internal behind the firewall services (soa.domain) and one external (ws.domain; 'ws' for web services), and I also try to have a trim set of basic elements that express generic functionality (search, user, session, database, properties, etc) wrapped in an even smaller and more generic set of domains (x, y, z, a, o, a, etc.). Through this, the first part of my design process is to play around with URIs and hierachial taxonomical ideas to see what feels right ;
http://soa.example.com/identity/user[/{userid}]
http://soa.example.com/user[/{userid}]
http://soa.example.com/user/id/[/{userid}]
http://users.soa.example.com/{userid}
Balance this with ideas on premature optimization (what, you thought that was axiomatically bad? It's allowed to think about these things, you know :) in terms of request times for a domain (the more domains involved in a series of calls, the longer the overall response time, generally speaking) and what feels right.
In my case, the first one seemed the most right. I've developed a small set of root categories in which I "place" my services, such as /search, /publishing, /identity, and so on. These categories are not canon; they are placeholders for loose ideas and thoughts, bound to change in the future as your SOA evolves.
Evolution
Evolution in your SOA is very important, so you should design for it in mind. For example, what about version control of services? Some talk about versioning being part of the XML schemas that services deliver, others talk about content negotiation (crazies :). I take a rather pragmatic and somewhat naughty approach (in the sense that you shouldn't put semantics in your URL's which humans will look at and try to pry apart and use / misuse) and put versioning into the URL at the base of the service defined. For example ;
http://soa.example.com/identity/user/v1[/{userid}]
I also set a rule to service development ; maintain backwards compatibility as far as you can. There's no need for an ever update to the version number if you design your XML schemas that pass through them in smart ways, and this reduce the overhead of deployment, introspection and dependency. Another rule to service digestion is to only react to what you understand, and ignore all that you don't; this again enables backwards compatibility as you, say, add a new (but non-critical) element to your metadata which older service users don't understand and simply ignore.
For proper development of a RESTful SOA, though, I'd suggest two things as a minimum ;
- use test-driven development for the service definitions (and use whatever methodology you like for the actual code for the service, although test-driven there too won't hurt you), so write your tests for your service (I use XPath with XSLT scripts for this) first and then develop the actual service until it passes all tests, and
- collect your services' tests into a large test suite ; whenever you add, subtract or change a service, make sure all tests pass. (If you can sneak this into a build farm of sorts, all the better. Automation for this type of development will probably save a lot of gray hairs) Through this you know what breaks and what's backwards compatible with your changes across the whole SOA. Don't deploy anything from development into test or production unless all tests pass. This is not a trivial task, and should be in the hands of someone who is full-time responsible for the SOA's well-being.
Time management of this development is also important. Because services must be allowed to break, be allowed to screw up, we must also allocate time for these screw-ups to happen. Trust me, it's a good thing ; a smaller failure now ensure we don't screw up big time later. (And this very point is probably the cause of so much bad management and so many failed [enterprise] projects as it's very easy to overlook or not taken seriously enough. I can write a whole book on this topic alone!)
And people who have some sort of ownership of a service (as developers, or analysts, or whatever) must be given time for short iterative development, for little updates, modifications and tweaks. Services won't be successful if you treat them as small bangs (meaning; gather requirements, write spec, make it, sign it off), and probably only can work through continuous tinkering. Such tinkering doesn't have to be time-consuming nor difficult to manage, but it does require you to plan for it. When Bob goes on to his next project, remember that he's also needs a half-day per week to tweak and fiddle with his service.
Introspection
One feature that I can't emphasize enough is service introspection, an area that most writers I've seen gloss over. And sure, you don't need it in order to create a SOA or a web service. But I'll assert that you need one if you're a) smart and b) want to create a healthy SOA that can stand the test of time.
Introspection in my world does three important things ;
- Handle the client state through hyperlinks (part of the REST paradigm)
- Documentation of interface, use and dependencies
- Provide test suite
http://soa.example.com/identity/user/v1?introspection
or, if you want to split the three up ;
http://soa.example.com/identity/user/v1?introspection=state
http://soa.example.com/identity/user/v1?introspection=docs
http://soa.example.com/identity/user/v1?introspection=tests
1. Handling state of a client through hyperlinks is a somewhat forgotten part of REST, which is easy to miss when your design is at an early stage (and it usually stays that way because you don't think you need it by the stage you're made aware of it). It basically comes down to either URL-driven or FORM-driven hyperlinks that takes you from whatever state the current URL gave you to the next one. For example, a resource soa.domain/search?q=fish might give back a list of URL's to pages of results, or a form to do a sub-search, all documented through hyperlinks. I personally think the use of XHTML is good for this, but a bit more formal and equally elegant is the use of the Atom Publishing Protocol (not to be confused with the Atom Syndication Format).
2. Documentation is important, and could be as easy as just returning an XHTML page with some text about what it is, how to use it, and so forth. However, I see a major part of documentation as to what dependencies the service has got, so I've got a section that looks a bit like this ;
Notice that this is perfect XHTML. All that's required to understand this list is understanding the identifier for the list, the "SOA-dependencies", which I can locate easily through DOM or XPath. Through this mechanism in services you can now map the whole dang thing, plot in your dependencies, check it against your test suite (talked about earlier) for ultimate coolness and power.
In this section I might add that I often incorporate a ping parameter which testing and monitoring systems can use to check the health of a running SOA, something like ;
http://soa.example.com/identity/user/v1?ping
or, if you've got the RESTful chutzpah required, use the HTTP method OPTIONS instead of a GET on a URL. I actually do both. The HTTP response code hence talks about the generic health of the service as far as it knows, and you can use this info not only for monitoring and testing, but also for automatic systems and smart clients.
3. It may seem a bit strange to ask a service to give you a test-suite, but it actually is a very encapsulating and clever thing to do, making sure that tests are all handled at the same place where development takes place. I can do ;
http://soa.example.com/identity/user/v1?introspection=tests
and I'll get back something like this ;
xpath="/response/item[@name='user']/id"
is-true="2456325786234985" />
... [more tests here]
Basic test-case skills are probably a plus at this point to understand what this is about, but basically we assert that the XML/XHTML that the URL returns will give the result "2456325786234985" when the XPath expression "/response/item[@name='user']/id" is run.
Your testing framework for the SOA simply collects these test files at intervals to build a larger test-suite that stands as the controller for the whole system.
Finally
Just a few finishing thoughts about rigidity, complexity and management of a RESTful SOA ;
If you don't have dedicated SOA people, then don't do it. If your people (developers, analysts, managers) aren't very flexible, then don't do it. If you don't understand REST, either really learn it (this book is the best there is on this subject!), or don't do it. If you think you need complex systems, don't do it. If you can't wrap your head around resource-orientation, then don't do it.
The thing is, you can perfectly well live without it, create SOA or some other well-meaning version of that concept with SOAP/WS-*/BPEL/ESB or whatever big vendors are more than happy to help you with. You can create POX services just fine. You won't be RESTful, but you will probably survive without it. You don't need it in as much as you can live on only water and bread for years and years, but of course I wouldn't recommend it. :)
Anyways, a few thoughts there on RESTful SOA design and implementation. I haven't digged into the semantics of modeling a full SOA yet, nor talked much about pipeline XML schemas (although the APP protocol is a good hint), system introspection through things like WADL, or even the hidden benefits of ROA (resource-oriented architectures). So. More to come, then. Until then, happy hacking.
Labels: application design, rest, soa, test-driven development