In episode #337 (jeez, can you believe we’ve produced so many shows?) Richard and I talked to Jim Webber, Global Architecture Lead for Thoughtworks, about his favorite topic, Guerilla SOA.
Carl Franklin: Jim, let me ask you a little bit about the history. Were you there at the original SOAP spec with the guys at Microsoft and who is the other guy, Dave Winer, and that whole bunch?
Jim Webber: No. My moment started probably a year or so later when the first .NET Framework came out and it had support for SOAP-based Web services. Me and a good friend of mine, a guy called Savas Parastatidis, who now works at Microsoft, were kind of intoxicated by this stuff because integration would be a real pain in the backside up until then. We thought about the notion of using XML for interoperability, which is very sensible and that was really when my involvement with the whole sphere of Web services with SOA started.
Carl Franklin: Yeah, and the whole term SOA has been thrown around quite a lot. It has been very difficult to sort of get everybody to agree on what it means. Do you agree with that?
Jim Webber: Yes. You browse through any SOA bookshelf. You go to your favorite Barnes & Noble or something or browse through Amazon.com just there’s a huge variance in what people mean by SOA. At one end of the spectrum, it’s exposing databases through WSDL-centric Web services. At the other end of the spectrum, there’s all this heavyweight integration middleware and the whole categorization of services and taxonomies of services and that kind of stuff and I just think both ends of that spectrum are fairly unhelpful and we need to start thinking about SOA in more business-centric terms-the same way that we think about other software in business-centric terms.
Richard Campbell: If we can’t make the case to the business owners, we just can’t get that technology installed. It’s never going to happen.
Jim Webber: Right. I have this feeling that in IT we kind of get carried away by the shiny new objects. We want to put that everywhere so we hear about these services. It means that you run off and want to build services and we think that we are adding business value and yet we tend to have very little collaboration with the business. The tools that we have at our disposal, particularly the integration middleware and the whole pile of WS*- standards really constitutes a bit of a barrier between us and the business because the business doesn’t understand any of this stuff.
Richard Campbell: Well, I’m still trying to get a handle on the difference between the Web service and this whole SOA thing. If I use Web services aren’t I automatically SOA? Isn’t it just that one is an architecture and one is a technology?
Jim Webber: Sure. I guess this is where it comes down to a matter of personal opinion and preference. For me, a service is just a technical mechanism for hosting a business process. Often I’ll choose Web services as my deployment technology because that model can fit quite naturally with the workflows that I’m automating, with the business process I’m automating. So, because Web services is kind of message centric, it gives me quite a good collaboration point with my business users who themselves tend to be quite document centric. So I can ask those guys to explain to me their workflow, the other triggers in terms of documents to get consumed and produced, and so on and I have a reasonable chance of reflecting that in the Web services I build. To kind of get it in my mind the SOA tick box, the service that I built has to encapsulate something that is completely business meaningful, whereas, if I did what the toolkits make so easy to do, just wrap WSDL-centric Web services around everything, I don’t really think you got SOA there. I think you got a bunch of services, which aren’t really representative of your business. They are probably just representative of your database schemas and legacy app that you’ve managed to service through WSDL and so on, rather than being, for me, a good SOA, which is an automated computerized reflection of your business processes in the real world.
Carl Franklin: Suffice it to say that there are some tenants of SOA I think of when I am going to sit down and architect a service-oriented architecture. I need to think separation of concerns at the business process level. Doesn’t that sort of get my islands together in terms of what are these different services going to be and what databases go where? Isn’t that really the trick?
Jim Webber: Absolutely. I think you really hit the nail on the head there. When I’m working with a client to develop a SOA, actually they will talk about services or computers or any of that stuff. In fact, the clients I currently work for in North America, we actually instituted this during one of our short analysis phases, we have a swear jar where if anyone said the word database or computer or system, they were fined and we double the fine every day. That’s a really good mechanism of focusing people’s minds and, really, what the business workflows were rather than jumping ahead to technical solutions. So, the framework I like to provide is this: if I’m a customer and at the end of the workflow is revenue, help me to fill in the blanks in between, probably just fill in the steps in between. So as we go through filling in those gaps and building out those workflows, we start to see that naturally some of the steps on those workflows start to clump together. This is a first approximation of our SOA because we’ve got a bunch business-related steps that are naturally cohesive that we might chose to clump into one service and then we’ll reiteratively refine that until we have a model that we’re happy with and this becomes our first cut of the solution architecture.
Carl Franklin: I would think this is where a SOA is going to sink or swim in the architecture phase.
Jim Webber: Absolutely. At this phase, you’re bringing together the business stakeholders with the IT stakeholders and they are both agreeing on the clear vision for what the business looks like. This is kind of important because the IT people are learning how the business works and learning to see how related business functionality goes together in a cohesive way before we then go down into solution architecture and figure out how we are going to deploy that onto computerized systems in the enterprise.
Richard Campbell: Jim, how is this different from how we build any application? This sounds like the normal cocktail napkin process I go through with the stakeholders just to figure out any application I’m going to build for them.
Jim Webber: I think that is perfectly accurate. What we are doing here is we are taking the same kind of sensible approaches we’ve used over the last decade or so when we built individual apps and we’re applying those to the enterprise scale. We bring together the key stakeholder groups but instead of doing that for a particular application, particular potential silo within a department, we are taking a broader view.
Richard Campbell: So it sounds like I’m going to be talking mostly to middle and relatively senior management about the whole of the enterprise to try and find these services.
Jim Webber: Yes. To reduce risk, you may not want to do the whole enterprise but you certainly want to do it as a scale, which is meaningful to the business but doesn’t jeopardize that, so I would shy away from betting the whole business-style SOA transformations because they are risky. I do think that good SOA, like good application development, can be done incrementally. I think as enterprise architects we have been historically been very big bang and we worked on very grandiose gestures and yet those same techniques which serve the application development community so well really do serve the EA community too.
Carl Franklin: Well, Jim, Guerilla SOA is sort of like the ultimate Agile expression, isn’t it? I mean, it doesn’t mean that there’s no architecture involved. What I meant when I said that it’s going to sink or swim on the white board is that once you get into the implementation, with a SOA, let’s talk implementation, you have not just one database but you have databases in these little islands. If you don’t have the separation right now you are dropping tables from one and adding them to another and you can see how the complexity kind of gets out of hand at the implementation level if you’re moving things around.
Jim Webber: That’s one of the hardest things for traditional data and enterprise architects to get their heads around. In a good SOA, your data model will absolutely be quite denormalized but you will have very normalized information models. So certain services will be authoritative with certain lines of business data even though there may be caching and normal authoritative copies of that data are consumed by other services but you’re right, if you get your business functionality misplaced at the architecture level you will then have to refactor later on.
Carl Franklin: Then I would think that refactoring in SOA is going to be a little more complex than refactoring a single silo application.
Jim Webber: I completely agree there that if you’ve got single silo application, even a large application if it is well-written, refactoring is not a trivial proposition but relatively straightforward. We have the tools, we know how to do it, we have good practices around that. To refactor enterprise level really is where governance comes in. So you have to really go back to the business. You need them to drive which particular business area is your authority for a particular piece of functionality, and from that you can derive the information and data needs to support that process. Now if you got it wrong up front then your governance process will have to kick in and we have to figure out how to transfer some particular business process from being hosted in one service with, of course, all of its databases and logic and so on, and get that moved out into another service. That tends to be a fairly uncommon thing. It is not unheard of but it tends to be uncommon if you got broad agreements upfront at the architecture level from your business.
So if you have the notion of teams, which lives with services for a long time, and projects that intersect those services, as you see your service ecosystem developing you can often spot potential problems and you think, “Wait a minute, we may have made the wrong call here.” Before you got too far into delivering that particular piece of functionality across your services you can go back to your business stakeholders, validate that you really got their processes correct, and then make a decision about what you want to actually deploy in that software to automate that process.
Carl Franklin: Now we’re sort of getting into what Guerilla SOA is all about.
Jim Webber: Sure. It’s probably instructive to think about how the big SOA works. Big SOA is really about mobilizing an army. You are thinking about a kind of large campaign, large number of developers, large number of management professionals to get the cogs oiled and so on, and logistics. It really is like mobilizing a march against Saint Petersburg. It is a long drawn out campaign in miserable weather through thick and thin and the outcome is pretty awful for both sides typically. Guerilla SOA, on the other hand, is somewhat akin to what I’m calling a coordinated set of some skirmishes. Some of the Guerilla SOA detractors have kind of said this is just cowboy coding at an enterprise scale. I really don’t think that is the case. What I want to do when I’m engaging someone to deliver a SOA is make sure that I’m doing it in a low risk way. So I want to be able to break down my set of deliveries into something that is business meaningful from the beginning, and delivers business value early and then continues to deliver small chunks of business value, which my employers can get immediate return of investment on as soon as possible. So as an enterprise architect, when I’m looking at a Guerilla SOA engagement, I’m looking at taking the most important work business processes first and getting them implemented in a service where it crosses a small number of services and deliver that into production early.
Richard Campbell: Jim, where do you see the SOA primary living? Is this all about providing services beyond the enterprise or providing services within the enterprise?
Jim Webber: For the stuff that we’ve been discussing so far, I really think it’s in the enterprise. I think the promise of Internet scale SOA, with obviously a few large notable exceptions, really is only emerging at the moment. I’m really thinking about Guerilla SOA as a technique for doing enterprise architecture.
Richard Campbell: So, it is really about being able to centralize key resources so we don’t keep recoding them.
Jim Webber: I think it is about having an authoritative automated business process in the right place at the right time. I kind of side-stepped your question there a little deliberately because I’m not so keen on the notion of centralizing and that kind of stuff. If you happen to have a business process that looks similar to mine but actually it is different, then I don’t mind that flourishing in the same SOA ecosystem. I think the drive toward completely dry systems often is a bit of a Holy Grail that we never reach. You see this in an approach where people have decided that they are going to create a single enterprise messaging standard, for example. So they go away and they develop all of the schemas and so on and they build this marvelous enterprise Esperanto except by the time they finished doing that large job, they get back in the enterprise and it has moved on and the language is no longer fit for that purpose.
Richard Campbell: Right. I’m thinking about trying to stay on the business context of this. I’m thinking about the customer service that I want one place where I can identify customers, I can check what their credit rating is, I can check their contact history across the enterprise and one service that does all of that because every application that I’m building within my enterprise touches a customer in some way.
Jim Webber: Right. Customer service, and this may be a little controversial, but customer service to me does not give me any business process knowledge. I understand that we have customer information and that the enterprise runs on it. I mean, this stuff is critically important, but the idea of a customer service doesn’t make a lot of sense because it doesn’t sound like it does any active data processing.
Richard Campbell: Right.
Jim Webber: Now, some of the things you just mentioned, for example, get credit rating-a credit rating probably is a very sensible service because it involves a whole bunch of process steps that validates whether me as a customer or a potential customer is credit worthy enough to do business with you.
Richard Campbell: Would you hang that off of a customer or would you hang it off as a purchase?
Jim Webber: Well, that will depend on the process that is consumed in. So if you are telling me that in your department, you take potential customers in and spit out credit worthiness ratings out of the other side, that may well be a good candidate for creating a service. We will take the work for it that you currently do here, potentially manually with factors and so on, and you will help me to learn how to automate that as part of a service, which will ultimately deploy into our SOA. Now you won’t know necessarily where your inputs come from and where your outputs go to, and similarly when we deploy that credit ratings service into the ecosystem. It won’t know where inputs come from or outputs go to, but as part of something larger, if you are orchestrating activity you may well have customers being lined up in some sales process and all of that kind of backend ordering. So what is happening is some other process, which will be joined together with the credit worthiness service to produce some end-to-end business values to get the customer in the door and to get boxes shipped to the mouth of the warehouse.
Carl Franklin: Speaking of non-sequiturs, which is my favorite non-sequitur segue, the Enterprise Service Bus. We’ve talked about this recently with Christian Weyer and they’re working at Microsoft on their own kind of thing. This plays a role in SOA typically. What are the pitfalls and, first of all, what is an Enterprise Service Bus? What are some of the gotchas that you can run into?
Jim Webber: I really wish I knew what an Enterprise Service Bus was but there are so many interpretations of ESB. There are almost as many as SOA, but I typically take Enterprise Service Bus to be a framework for connecting services or systems together. That sounds fairly innocuous except that the major pitfall in the ESB approach is that it increases coupling, which is sort of ironic because SOA is meant to be loosely coupled. So what happens when you tend to deploy an ESB is that your integration work becomes entirely ESB-centric. So, instead of correlating the necessary integration software with your service with your process logic, you tend to push your integration stuff into the ESB box. These are a couple of rather dangerous side effects really. First, you stop becoming open. You become very much more centered around the framework that you are using for integration, which is typically someone else’s framework that you buy or download. Second, and perhaps much more damaging, is that it encourages the notion that integration is an afterthought. So I’m going to build this process. I’m going to buy this application. I’m going to build this software out. Then later on, the integration team is going to come along like a sweeper and manage to plumb all these things together for me so integration becomes an afterthought rather than being a first class citizen in the development of enterprise software. Both of those taken together tends to mean that you start to build proprietary integration as an afterthought in your enterprise, which is kind of weird because the life blood of an enterprise is it’s network. Its ability to flow information around people, around computer systems and so on and yet, the ESB encourages these two rather dangerous anti-patterns, which I kind of like to avoid.
Carl Franklin: So, it’s sort of like throwing everything in the ESB rather than keeping it separate so you are sort of violating the explicit boundary rule there.
Jim Webber: In fact, although your services probably won’t know about each other relative to one another, that coupling isn’t increased, and all of them become a quite tightly coupled integration framework you have chosen, which really violates that sense. Services really shouldn’t know about whichever particular kind of support that they are using to communicate with each other. They should also be interested in the content of messages that they consume and produce.
Carl Franklin: Now, on the other side, what’s the benefit of using an Enterprise Service Bus? What’s the touted benefit anyway?
Jim Webber: The touted benefit is that you get a single integration framework for all of your needs and as a developer, you can forget about all the hard stuff that we developers can’t tackle because we demo something like security and reliability and those things are out of reach of your average developer so forget about them. There is magic here that takes care all of that for you-a single framework for all of your integration needs and for the developer, that is it. You give him this software, you plumb into it, magic happens, and you don’t have to worry about it. Up to the CIO level, similar kind of magic occurs so you finish your round of golf with your favorite vendor. You hand over several million dollars and he will tell you that all of your integration problems are gone because the bus takes care of them all. That is just an incredible growth service simplification and in those situations, I can’t help but think that my CIO has really neglected his duty of care to the enterprises longevity.
Richard Campbell: So, it seems to me with enterprise service bus, it almost sounds like on one hand you could say that this is the total manifestation of SOA and on the other hand, you could say this is moving away from some of those principles because we get to this much more closed architecture. It seems to me that SOA implementing a lot of the WS*- standards is a very open form of what could be called Enterprise Service Bus-just bringing all of those different resources to a common bus. If that bus is Web services with sort of standard rules, the WS-* style rules for defining a transaction in the security role and so forth, you got a very open enterprise service bus.
Carl Franklin: Just to add to that Richard, I think what we get out of the box, and Windows Communication Framework goes pretty far to helping deal with those issues that Jim was just talking about like security and integration, and sure, it makes it easy to do the “plumbing code” that is supposedly so difficult.
Jim Webber: Absolutely. I have the utmost respect for WCF because it really tried to do a good job in this arena. One thing about WCF and its friends over on the Java side of the world-it gives you a very protocol-centric viewpoint into services. If you look at the metadata and the WCF producers when you deploy a service, it tells you about the messages the service can consume and produce. It tells you about the transport that it binds to. It gives you a whole round of metadata around security and reliable messaging and so forth. The nice thing about that is it is open. It is often being called WS Fabric to distinguish from the ESB-style approaches and it provides a completely neutral fabric called for sending messages around your enterprise and indeed more broadly. It does kind of do the same stuff that an ESB does for you but it does so in a way that doesn’t lock you in for a particular piece of middleware. In fact, for a given service, I’m going to make a decision to deploy typically my favorite platform because it’s least bad Web services form is WCF. So I want to deploy my service and the other WCF service and had it take care of its own plumbing. It is important as it ties back into the incremental stuff we spoke about earlier. If I’m deploying just enough integration middleware to support my service, I don’t need to deploy any more. So I don’t need to take a big upfront gamble on deploying a bunch of wide infrastructure like an ESB because I’m just delivering incrementally my integration infrastructure as part of my normal day-to-day delivery of business process automation. This also makes sure that integration is not an afterthought-that there is no notion of an integration team coming along after the fact-and sweeping up as I’m building my business logic. I’m building up the messages that the business logic emits and consumes. I’m capturing that in my WCF contract and I’m showing it around my SOA ecosystem.
Carl Franklin: So it sounds to me like you’re sort of down on ESB and pro WCF.
Jim Webber: Yes. I think I want to love WCF. I really do because it really did a good job of separating out a bunch of concerns that weren’t separate in ASMX Web services and early Java Web services and so on. So the notion of a contract being a first class citizen in WCF really warms my heart.
Carl Franklin: I hear a however coming.
The conversation continues online at www.dotnetrocks.com/default.aspx?showNum=337
