SOA may have seemed the saviour of bad software architecture and poor development project planning, but the reality is that it's a complex and difficult venture. Thus, the number of failed SOA projects is about equal to the successful ones.
But key patterns are emerging from the successful SOA efforts, patterns that can help determine whether an SOA project is a failure or a success.
The fundamental cause of SOA failure is the lack of good people working the architecture, from leadership to staff. This lack is not of numbers, but of knowledge and the vision to drive change.
Critical to SOA success is the presence of a leadership team that values an investment in infrastructure and understands the long-term value of being more agile and efficient, and, better yet, is willing to make the investment. SOA is not cheap. Indeed, it's a huge systemic change in how you build, deploy, design, test, and architect your information technology. It's an investment that goes well beyond the millions of dollars required to acquire the necessary training, consulting, and then the technology required to make the changes.
The SOA investment cannot be treated as a one-time "transformation" project. Instead, you need to consider SOA as a journey, not a project, and — for the purposes of budgeting -- treat it as a series of projects that make up the journey. The SOA effort does need its value clearly defined, and the investment and commitment to achieve that value made up front.
Because SOA is really about a journey, not a project, organisations need to take a long-term view of SOA. It's a multi-year and typically multi-million-dollar commitment to drive a systemic change to the core IT mechanism. But most SOA "projects" are stopped at some point, typically due to budget issues or to the need to redirect resources at some tactical short-term need. Thus, the SOA never had a chance; there was no follow-through. Executive and IT management must not allow this.
SOA also involves two fundamental business changes that IT historically doesn't have the clout to make happen: One is the sharing of processes across political fiefdoms that fear loss of control. The other is that SOA requires a rethinking of fundamental processes, which is not only hard but also challenges established practices, resource allocations, political power, and so on.
At the management and staff levels, the people requirements for successful SOA are even more fundamental. Although many would love to have the existing team drive them to the new world of SOA, the harsh truth is that many in that team won't have the skills or the aptitude. You need to make some hard decisions up front as to who will take you forward. This means you must replace people or you must augment staff. Either approach is costly.
Many companies employ mentors or consultants to help learn the new ways of SOA. Others invest in SOA training, or even bring in entirely new teams as either consultants or employees to assist in "the change".
No matter what you do, never, ever try to drive SOA with the wrong people; it just won't work.
Approaching SOA means changing how you approach architecture and systems development. In the past, many companies have approached systems by dragging whatever seemed cool at the time into the enterprise to solve a tactical problem. Of course, tactical problems lead to other tactical problems, and these companies kept digging a deeper hole in terms of architectural complexity and efficiency.
Thus, you need to approach SOA using a well-thought-out, practical process that lets you break the architectural domain down to a functional primitive, then build it back up again as a SOA. Unless you're the smallest of enterprises, you also need to break these domains into achievable chunks that can be worked on in sequence, or later in parallel.
Then it's time to do some real work, including looking at the information within the problem domain, meaning application semantics and metadata. This takes a ton of work, because most enterprises typically don't have a good semantic-level understanding of their enterprise systems. Thus, this effort is usually not a process of reviewing common artifacts, but of creating new ones. Many SOA efforts skip this step, which cripples what they end up delivering.
From the information step, you move to the services: defining existing and new services that will make up your SOA. This is all about figuring out what you have, determining what you need, and then normalizing the services into a usable and well-defined grid. You need to make sure you define and document the services, and then link them back to the metadata model you've created.
Keep in mind that most of your services will be existing services, externalised through some sort of middleware mechanism. Most people think SOA is about building all-new services, but that's almost never the case. Many SOA efforts fail because they become excuses to reinvent the wheel, not solve pressing business needs.
You then need to define and create processes that bind the services together into business solutions. There are three approaches here you can use. First, you can create service-oriented business applications, a way to programmatically bind services together into processes or applications. Second, you can leverage an orchestration layer, such BPEL, to bind services together to form business solutions. Third, you can leverage traditional process integration tools to bind services into solutions. No matter what approach you use, you need to make sure to include requirements as well as design.
The "A" in SOA is architecture. Most people forget that, opting instead to jump right into the technology. Why? Because architecture requires a huge effort and is boring, whereas playing with cool technology is fun and fast. Unfortunately, you really have to spend the time to plan and lead the enterprise in the right directions. Create a strategy, processes, architecture, links to business realities, budgets, and then -- the most important action -- lead the technology teams in the proper directions.
Although architecture is fundamental to SOA, it is often neglected in so-called SOA efforts. That's because most enterprise architects within the larger enterprises don't have the authority and reach they need to drive the technology in the right directions. In most cases, they try only to influence and inform, and are more "ivory tower" guys when you get right down to it. This unrespected, powerless approach to architecture won't drive change in the long-term unless there is true dynamic leadership to back them up.
But if you focus on the "A" in SOA, you are much more likely to succeed. Being architecture-focused means driving changes in how the IT department thinks about technology -- having it do so in the context of architecture, and never with the technology being the de facto architecture. This is a significant cultural shift that most IT staff won't make unless forced to.
There is no one-size-fits-all SOA technology solution. You have to understand your requirements and match the requirements with the right technology. It's that simple.
But most companies that build SOA mistakenly lead with technology. They go out and purchase an ESB or some other SOA-in-a-box solution with the hopes that it will solve all their problems. The end result is a technology solution that's mismatched to their real issues and thus does not work. The SOA effort fails.
The process of matching the right technology with the problems that need to be solved is very simple, but it's also time-consuming. The process is to list your requirements down one side of the page, and list the candidate technology down the other. From there, create a short list of technology that's practical to test. Next, work with the vendors to bring the technology in and test its form and function. Does it do what it should do, and under the stresses that the SOA will place upon it?
Sure, it's tempting to call one of the big stack players and just leverage everything they have in their portfolio. Whereas some portions of such a technology suite will indeed be a good fit for your SOA, the rest of the stack will be the wrong match for your requirements. Accepting wrong technology for the convenience of a suite or single vendor relationship leads to SOA failure. Instead, in most cases the right technology solution for SOA will be a mix-and-match kind of exercise with many different vendors and technologies.
Make sure your architecture is technology-independent. It might sound easier to build architecture around a particular technology, as some vendors promote, but remember that technology is always in a state of change, whereas the core notion of the architecture needs to remain fairly consistent. You need to work the architecture out first, and then be concerned with the technology that supports it, understanding that the technology will change.
SOA success is not rocket science; it's really a matter of good old IT disciplines that focus on all dimensions, including the people, processes, technology, and the business. Organisations that fail did not resist the temptation to chase the "quick and easy" technology-oriented solutions, and they did not consider the fundamentals.