To many of us, it appears that every move toward ma king technology simpler has been matched by a corresponding move toward increased complexity. This is a prime paradox of IT today: on the one hand, technology for the business user has become dramatically simpler as end users have become shielded from complexity; on the other hand, the actual development of systems architectures and business solutions has become far more complex.

Distributed computing environments and architectures that span the enterprise have meant that our IT work is no longer a point solution for one department or division of a company. In most cases today, a systems development effort takes place with greater expectations that it will have a significant impact on the enterprise’s book of business.

Where there is greater potential impact, there is also greater potential risk. Companies are making substantial investments in their technologies; they expect to see business value from that investment quicker; and they expect that the solution that is delivered will be robust enough to serve as a transition platform as technological change continues to compress years into months.

GROWTH IN PROJECT SCOPE

The new complexity of systems development can be viewed in several ways. First, one finds that more people are now involved in development than one saw either in the mainframe development days or in the early years of client/server. Frequently, projects today may involve anywhere from 100 to 500 people, and this figure will continue to increase until thousand-person projects become common over the next several years.

Second, the number of years required to develop the more complex business solution has also increased. Enterprisewide solutions, delivered over several releases, may require three to five years, or more, to bring all aspects to fruition. This, in turn, adds additional complexities. For example, with longer development periods, the chances are good that management may go through at least one change during the course of the development project. If the project has not been careful to communicate and gain sponsorship at many different management levels, a change in management may put the investment at risk.

The New Development Environment

This new systems development environment is what this author’s firm has come to call “large complex systems” (LCS). It is an environment where the solution:

-   Requires many years to develop

-   Requires a hundred or more people to be involved

-   Is expected to have a significant business benefit

-   Has both a high potential value and a high potential risk

Management Strategies

The author has recently completed a year-long field review that looked in some detail at some of his firm’s largest complex systems development efforts. Based on a set of going-in positions about the challenges of such efforts, extensive interviews were held with personnel at many different levels of the projects. From these interviews, definite repeated patterns about these projects began to emerge, and it became clear that it is possible to set forth a number of factors necessary for a successful implementation of a large complex systems effort. Although a full treatment of all these factors is outside the scope of a single article, the focus here is on several that have to do with new ways of leading and managing a large complex systems effort :

-   Business vision

-   Testing and program management

-   Phased-release rollout plan

BUSINESS VISION

A vision of a new way of doing business that will result from the large complex systems (LCS) development is critical to the success of the LCS. Although intuition, as well as prevailing business management thinking, would indicate this is so, it was important to see the real benefit of business visions played out.

For example, one major project studied was one for a global stock exchange. There, the business vis ion was an integral part of the project — a crisp articulation of the eight essential capabilities that the final system was to provide. It was integrated into the project training and displayed in all essential documents of the project. Most important, all projects within the larger engagement had to be tied back to the vision and justified according to how they contributed to the realization of that vision.

Another development effort at a national financial institution also began with a business vision and a rollout plan that clearly delivered the long- term vision in a sequence of steps. The business vision and rollout plans have served as the basis of work since they were created. The vision deals with the concept of a “model bank” — a consistent set of processes and systems that permits customers around the country to get a standard high quality of service, and also permits employees to move within the company without having to learn new processes. The vision is owned by the senior management of the bank, and communicated to all employees in powerful yet simple ways.

Changes in management can frequently have a negative effect on the power of a business vision. For this reason, it is essential that the business vision be held by more than one individual. On a large complex system built in the U.K., for example, there were several management personnel changes over the years required for the complete development. The key to success here was ensuring that at any given time, there was a set of senior management personnel committed to the effort. As natural career progression took these people to other roles, there was always another core set coming in who continued to own the vision and push it forward.

TESTING AND PROGRAM MANAGEMENT

An LCS effort consists of a set of projects with many interdependencies. Many of these interdependencies can be rather subtle, but all of them must work. The traditional approach for determining whether “things work” is systems testing.

Everyone knows that traditional testing works well for a single application. However, for an LCS with many projects and many interdependencies, the systems testing approach comes up against some real limitations. Experience in these efforts is showing that it is not reasonable to expect a single project leader to define, design, and execute all the tests that are needed to verify that the LCS works as a whole. It is not the responsibility of individual projects to test the LCS release as a whole. An architecture group typically does not have the applic ation skills and user contacts to undertake the testing. In other words, the testing of a large complex system as a whole, using traditional approaches, is an undertaking that has no clear owner.

This creates a dilemma. Program management is not positioned to underwrite the quality of the timely delivery of an LCS effort. Individual project leaders cannot be expected to underwrite the quality of the LCS as a whole. At most, they can underwrite that their project works with its primary interfaces. An intense need today, therefore, is to develop the means to underwrite the quality and timeliness of an LCS release as a whole.

In practice, successful LCS efforts have found a way to resolve the dilemma. This approach, as it turns out, is actually a synthesis of program management and what is called the “V-model” testing strategy. This new synthesis in LCS engagements is called engineering management.

Engineering management adds a testing responsibility to traditional program management. This testing role is charged with validating and verifying that the LCS effort works as a whole, as a system of systems, to meet user expectations of a release of an LCS. For example, it will test that when all the online applications are running as a whole, online response time, reliability, and availability meet servicelevel agreements (SLAs). Individual projects can be expected to have confirmed that they meet their SLAs. The projects often, however, cannot confirm that they continue to meet SLAs when the entire LCS release runs. They do not have access to the rest of the LCS release. They may find it difficult or impossible to create a high transaction volume with multiple LCS applications running in a production-like environment.

PHASED-RELEASE ROLLOUT PLAN

Finally, one of the critical success factors with LCS development involves a move away from a single -release rollout strategy and toward a phased- release plan. Only one of the projects reviewed had attempted to use a big- bang strategy and, even in this case, it became apparent that the approach would prove problematic as the conversion approached. The effort encountered significant delays as it reworked the release plan and moved instead to the view that a phased release was most desirable.

The remaining projects followed a phased delivery. The phased-release approach serves a number of functions:

- Reduced risk. Using a number of releases with partial functionality can reduce the risk of implementation. At the global stock exchange, for example, initial discussions with the company’s management were key in moving from the riskier single -release approach and toward a phased rollout, although the phased approach appeared to delay the benefits of the system. The balance was the reduced risk of achieving the benefits.

- Early verification. A phased approach permits early verification by business users of essential components as they work with the system in their business. From the review work conducted, it appears that a first release tends to take from 18 to 24 months. Subsequent releases tend to occur in the range of 6 to 12 months after earlier phases. This is in contrast to a more typical three- to five-year development that a single -release approach may require. The result of the iterations is that the user has worked with the system and provided verification of the value of the system.

- Ability to make mid-course corrections. Inherent in the release approach is the ability to review the overall situation as the releases are rolled out and thereby to make mid-course changes. As noted, an LCS effort can go on for many years, during which time a company and its business environment can go through a great deal of change. The release strategy can provide the means to deal with the changes in a controlled manner. It can also address issues in a long systems development where for periods of time the design must be “frozen.”

- Closer user involvement. Business impact can be seen faster when rollouts are provided to the user earlier through iterations. This allows the user to build up experience with and support for the system, rather than facing a sudden conversion at the end of a large development.

The downside of a phased rollout strategy is the significant increase in development cost. To this author’s knowledge, there are no widely accepted estimates of the increase in cost caused by the phased-release approach. However, there have been some evaluations that showed more than a 50 percent increase in costs when a multiple -release strategy was compared to a single release. This would seem to suggest not going with a multiple -release strategy. The trade-off is the very high risk of failure in a single release, combined with the benefits noted above of a multiple release.

The points discussed above are key to understanding the value of a release strategy. Phased releases allow a company to reduce risk, increase buy -in, and build a system that is closer to the company’s business needs. The lower apparent costs may make a “big bang” approach appear desirable, but the hidden costs and greater ris ks may prove unacceptable in the longer term. It is vital that management involved in this decision carefully weighs the costs and risks of either approach.

CONCLUSION

When beginning the review of large complex systems, the author’s first thought was that the most important thing to do was simply to figure out how to eliminate the complexity. Based on the two years of review, the author is convinced that eliminating the complexity is not possible. One needs to accept complexity as a part of the systems development world for the future. The size of projects that affect the enterprise as a whole tends to be large and it will continue to increase. A project that affects the entire enterprise will increase complexity. Only when one accepts the complexity can one come to grips with managing that complexity.

Finally, today’s business environment — with its increasing focus on business partners, virtual enterprises, and the global span of business — makes complexity a reality that cannot be overcome. Delivering quality solutions in this environment must start with a recognition that complexity is inescapable. From that point, then one initiates a set of strategies to manage the complexity and risk. There is no “silver bullet” in these strategies. The three points discussed in this article are examples of such strategies. Each one of them is necessary; but none of them alone is sufficient to guarantee success. From a base of well-defined and -directed strategies, managing the ongoing complexity must become the focus on management in such large complex systems.