Purpose

Whether for software, hardware, or process, a comprehensive test plan, as the name implies, is designed to guide an exhaustive test of the system in question. It identifies the potential areas of risk and explains how the system will be evaluated to determine its susceptibility to problems. The test plan also delineates how the test results will be validated. As test results are received, they are amended to the test plan so the process and its results can be evaluated in toto.

Application

Because a comprehensive test plan expresses a process to be used and then, later, is married to the results of that process, it is an evolutionary document. It grows as the available information grows. Initially, the comprehensive test plan serves as a road map for test conduct and validation. Later, it serves as a repository for the outputs of the tests, as well. As such the document is essential to those who will be conducting the tests and may be used as a defense for the costs of the tests or the approach.

Content

A comprehensive test plan includes a history and step-by-step process guidance for the evaluation.

Background.    The rationale for the tests is normally the first component of a comprehensive test plan, explaining why the tests were deemed necessary and what environmental considerations led to the primary approach(es) being considered.

Testing approach. The actual technical approach to the testing should be defined in detail, including any materials required, the range of parameters the test will evaluate, specific areas to be tested, and indicators or metrics to be tracked. In addition, this may include testing of individual components of the system, as well as integrated systems.

Validation. Any validation processes, whether internal or independent, should be defined as objectively as possible. The validation processes should address the specific components being tested, as well as any integrated systems.

Outcomes anticipated. Some test plans (not all) will include some explanations of the outputs anticipated from the testing process, so the anticipated outcomes can be compared with the actual outcomes. This should not be used to guide the testing outcomes, but should be used to identify if the testing process spelled out in the plan will actually produce outcomes of the type desired.

Outcomes tested. As stated earlier, the test plan is evolutionary. This information will only be incorporated after the testing is completed. This information may be incorporated in stages as preliminary and final testing may span several weeks, months, or years.

Conclusion(s). Based on the tested, validated outcomes, any conclusions that can be drawn based on the system, the testing process, the validation, and the anticipated outcomes should be expressed as conclusions. The conclusions should be part of the comprehensive test plan for the organizational archive.

Approach

Unlike other documentation where there may be heavier and lighter versions of the documentation, the comprehensive test plan is, by its nature, as exhaustive as possible. Every element of information that can be included in the test plan should be included to preclude misunderstanding or misapplication of the testing process. Also, while some plans by their nature incorporate some measure of variability,  that  does  not  apply with a test plan. Test plans are relatively rigid, and if the process is to be changed, the test plan documentation should be changed as well.

Considerations

The comprehensive test plan can apply to materials, systems, processes, hardware and software, or integrated systems thereof. For some of those tests, the results are purely quantitative, and variance is easily detected. For processes, however, testing can become speculative if the outputs of the process are not clearly identified in a way that can be objectively measured. Subjective measurement leads to inadequate testing, as the “hard” metrics are lost. If no concrete measures can be found for a process testing evaluation or validation, such shortcomings should be highlighted in the test plan, so the results are not misconstrued as hard fact.