A guide to the assessment of software development methods

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

Technical Report

CMU/SEI-88-TR-008

ESD-TR-88-009

A Guide to the Assessment of Software Development Methods

Bill Wood

Richard Pethia

Lauren Roberts Gold

Robert Firth

April 1988

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

Preliminary Report

CMU/SEI-88-TR-008

ESD-TR-88-009

April 1988

A Guide to the Assessment of Software

Development Methods

Bill Wood

Richard Pethia

Lauren Roberts Gold

Robert Firth

Tools and Methodologies for Real-Time Systems Project

Unlimited distribution subject to the copyright.

Software Engineering Institute

Carnegie Mellon University

Pittsburgh, Pennsylvania 15213

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

This report was prepared for the SEI Joint Program Office HQ ESC/AXS

5 Eglin Street

Hanscom AFB, MA 01731-2116

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange.

FOR THE COMMANDER

(signature on file)

Thomas R. Miller, Lt Col, USAF, SEI Joint Program Office

This work is sponsored by the U.S. Department of Defense.

Copyright 1988 by Carnegie Mellon University.

Permission to reproduce this document and to prepare derivative works from this document for internal use is granted, provided the copyright and \‘No Warranty\’statements are included with all reproductions and derivative works. Requests for permission to reproduce this document or to prepare derivative works of this document for external and commercial use should be addressed to the SEI Licensing Agent.

NO WARRANTY

THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN \‘AS-IS\’ BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTIBILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This work was created in the performance of Federal Government Contract Number F19628-95-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at 52.227-7013.

This document is available through Asset Source for Software Engineering Technology (ASSET) / 1350 Earl L. Core Road ; P.O. Box 3305 / Morgantown, West Virginia 26505 / Phone: (304) 284-9000 / Fax: (304) 284-9001 / e-mail: sei@http:// / WWW: http:///sei.html

Copies of this document are available through the National Technical Information Service (NTIS). For information on ordering, please contact NTIS directly: National Technical Information Service / U.S. Department of Commerce / Springfield, VA 22161. Phone: (703) 487-4600.

This document is also available through the Defense Technical Information Center (DTIC). DTIC provides acess to and transfer of scientific and technical information for DoD personnel, DoD contractors and potential con tractors, and other U.S. Government agency personnel and their contractors. To obtain a copy, please contact DTIC directly: Defense Technical Information Center / 8725 John J. Kingman Road / Suite 0944 / Ft. Belvoir, VA 22060-6218. Phone: 1-800-225-3842 or 703-767-8222.

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademark holder. CMU/SEI-88-TR-8

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

Table of Contents

1. Introduction1

2. Context3

2.1. Key Aspects3

2.2. Development Stages3

2.3. Views of the System4

2.4. Classification Scheme5

3. System Characteristics7

3.1. Operational Characteristics8

3.1.1. Functional8

3.1.1.1. Environment8

3.1.1.2. I/O8

3.1.1.3. Data Transformations9

3.1.1.

4. Math Representations of Engineering Phenomena9

3.1.2. Behavioral9

3.1.2.1. Modes and States9

3.1.2.2. Capacity, Workload, and Performance10

3.1.2.3. Human Interface11

3.2. Structural Characteristics12

3.2.1. System Architecture12

3.2.1.1. Distributed Processing12

3.2.1.2. Robustness12

3.2.2. Data Modeling13

3.2.3. Language Platform13

4. Constraints15

4.1. Software Architecture15

4.1.1. Modularity16

4.1.1.1. Size and Complexity16

4.1.1.2. Coupling16

4.1.1.3. Cohesion17

4.1.2. Information Hiding17

4.1.3. Exception Handling17

4.2. Integration and Test Constraints18

4.3. Evolution Constraints18

5. Representations21

5.1. Abstraction21

5.2. Consistency22

5.3. Completeness22

5.4. Complexity22

5.5. Traceability23

5.6. View Integration23 CMU/SEI-88-TR-8i

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

5.7. Ambiguity24

5.8. Duplication24

5.9. Changes24

5.10. Compliance with Standards25

6. Deriving Representations27

6.1. Partitioning27

6.2. Refinement28

6.3. Elaboration28

6.4. Reuse29

6.5. Evaluation of Alternatives29

7. Examining Representations31

7.1. Examination Goals31

7.1.1. Feasibility31

7.1.2. Conformance32

7.1.3. Safety32

7.2. Examination Techniques32

7.2.1. Walkthroughs and Inspections33

7.2.2. Analysis33

7.2.3. Testing34

7.2.4. Data Extraction, Reduction, and Analysis34

8. Management Characteristics35

8.1. Process35

8.2. Cost36

9. Problem Areas39

9.1. Experienced Personnel39

9.2. Transformation Across Stages40

9.3. Feasibility Analysis41

9.4. Development Constraints42

9.5. Large-Scale Problems43

9.6. User Interface43

9.7. Implementing Designs44

10. Conclusion47 ii CMU/SEI-88-TR-8

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

List of Tables

Table 2-1:Stages of Development6 CMU/SEI-88-TR-8iii

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

A Guide to the Assessment of Software Development Methods

Abstract.Over the past decade, the term "software engineering method" has been

attached to a variety of procedures and techniques that attempt to provide an orderly,

systematic way of developing software. Existing methods approach the task of

software engineering in different ways. Deciding which methods to use to reduce

development costs and improve the quality of produced products is a difficult task. This

report outlines a five step process and an organized set of questions that provide

method assessors with a systematic way to improve their understanding and form

opinions on the ability of existing methods to meet their organization’s needs.

1. Introduction

This is the third report in a series of reports concerned with classification and assessment criteria for software development methods and tools. The first two reports describe guidelines for classifying and evaluating software engineering tools [Firth 87a], and a classification scheme for software development methods [Firth 87b]. This (third) report describes issues in assessing methods for use in the specification and design of real-time software systems. Throughout this report, the term "development" is used in the restricted sense of specification and design.

Over the past decade, the term "software engineering method" has been attached to a variety of procedures and techniques that attempt to provide an orderly, systematic way of developing software. Existing methods such as Design Approach for Real-Time Systems [Gomaa 86]; Problem Statement Language/Problem Statement Analyzer [Teichrow 77]; Jackson System Design [Cameron 83]; Object-Oriented Design [Booch 83]; and Structured Analysis, Structured Development with Real-Time Extensions [Ward 85] (see [Firth 87b] for additional examples); approach the task of software engineering in different ways and prescribe different techniques. Some, such as Distributed Computing Design System [Alford 85], attempt to cover a broad range of activities while others, such as Statecharts [Harel 86], focus on particular areas that traditionally have been especially troublesome.

An assessor of methods is basically concerned with answering the question: What is a good software engineering method that my organization can use to reduce development costs and produce quality products? Answering this seemingly simple question is, however, a difficult task given the diversity of existing methods and the complexity of software engineering. In addition, methods alone provide no guarantees of productivity improvement or of high quality in the software product. Methods, procedures, and techniques are valuable only if those who use them understand their underlying concepts and apply them with thought and judgement.

The rest of this report describes a five step process (outlined in [Firth 87b]) to be used in evaluating methods. The five steps are:

1.Needs Analysis- Determine the important characteristics of the system to be

developed and how individual methods help developers deal with those CMU/SEI-88-TR-81

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

characteristics. Chapter 3 emphasizes the need to evaluate individual methods by

their ability to deal with specific engineering problems.

2.Constraint Identification- Identify the constraints imposed on the permitted

solutions and determine how individual methods help developers deal with those

constraints. Chapter 4 reminds the assessor that methods must support

developers’ efforts to design systems that exhibit a variety of required

characteristics.

http://er Requirements- Determine the general usage characteristics of the individual

methods. As described in Chapter 2, a method can be examined by developing an

understanding of: how it represents a system under development, the guidelines it

gives developers to derive the representations, and the guidelines it provides to

examine the representations. This understanding is best developed by applying the

method to a sample problem that is representative of the system to be developed.

Chapters 5, 6, and 7 discuss representations, deriving representations and

examining representations in turn.

4.Management Issues- Determine the support provided by the method to those who

must manage the development process as well as the costs and benefits of

adopting and using the method. Chapter 8 emphasizes the need to recognize that

methods are used within particular organizations that have established ways of

conducting business.

5.Introduction Plan- Develop an understanding of the issues that the method does

not address and a plan to augment the method in areas where it is deficient.

Chapter 9 reinforces the notion that methods do not guarantee success and points

out several problem areas in existing methods and their use.

Chapters 3 through 9 of this report each contain an organized set of topics supported by a list of questions for each topic. The questions have the following characteristics.

1.Some are rhetorical and do not require answers. These serve to remind the

assessor of important software engineering issues and the need to form opinions

on how individual methods address the issues.

2.Some require an understanding of the method alone and emphasize the need to

examine individual methods thoroughly.

3.Some require that the method be applied to a problem or set of problems;

demonstrating the need to match the method to the engineering problem at hand.

4.Some require that the assessor form opinions on the needs of those involved in the

development process and how they would use a method to satisfy those needs.

These emphasize the need to use selected methods with judgement and

understanding.

There has been no attempt to reduce the evaluation of methods to a completely objective process supported by a list of questions with quantifiable answers. To do so would oversimplify what is and must be viewed as a difficult task in which a large number of tradeoff decisions must be made. By using the framework and questions provided in this document, method assessors will have a systematic way to improve their understanding and form opinions on the ability of existing methods to meet their organization’s needs.

2CMU/SEI-88-TR-8

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

2. Context

This chapter contains a discussion of a framework for the comparison of methods. The framework is introduced here to lend focus to the remainder of the report and to introduce terms that are used in later chapters. The key aspects of the methods are discussed first, then a two dimensional method classification scheme is introduced. A more detailed discussion of methods is available in [Firth 87b].

2.1. Key Aspects

In general, a method is "a systematic procedure, technique, or mode of inquiry employed by or proper to a particular discipline or art [Webster 81]." When applied specifically to software engineering, it could be defined as a systematic approach to providing a software solution. Ideally, the method should cover all aspects of the problem and, in context of software engineering, should lead from and initial (imperfect) set of requirements to a satisfactory implementation, passing systematically through intermediate stages.

At all stages in the development process, a representation of the system must be created. It is important to understand how the method contributes to this process. A detailed understanding of individual methods can be obtained by examining each method from three points of view, namely:

1.What is the content and form of representation of the artifacts dictated by the

method? Desirable characteristics of representations are discussed in Chapter 5.

2.What procedures or techniques does the method provide for deriving the

representations? Chapter 6 addresses the issue of deriving representations, and

discusses five major areas where a method should provide guidance.

3.What procedures or techniques does the the method provide for examining the

representations? Representations are typically examined in a variety of ways for

several reasons. Chapter 7 discusses examinations and elaborates on the goals of

examinations and the techniques for conducting examinations.

2.2. Development Stages

One axis of the classification scheme deals with the software development process and a large-grained view of development stages. The different development stages are enumerated below. Each development stage is characterized by what it represents and the way it represents it.

1.The requirement is a description of what the end-user audiences view as their

needs and is often a rather eclectic description. It usually covers the needs of each

audience in the end-user community in a very uneven manner, with some aspects

(such as ease of installation) often overlooked. It often describes some functions

(such as a scheduling mechanism) in very general terms, but others (such as a

communications protocol) are discussed thoroughly. There are often important

requirements that are not addressed, and they need to be exposed and handled as

they arise. The earlier these issues are addressed and resolved, the better the

prospect of delivering a high-quality product within budget and schedule.

CMU/SEI-88-TR-83

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

2.The first step is to take the ambiguous, incomplete, and inconsistent requirement

and turn it into an almost flawless specification. This is not yet possible with

current technology, but there are many reasonable ways of proceeding that give a

serviceable specification. The specification describes what the software is to do

and the constraints to be imposed on the designers. It should be noted that

production of the specification is not limited to a front-end activity, but that the

specification will change throughout the life cycle of the system.

3.The design representation describes how the system is structured to satisfy the

specification. It describes the system in a large-grained manner and defines the

breakup of the system into major tasks. It describes persistent data objects and

their access mechanisms, the important abstract data types and their encapsulation

in the heavyweight tasks, and the message structures between the tasks. There

must also be some consideration for how the resources are to be allocated and how

the performance requirements are to be satisfied.

4.The final development stage is implementation with source code, object code,

resource usage, and initialized data structures. This is the level at which algorithms

are represented explicitly.

2.3. Views of the System

The second axis of the classification scheme deals with various views of the system. As suggested in [Harel 86] these views are needed to describe the system’s intended and actual operation. The views are relevant at each stage of the system’s development and are enumerated below.

1.The functional view shows the system as a set of processes operating on data.

This includes a description of the task performed by each process, the flow of data

between processes, and the underlying math model, if required. The functional

view is often the starting point for the design process, since it deals with what the

system is supposed to do, relates the system to its environment, and focuses on

the needs of the key participants in the development process: customers, users,

and developers.

2.The structural view shows how the system is put together: the system’s

components, the interfaces between them, and the distribution and flow of data and

control between the components through the interfaces. It also shows the

environment and the interfaces and information flows between it and the system.

Ideally, the structural view should be an elaboration of the functional view. Each

entity in the latter view is decomposed into a set of primitive software components

that can be implemented separately and then combined to build the entity. The

design process therefore generally converts a functional view into a structural view.

However, the structure of a system is influenced by resource constraints that

prevent the use of arbitrarily many or arbitrarily large components. The structure is

also influenced by certain implementation constraints that require the use of specific

types of component (e.g., MIL-STD-1750a processors), or require that components

be connected in a specific manner (e.g., by MIL-STD-1553 buses). The structural

view should include a definition of the number and dimensions of entities to allow

for resource estimates.

3.The behavioral view shows the way the system will respond to specific inputs:

what states it will adopt, what outputs it will produce for each combination and time

sequence of inputs and state transitions, what boundary conditions exist on the

validity of inputs and states. This includes a description of the environment that is

4CMU/SEI-88-TR-8

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

producing the inputs and consuming the outputs. It also includes constraints on

performance that are imposed by the environment and function of the system.

Real-time systems especially have performance requirements as an essential part

of their correct behavior. The behavioral view should include a definition of the

expected workload and the required responses of the system to this workload.

Ideally, the behavioral view should complement the functional view. Each transaction in the functional view should be traceable through the system from the initial input through the interfaces and functional units to the final output.

2.4. Classification Scheme

The classification scheme shown in Table 2-1 uses the system stages (specification, design, implementation) on one axis, and the views of the system (behavioral, functional, and structural) along the other axis. The requirements stage is not included, since it is informal, and the authors of this report believe there is little to be gained by including it. A high-level classification of a method can be obtained by determining if it supports the activities of a particular stage and, for each stage supported, whether or not the method provides a means to represent each of the three views. In addition, as discussed in Chapter 3, the same framework can be used to identify the operational characteristics of a system to be developed.

CMU/SEI-88-TR-85

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

6 CMU/SEI-88-TR-8

Specification Design Implementation Functional Structural Behavioral

V i e w s o f t h e S y s t e m

Table 2-1: Stages of Development

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

3. System Characteristics

It is important to consider the characteristics of a system when choosing a method to use in its development. A method that is suitable for one system may be unsatisfactory for another. For example, real-time systems deal with the processing of data by a computer in connection with another process outside the computer according to time requirements imposed by the outside process [IEEE 83]. "Real-time systems typically sense and control external devices, respond to external events, and share processing time between multiple tasks. Processing demands are both cyclic and event driven in nature [Fairley 85]." Real-time systems are very different from, for example, batch oriented, data processing systems where the rate of data input and output is controlled by the system rather than by an external process. Methods suitable for data processing systems need not model an external process or the unique characteristics of devices that sense and effect the environment. Methods for use in the development of real-time systems must model both.

One of the first steps in evaluating a method is to characterize the operational system that is to be built and to understand how the method will help developers deal with each of its characteristics. Using the framework discussed in Chapter 2, the characteristics of an operational system can be well described by considering three views of the system: the functional view, the structural view, and the behavioral view. During the development process, the functional and behavioral views are developed and gradually transformed to the actual software represented by the structural view.

Our discussion of these views focuses on the ability of methods to represent the views, analyze the representations, and provide guidance to developers in deriving the views. General, system-independent characteristics of methods and their representations are discussed in following chapters.

Beginning with this framework, the first questions to ask when evaluating methods for requirements analysis and design are:

1.Does the method allow the representation of all three views?

2.Are the views complementary?

•Can there be integrated views of function and behavior as well as

independent views?

•Are there suggested techniques or rules for deriving the structural view from

the functional and behavioral views?

The discussion on views is divided into two major sections. First, the functional and behavioral views are discussed in the section on operational characteristics. Second, the structural view and the constraints often placed on software structure are discussed in the section on structural characteristics.

CMU/SEI-88-TR-87

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

3.1. Operational Characteristics

The operational characteristics of a system primarily describe what a system does (functional view): what functions it performs, what input the system receives, what output the system gives, and the relation between the inputs and outputs. In addition, operational characteristics also address when the functions occur (behavioral view); which functions occur under which conditions, and how quickly inputs are transformed to outputs.

Each of the following subsections deals with one of these views and lists evaluative questions that should be asked of a method when considering the view.

3.1.1. Functional

The functional view shows the system as a collection of processes operating on data. This view deals with processes; data flows, including system inputs and outputs; and data stores. The functional view is discussed below in terms of the environment in which the system operates, the system’s inputs and outputs, and the data transformations performed by the system’s processes.

3.1.1.1. Environment

An important characteristic of any system is its relationship to the environment in which it operates. Specific systems are built to operate in specific environments and these environments differ dramatically across different types of systems. Characterizing the environment involves specifying the entities with which the system interacts: users, devices that sense and effect the environment, communications channels that connect the system to other systems.

1.Does the method provide a representation that clearly draws a boundary around the

system and separates it from its environment?

•Does the representation clearly identify the specific entities that the system

interfaces with?

•Does it provide a mechanism to detail and describe the interfaces between

the system and these entities?

3.1.1.2. I/O

Another system aspect that is dealt with early in the analysis process deals with the characteristics of the system’s individual data inputs and outputs. Initial analysis work often begins with a hazy picture of the inputs and outputs. In other cases, inputs and outputs are well defined and detailed since the system must interface with existing devices.

1.Does the method allow the representation of data that flows across these interfaces

using an appropriate level of abstraction?

•Are abstract representations available to describe data flows that are to be

detailed by analysts and designers?

•Are detailed representations available to describe data flows that must meet

rigidly defined interfaces to existing devices or must conform to established

communications protocols?

8CMU/SEI-88-TR-8

The ideas and findings in this report should not be construed as an official DoD position. It is published in the interest of scientific and technical information exchange. FOR THE COMMANDER (signature on file)

3.1.1.3. Data Transformations

Characteristics of individual data transformations (processes) performed by the system must also be considered when selecting methods. Different systems or different parts of an individual system require processes that differ in size, complexity, and dynamic behavior. Development methods must deal with the characteristics of the processes required in the system.

1.Does the method provide a technique to represent each process including its

inputs, outputs, functions, and the exceptions that it may raise?

2.Does the method provide a representation (decision tables, for example) that allow

analysts and designers to define and detail complex logic when required?

3.1.1.

4. Math Representations of Engineering Phenomena

Many of the results of engineering studies associated with systems are represented as mathematical models of the system operation. These models may represent the kinematics of motion of relevant objects, detail the processing of received signal data, or describe the scheduling algorithms used to service asynchronous requests for scarce resources. These models are derived from engineering studies, and simulation is often used in the development and validation of the algorithms. Unfortunately, the models are often incomplete, are developed somewhat independently of the final target environment, and must be carefully incorporated into the software design. If the algorithms themselves are still under development during the software design, the method must accommodate this process and provide aids to minimize the associated risk.

1.If mathematical algorithms must be devised, does the method provide a

representation that is familiar to the algorithm developers and can be understood by

the algorithm implementors?

•Do the representations allow description of the precision and accuracy or the

calculations dictated by the requirements?

•Do the representations allow specification of functionality under adverse

conditions such as loss of data or single sensor failure?

2.Does the method provide techniques to refine and validate the algorithms over

time?

3.1.2. Behavioral

The behavioral view shows the way the system will respond to specific inputs: what states it will adopt, and what outputs it will produce for each combination and time sequence of inputs and state transitions. The behavioral characteristics of individual systems vary considerably and an effective development method must deal with the required characteristics of the system to be developed. Behavioral characteristics are discussed below in terms of: modes and states; capacity, workload, and performance, and human interfaces.

3.1.2.1. Modes and States

Real-time systems are, to a large extent, event driven and a system’s response to any particular event is dependent not only on the event itself, but also on a variety of conditions that have occurred prior to the event. A system’s operational state represents an externally observable mode of behavior [Ward 85] where the system’s response to an event or set of events differs

CMU/SEI-88-TR-89