Agile Software Development Methods

Gary Mrenak, Territa Poston, Jeffrey Schein

Contents

Origins and Motivations

Non-Agile or “Plan-Driven” Methods

Agile Methods are a reaction to traditional methods of Software Development, the so-called “Plan-Driven” methods. One of the earliest and best examples of “Plan-Driven” methods is the “Waterfall” method. Although there are different implementations of this method (for example DoD-Std-2167), all are similar to the one that [Schach, 2002] describes with a requirements phase, a specification phase, a design phase, an implementation phase, and a maintenance phase.

Characteristics of Plan-Driven Methods

Plan-Driven methods have characteristics that starkly contrast with those of Agile methods, and they revolve around documentation and development milestones.

Plans/Documentation

“Plan-Driven” methods, as the name implies, usually involve formal and continuous planning and depend on documentation to express and communicate the resulting concepts and decisions. [Schach, 2002] describes the Waterfall method as a “documentation driven” method where “..no phase is complete until the documentation for that phase has been completed and the products for that phase have been approved by the SQA [Software Quality Assurance] group.” The requirements and analysis phases of the method support the planning effort for the project and produce a specification document and a software project management plan. Once approved by the client, the specification document becomes the basis for the design phase which produces architectural and detailed design specifications.

A concrete example of a Plan-Driven method with a strong emphasis on plans/documentation is Dod-Std-2167, Defense System Software Development. Although now formally obsolete and long superceded by less prescriptive methods, the purpose of this standard was to “.. establish requirements to be applied during the acquisition, development, or support of software systems” [BMP 2002]. It described a comprehensive document-driven process that specified the required activities of a substantial software development process:

              •            Systems requirements analysis/design

              •            Software requirements analysis/design

              •            Preliminary design

              •            Detailed design

              •            Coding and CSU (computer software unit) training

              •            CSC (computer software component) integration and testing

              •            CSCI (computer system configuration item) testing

              •            System integration and testing

To support this process, developers were required to “..use systematic and well documented software development methods..” while performing each of these activities; they were required to “..document and implement plans for the installation, configuration control, and maintenance of each item of ..” the software development environment that they would establish; they were required to produce a Software Development Plan, Software Requirements Specification, and an Interface Requirements Specification; they were required to document and implement design and coding standards; they were required to document the development of CSUs, CSCs, and CSCIs in Software Development Files, one for each CSU, CSC, and CSCI, as well as document and implement procedures for establishing and maintaining SDFs, and they were required to produce a Software Product Specification for each CSCI. And there were similar requirements for Formal Qualification Testing, Software Product Evaluations, Software Configuration Management, and Transitioning to Software Support.

To underline the importance of documentation to the process, DoD-Std-2167 specified Data Item Definitions, templates prescribing the contents of each individual required document. Table 1 is a list of the documents specified by DoD-Std-2167 and their associated Data-Item-Descriptions.

This long list of documents and document content definitions is impressive. It is easy to see why [Schach, 2002] describes the software development process prescribed by DoD-Std-2167 as “document driven.”

Milestones

“Plan-Driven” methods use milestones (decision points) to both synchronize client and developer understandings and establish a point of common departure for the next phases and iterations of development. In DoD-Std-2167, formal milestones were prescribed by another standard, Mil-Std-1521,Technical Reviews and Audits for Systems, Equipments, and Computer Software, and were tied to the completion of seven of the eight prescribed major phases of the process: to formally complete the System requirements analysis/design phase, developers were required to support both a Systems Requirements Review and a Systems Design Review. To formally complete the Software requirements analysis/design phase, one or more Software Specification Reviews. For the Preliminary design phase, one or more Preliminary Design Reviews. For the Detailed design phase, one or more Critical Design Reviews. For CSC integration and testing, one or more Test Readiness Reviews. For CSCI testing and systems integration and testing, one or more Functional Configuration Audits and Physical Configuration Audits.

Documents are the currency of these milestones. That is, the end of one phase of development and the beginning of a subsequent phase directly depended on the production and approval of documents. For example, to complete the software specification phase in DoD-Std-2167 required the production of a Software Specification, and to begin the subsequent Preliminary design phase required the review and approval of that Software Specification at a Software Specification Review.

Criticisms

[Fowler, M. 2002] summarizes the complaints as follows: “These methodologies have been around for a long time. They've not been noticeable for being terribly successful. They are even less noted for being popular. The most frequent criticism of these methodologies is that they are bureaucratic. There's so much stuff to do to follow the methodology that the whole pace of development slows down.”

Efficacy

“Plan-Driven” methods have been the center of software development philosophy since organized approaches were first used in the 1970's. However, their record of success has been less than resounding. [Suzuki, 2002] summarizes the results of an ongoing study of over 8,000 software projects and 365 respondents by the Standish Group, called the “Chaos Study,” where success was defined as on-time and on-budget:

•            53% of projects were both late and over budget

•            31% were outright failures and cancelled

•            cost overruns averaged 189% of the original cost estimates

•            schedule overruns averaged 222% of the original time estimates

•            61% of originally specified features and functions were available in the released products

Documentation

[Fowler, M. 2002] observes that, largely because of the CASE tools used, software documentation often assumes a “dictionary mentality” where every aspect of the development is treated with equal importance in an effort to ensure that everything has been included. He suggests that documentation too often describes what everything “is” rather than describing what it “is meant to be” and often miring the reader in a sea of details where the essence of the development is hard to extract, particularly in complex systems.

Handling Changes

As characterized by [Noyes, 2000], change is a feature of every software development effort, and “..the [traditional method] approach was to prepare for change by trying to anticipate it all and predict what might change ahead of time. This involved performing extensive requirements analyses, capturing them all in comprehensive documentation, and reviewing them with customers and developers until everyone knew exactly what needed to be built before the work ever started.” However, [Mrenak, 1991] observes that the assumption that customers know what they want, can communicate to developers what they want, and that what they want won’t evolve over time is a fundamental error in the traditional method.

Agile Methods

The Agile Methods Manifesto

This “Manifesto” is endorsed by a number of individuals, most recognized as long time participants in the evolution of software development methods.

[Russell, 2002] presents an illuminating if somewhat biased contrast of light (Agile) methods against heavy (Plan-Driven) methods with the following recipes for “Designing a custom coffee maker:”

From this example, the flavor of Agile methods is clear: prioritize requirements, implement a subset of top requirements quickly, deliver the implementation to the customer, do it again until the customer is satisfied with the resulting software product. In general we can recognize the following characteristics of Agile Methods: 

•            the importance of individuals

•            customer commitment

•            volatile requirements

•            current-version focus

•            continuous refactoring

•            small project size

Current Variations of Agile Methods

Extreme Programming

Extreme Programming (XP) attempts to address the basic risks faced in software development including schedule slips, project cancellations, cost overruns, defect rates, mission misunderstandings, mission change, and staff turnover. There are several practices that XP utilizes: 

•            Planning,

•            Small Releases,

•            Scenarios,

•            Simple design,

•            Testing,

•            Refactoring,

•            Pair Programming,

•            Collective ownership,

•            Continuous integration,

•            40-hour week,

•            On-site customer,

•            Coding standards,

•            Open workspace and

•            Team rules

XP stresses customer satisfaction through iterative communication between the customer and the programmers. In an XP project, programmers and business managers setup scenarios to represent system requirements. Each scenario describes what will be developed and the amount of time it will take. According to a recent a study by Sanjay Murthi, using XP project teams were more enthusiastic and eager. Because of XP’s concept of pair programming, team members where able to fix anyone else’s bugs. XP also promotes regular builds, version control and working systems, which allows for early problem detection. By maintaining good version control, roll back to a previous build was an easy process if a build went bad. Software and system design in XP is kept simple and clean. Testing is performed on a daily basis to ensure a working “bug-free” system at the start of programming every day.

Rules and practices for extreme programming:

Crystal Methods

Crystal Methods is a set of human-powered software development methodologies that are collected together into a self-adapting family. The methods are based on the premise that every project needs a slightly different set of policies and conventions. The development of a project using Crystal Methods is sensitive to people issues, and improves as the people issues improve.

Crystal collects together self-adapting family of “shrink-to-fit,” human-powered software development methodologies based on the following understandings:

•           Every project needs a slightly different set of policies and conventions, or methodology.

•            The workings of the project are sensitive to people issues and improve as the people issues improve, individuals get better and their teamwork gets better.

•            Better communications and frequent deliveries reduce the need for intermediate work products.

Crystal Methodologies is a people-centric methodology that focuses on achieving project success through enhancing the work of the people involved. Crystal family methodologies work to reduce the paperwork, overhead and bureaucracy to the least that is practical for the parameters of that project. The “shrink-to-fit” attribute of Crystal refers to the fact that development starts with something small enough and continues working to make it smaller and better fitting for the project. The principle of Crystal is that

“Software development is a cooperative game – using props and markers to remind and incite, to reach the next move. The endpoint of the game is a delivered system. The next game is to alter or replace the system.” [Cockburn, A., 1999]

The description of Crystal Methodologies refers to intermediate work products. These intermediate work products are necessary to help the team make their next move in development. Their primary purpose is to help the team create the final work product [Cockburn, A., 2002]. According to Cockburn, the intermediate work products benefit the team by reminding them of the project’s purpose and to incite new ideas or thoughts that would benefit the project. Finally, Cockburn summarizes Crystal with the following:

Values: Communications using Crystal Methodologies is strong and deliverables are not extensive. Crystal “Clear” strives to keep the communications paths short, rich and informal. Have frequent Release builds. Reduce overhead. The intent is that more deliveries and better communication will reduce the need for intermediate work products.

People: For each of the roles in the software development project using Crystal Methodologies, there should be a distinct person in that role.

•           Sponsor

•         Senior Designer

•            User

•            Designer-programmers (designers)

Merged roles needed that may come from the above roles:

•            Business Expert (maybe sponsor, user or senior designer)

•            Coordinator (maybe senior designer)

•            Tester (maybe from designers)

•            Writer (maybe from designers)

Deliverables for each role:

Techniques used: Projects using Crystal Methods should be managed using milestones and risk lists. Other techniques employed during the development life cycle (i.e., project planning, requirements gathering, design, programming, testing) are decided by the team.

Dynamic Systems Development Methodology

RAD group met in 1994 to discuss definition of standard iterative process to support RAD development. [[IEEE Computer, June 2003]

The need for a Dynamic System Development Method (DSDM) came from the growing demand for Rapid Application Development technologies. DSDM is an Agile Method that is a project delivery framework that aids the development and delivery of business solutions to tight time scales and fixed budgets. In all software development projects, requirements creep has been a major barrier to conquer. To overcome this barrier, DSDM allows for requirements to change but maintains a fixed time schedule and fixed resources as much as possible. DSDM operates on the assumption that “nothing is built perfectly first time, but that a usable and useful 80% of the proposed system can be produced in 20% of the time it would take to produce the total system.” [DSDM Consortium]

Source: DSDM Consortium, Timebox

The mechanism used for handling the flexibility of requirements is the time-box. The DSDM time-box is developed based on the fixed completion date of the project. An overall time frame is constructed by nesting shorter time-boxes of two to six weeks in duration. As depicted in the following figure there are three phases in time-boxing 1) Investigation, 2) Refinement and 3) Consolidation. Investigation in time-boxing is a quick pass to see if the team is taking the right direction. Secondly, Refinement builds on the comments resulting from the review at the end of the investigation. The final phase, Consolidation, ties up any loose ends in the project. Each time-box has a fixed date and a prioritized set of requirements. Requirements included in each time-box should be mixed between mandatory and of lesser priority to allow room for flexibility.

DSDM gains user acceptance early because it is an iterative process based on prototyping and involves the users throughout the project life cycle. The benefits of using DSDM include:

•            Users are more likely to claim ownership of the system.

•            Risk of building the wrong system is greatly reduced.

•            The final system is more likely to meet the users’ real business requirements.

•            Because users are involved from the beginning, they will be better trained.

•            The implementation is more likely to go smoothly because of the cooperation of all parties with vested interest in the development.

It’s important to note that users must be licensed to use the DSDM Framework. That is individuals or organizations using the Framework must be full members of the DSDM Consortium.

The DSDM Framework should be used on projects that can identify classes of users who will use the end result so that knowledgeable representatives can participate throughout the life of the project. These representatives are known to DSDM as “Ambassador Users” because they provide the two-way communication channel between the business community and the project managers. Large projects should be able to be decomposed into smaller components for either incremental delivery or for development by parallel teams. Projects employing the DSDM Framework should focus controls on ensuring that iterative development will lead safely to the right business solution.

Source: DSDM Consortium, DSDM Life-Cycle

The DSDM life-cycle consists of five phases as well as a Pre- and Post- Project phase. As depicted in the figure below, the phases are Feasibility Study, Business Study, Functional Model Iteration, Design and Build Iteration and Implementation.

The Pre-project phase, Feasibility and Business Studies are done sequentially because they set the ground rules for the rest of iterative and incremental development on the project. When the project solution is delivered, the project team is disbanded and the Post-project phase begins. This activity includes maintenance of the delivered product and checking that the expected business benefits have been achieved. Important to note, testing is not a separate function of the lifecycle because it is happening throughout both the Functional Model and Design and Build Iterations. The following table summarizes the activities in each phase of the lifecycle:

Finally, there are nine principles that are the foundations on which DSDM is based:

•            Active user involvement.

•            Team must be empowered to make decisions.

•            Frequent delivery of products.

•            Fitness for business purpose is essential for acceptance of deliverables.

•            Iterative and incremental development is necessary to converge on an accurate business solution.

•            All changes during development are reversible.

•            Requirements are baselined at a high level.

•            Testing is integrated throughout the life-cycle.

•            Collaboration and cooperation between all stakeholders is essential.

Feature Driven Development

In 1997 Peter Coad and Jeff DeLuca resurrected and restructured a Singapore logistics system project as an IID project. [IEEE Computer, June 2003]

Developed by Nebulon’s Jeff De Luca, Feature-Driven Development (FDD), incorporates several ideas and practices from the last 30 years of the IT industry. De Luca combined these varied ideas and concepts to create patterns of play that bring success [Nebulon]. FDD is a model-driven development approach that uses modeling exercises at the start of a project as a knowledge transfer and requirements, problem solving techniques, and reporting guidelines providing every stakeholder of a project with the information necessary to make sound, timely decisions. De Luca suggests that “one of the keys to good object modeling is to model the domain…” [Nebulon]. Because domain modeling involves the users, one of the outcomes of the modeling exercise is that the same language is used.

Source: http://www.nebulon.com/articles/fdd/download/fddoverview.pdf

There are several processes that FDD uses. FDD starts with creating the domain object model in collaboration with the domain experts. The developers then use information gathered from this modeling activity, and other requirements activities to create a features list. After the requirements and features are known, a rough plan is developed and responsibilities assigned to various teams. These teams develop the features by repeatedly performing design and build iterations that last no longer than two weeks at a time. Each process has entry and exit criteria, tasks, and verification procedures. The original FDD processes consisted of:

•            Object Modeling

•            Features List

•            Development Planning

•            Design by Feature

•            Development by Feature

The figure depicts the FDD process flow.

The following table summarizes the latest FDD processes published by Jeff De Luca of Nebulon.:

Source: http://www.nebulon.com/articles/fdd/latestprocesses.html

Lean Development