An example of the sequences of Poor Quality Software

                1  while (ring receive buffer not empty and side buffer not empty) DO
                2    Initialize pointer to first message in side buffer or ring receive buffer
                
                3    get copy of buffer

                4    switch (message) {
                5       case (incoming_message):
                6             if (sending switch is out of service) DO {
                7                 if (ring write buffer is empty) DO
                8                     send "in service" to status map
                9                 else
                10                    break
                              } // END IF
                11            process incoming message, set up pointers to optional parameters
                12            break
                     } // END SWITCH
                13   do optional parameter work
              

Software Crisis

But we could think that this bug occurred a while ago and that nowadays we have more advanced technologies, methodologies, training systems and developers.

Is this really true? Just partially, it's true Software Development has evolved a lot, but the type of problems that are solved via Software has also evolved, every day with try to solve more problems and more complex via software.

The Software Crisis term was coined by USA Department of Defence years ago in order to describe that the complexity of the problems addressed of software has outpaced the improvements in the software creation process as shown graphically in .

In other words, the software creation process has evolved very little while the problems software is solving are way too much complex

Additionally, as depicted in , the need of software developers has increased exponentially, because more software is needed as software is used in nearly every product with a minimum of complexity. Whereas the need of developers has increased exponentially, the availability of developers has unfortunately not grown at the same pace, i.e. there are less developers than what is needed. Due to that, people without the right skills have started developing software, with the believe that developing software is an easy task that nearly everybody could do. Developing software with people not properly trained or without the right skills inherently leads to bad quality software.

Legal Warranties

Mortenson, a construction contractor purchased software from Timberline Software Corporation, which Timberline installed in Mortenson's computers. Mortenson, relying on the software, placed a bid which was $1.95 million too low because a bug in the software of which Timberline was aware. The State of Washington Supreme Court ruled in favour of Timberline Software. However, a simple bug in the software lead to multiple problems to both companies. In the US Warranty Laws, the Article 2 of the Uniform Commercial Code includes the "Uniform Computer Information Transaction Act" UCITA) that allows software manufacturers to:

• Disclaim all liability for defects
• Prevent the transfer of software from person to person remotely
• Disable licensed software during a dispute

That act, practically means that software distributors can limit their liability through appropriate clauses in the contracts. For instance, below is shown the disclaimer of warranties of a Microsoft product. Although the law overprotects software developers and distributors, using these disclaimers may prevent legal problems, but there are multiple additional problems related with poor software that are not avoided by them.

Although the law overprotects software developers and distributors, using these disclaimers may prevent legal problems, but it is just a way to avoid the legal problems of having bad quality software not solving the real problem that is what affect and frustrates end users.

View 1: Formal Definition

Software quality may be defined as conformance to explicitly stated functional and performance requirements, explicitly documented development standards and implicit characteristics that are expected of all professionally developed software.

This definition emphasis from three points:

1. Software requirements are the foundations from which quality is measured: Lack of conformance to requirement is lack of quality.
2. Specified standards define a set of development criteria that guide the manager is software engineering: If criteria are not followed lack of quality will almost result.
3. A set of implicit requirements often goes unmentioned, like for example ease of use, maintainability, etc.: If software confirms to its explicit requirement but fails to meet implicit requirements,software quality is suspected.

For the first item, explicit software requirements, it is going to be relatively easy to check objectively the conformance to them, for the second one, it is going to be more complicated and depends on how documented those standards are, for the implicit characteristics expected, it is going to be even tougher, as measuring conformance to something that is implicit, is, by definition, impossible.

View 2: The Human Point of View

Those "implicit" requirements mentioned in the formal definition are a hint to indicate that there is something more about Software that goes beyond the explicit requirements. At the end of the day, software is going to be used by people, which do not care about the requirements but about their expectations. Hence, the need to look for another point of view.

"A product's quality is a function of how much it changes the world for the better." [[MANAGEMENT-VS-QUALITY]] or "Quality is value to some person" [[QUALITY-SOFTWARE-MANAGEMENT]]. Both definitions stress that the quality may be subjective. I.e. different people are going to perceive different quality in the same software. The software developers should also think about end users and asking themselves questions such as "How are users going to use the software?".

In order to provide a more complete picture, IEEE standard 610.12-1990 combines both views in their definitions of quality:

View 3: Internal vs. External Quality

There is another dimension of Software Quality that depends on whether we focused on the part of the Software that is exposed to the users or on the part of the Software that is not.

External Quality is the fitness for purpose of the software, i.e. does the software what it is supposed to do?. The typical way to measure external quality is through functional tests and bugs measurement.

Usually this is related to the conformance requirements that affect end-users (formal definition) as well as to meeting the end-user expectations (human point of view).

Some of the properties that determine the external quality of software are:

• Conformance to the product specifications and user expectations.
• Reliability: Is the software working with the same level performance under different conditions and during all the time.
• Accuracy: Does the software do exactly what is supposed to do.
• Ease of use and comfort: Is the software easy to use and responds in an amount time according to user expectations?
• Robustness: Does the software adapt to unforeseen situations, e.g. invalid input parameters, connectivity lost...

Internal Quality is everything the software does but is never seen directly by the end-user. It's the implementation, which the customer never directly sees. Internal quality can be measured by conformance requirements (not focused on end-users but on software structure), software analysis and adherence to development standards or best practices.

If it is not visible to end-user, and our target is make customers happy, we could ask ourselves if Internal Quality is something we should pay attention to.

Internal quality is related with the design of the software and it is purely in the interest of development. If Internal quality starts falling, the system will be less amenable to change in the future. Due to that, code reviews, refactoring and testing are essential as otherwise the internal quality will slip.

An interesting analogy with debts and bad code design was developed Ward [[DEBT-ANALOGY]]. Sometimes companies need to get some credit from the banks in order to be able to invest, however, it is also critical to understand that is impossible to ask for credit continuously as the paying interest will kill the company financially. The same could be used for software, sometimes it is good to assume some technical debt to achieve a goal, for instance, meeting a critical milestone to reach users before our competitors, but it is important to understand that assuming technical debt endlessly would kill the project as it will make the product unmaintainable.

Sometimes, after achieving the target External Quality, we need to refactor our code to improve the Internal Quality. Software Quality is sometimes the art of a continuous refactor.

Let's go back to the analogy of writing an essay or a paper, in that case most people write out the first draft as a long brain-dump saying everything that should be said. After that, the draft is constantly changed (refactored) until it is a cohesive piece of work.

When developing software (for instance in University assignments :-D) the first draft is often finished when it meets the general requirements of the task. So, after that, there is an immediate need to refactor the work into a better state without breaking the external quality. Maybe writing software is also kind of an art?

This is universally true, and the danger of not paying attention to refactor your code is bigger on a larger project where poor quality code can lose you days in debugging and refactoring.

Some of the properties that enable the process of product with good internal quality are:

• Concision: The code does not suffer from duplication.
• Cohesion: Each [module|class|routine] serves a particular purpose (e.g. it does one thing) and does it well.
• Low coupling: Minimal inter-dependencies and interrelation between objects and modules.
• Simplicity: The software is always designed in the simplest possible manner so that errors are less likely to be introduced.
• Generality: Specific solutions are only used if they are really needed.
• Clarity: the code enjoys a good auto-documentation level so that it is easy to be maintained.

The external quality is sometimes compared with "Doing the right things" as opposed to "Doing the things right" which should define what internal quality is.

Usually, the problems with the external quality characteristics (correctness, reliability...) are simply visible symptoms about software problems, that usually are related with internal quality attributes: program structure, complexity, coupling, testability, reusability, readability, maintainability... Sometimes, when the internal quality is bad, external quality can be met during a short period of time, but in the longer term, the external quality will be affected.

An excellent analogy is the Quality Iceberg created by Steve McConell (see ).

View 4: ISO 9126

ISO 9126 defines Software Quality as the totality of characteristics of an entity that bears on its ability to satisfy stated and implied needs.

It recognizes that quality is not only determined by the software itself but also by the process used for software development and the use made of the software. Hence, the following identities are defined:

• Quality of the process of constructing software: process quality
• Quality of software product in itself: internal and external quality
• Quality of software product in use: quality in use

ISO identified 6 characteristics of the software quality that are sub-divided into sub-characteristics:

• Functionality: A set of attributes that bear on the existence of a set of functions and their specified properties. The following sub-characteristics were defined: Suitability, Accuracy, Interoperability and Security.
• Reliability: A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time. The following sub-characteristics were defined. The following sub-characteristics were defined: Maturity, Fault Tolerance and Recoverability.
• Usability: A set of attributes that bear on the effort needed for use. The following sub-characteristics were defined: Understandability, Learnability and Operability.
• Efficiency: A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used. The following sub-characteristics were defined: Time Behaviour and Resource Behaviour.
• Maintenability: A set of attributes that bear on the effort needed to make specified modifications. The following sub-characteristics were defined: Analyzability, Changeability, Stability and Testability.
• Portability: A set of attributes that bear on the ability of software to be transferred from one environment to another. The following sub-characteristics were defined: Adaptability, Installability, Conformance and Replaceability.

Quality in use is defined by ISO as "the extent to which a product used by specified users meets their needs to achieve specified goals with effectiveness, productivity, and satisfaction in specified contexts of use". The quality in use hence depends on the context in which the product is used and its intrinsic quality.

Summary

There is no a single definition of quality. However, the importance of Software Quality is continuously increasing. The concepts of external and internal quality are commonly used across the software industry, but despite that, the properties used to measure the quality diverge across different methodologies, standards or companies.

Introduction to SQA

Software Quality Assurance (SQA) is the set of methods used to improve internal and external qualities. SQA aims at preventing, identifying and removing defects throughout the development cycle as early as possible, as such reducing test and maintenance costs.

SQA consists of a systematic, planned set of actions necessary to provide adequate confidence that the software development process or the maintenance process of a software system product conforms to established functional technical requirements as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines.

The ultimate target of the SQA activities is that few, if any, defects remain in the software system when it is delivered to its customers or released to the market. As it is virtually impossible to remove all defects, another aim of QA is to minimize the disruptions and damages caused by these remaining defects.

The SQA methodology will also depend on the software development methodology used, as they are inherently couple. For instance, different software development models will focus the test effort at different points in the development process. Newer development models, such as Agile, often employ test driven development and place an increased portion of the testing in the hands of the developer, before it reaches a formal team of testers. In a more traditional model, most of the test execution occurs after the requirements have been defined and the coding process has been completed.

An example of an SQA methodology is available at [[IEEE-QA-TEMPLATE]].

SQA activities are not only carried out by the Software Quality group, the software engineer group is responsible for putting in place the SQA methodology defined, which may include different activities such as testing, inspection, reviews...

SQA Activities

Pre-QA Activities: Quality Planning

Before doing any QA activity, it is important to consider some aspects such as the target quality, the most appropriate QA activities to be done and when should be done, how are the quality going to be measured. All those activities are usually called Pre-QA or Quality Planning Activities.

The first activity that should be done in SQE is defining what are the specific quality goals for the software to be delivered. In order to do so, it is important to understand what are the expectations of the software end-user/customer. Obviously, it is also key to recognize that the budget is limited and that the quality target should be financially doable. The following activities are key to identify the target quality of the software:

1. Identify quality views and attributes meaningful to target customers and users. Which aspects will be key for them to perceive the software as high quality one? This may depend a lot on the type of product and on the target customers.
2. Select direct quality measures that can be used to measure those quality attributes that are key for the customers.
3. Quantify these quality measures to set quality goals while considering the market environment and the cost of achieving different quality goals.

Once that the quality goals are clear, the QA strategy should be defined. Two key decisions should be made during this stage:

1. Which QA activities are the most adequate ones to meet the customer quality expectations. For doing this, it is important to translate the quality views, attributes and goals into the QA activities to be performed. It is also very important to determine when every QA activity is going to be executed as part of the full Software Development Process.
2. The external quality measures should be mapped into internal indirect ones via selected quality models. Good models are required in order to predict external quality based on internal indicators. It is also very important to identify how the results of this measure are going to be collected and used (e.g. what happens if the quality is not good enough or how the feedback is going to be used).

In-QA Activities

These activities have been described in section 1.4.2 and basically consist in executing the QA activities planned and handling the defects discovered as a result of them.

Post-QA Activities

These activities consist in measuring the quality of the software (after the QA activities), assess the quality of the software product and the definition of the decisions and actions need to improve its quality.

All these activities are usually carried out after normal QA activities have started but as part of these "normal" QA activities. Their goal is to provide feedback so that decisions can be made and improvements can be suggested. The key activities include:

• Measurement: Besides the direct measure of tracking the defects during the in-QA activities, various other measurements are needed in order to the track the QA activities and for project management purposes. The data resulting from this analysis is important to manage software project and quality.
• Analysis and Modelling: These activities analyse measurement data from software projects and fit them to analytical models that provide quantitative assessment of selected quality characteristics and sub-characteristics. This is key to obtain an objective assessment of the current product quality, predict future quality or identify problematic areas.
• Providing feedback and identifying improvement potentials: The results of the previous activities can lead to some suggestions to improve the process followed with the software being assessed (e.g. more testing resources are needed, test cases are not sufficient...) or the general SQE methodology.
• Follow-up Activities: Besides immediate actions, some actions resulting from the analysis may require a longer time. For instance, if major changes are suggested to change the SQE process, they cannot be usually implemented while the current process has not finished.

The Deming Quality Cycle

W. Edwards Deming in the 1950's proposed that business processes should be analysed and measured to identify sources of variations that cause products to deviate from customer requirements. He recommended that business processes be placed in a continuous feedback loop so that managers can identify and change the parts of the process that need improvements. As a teacher, Deming created a (rather oversimplified) diagram to illustrate this continuous process, commonly known as the PDCA cycle for Plan, Do, Check, Act:

• Plan Quadrant: one defines the objectives and determines the conditions and methods required to achieve them.
• Do Quadrant: the conditions are created and the necessary training to execute the plan is performed (new procedures). The work is then performed according to these procedures.
• Check Quadrant: One must check to determine whether work is progressing according to the plan and whether the expected results are obtained.
• Action Quadrant: If the checkup reveals that the work is not being performed according to plan or results are not what was anticipated, measures must be devised for appropriate action.

Deming's PDCA cycle can be illustrated as in :

By going around the PDCA circle, the working methods are continuously improved as well as the results obtained. However, it is important to take care avoid a situation called "spiral of death" It happens when an organization goes around and around the quadrants, never actually bringing a system into production.

Intrinsic Product Quality Metrics

Reliability, Error Rate and Mean Time To Failure

Software reliability is a measure of how often the software encounters an error that leads to a failure. From a formal point view, Reliability can be defined as the probability of not failing during a specified length of time:

R(n) (where n is the number of time units)

The probability of failing in a specified length of time is 1 minus the reliability for that length of time and it's usually denoted by a capital F letter:

F(n) = 1 - R(n)

If time is measured in days, R(1) is the probability of the software system having zero failures during one day (i.e. the probability of not failing in 1 day)

A couple of metrics related with the software reliability are the "Error Rate" and the "Mean Time To Failure" (MTTF). The MTTF can be defined as the average time that occurs between two system failures. Error Rate is the average number of failures suffered by the system during a given amount of time. Both metrics are related with the following formula:

$\mathrm{<mtext>Error\; Rate</mtext>}=\frac{\text{1}}{\text{MTTF}}$

The relationship between the error rate and the reliability depends on the statistical distribution of the errors, not only on the error rate.

For instance, the following table shows the errors that occured per day in two different systems during one week.

 

Example of same error rate with different distribution

Defects per Day
	DAY 1	DAY 2	DAY 3	DAY 4	DAY 5	DAY 6	DAY 7
Project A	1	2	3	4	5	6	7
Project B	7	6	5	4	3	2	1

It can be seen that both systems suffered the same amount of errors during the week 28 and hence the error rate for both systems is the same: 28/7 = 4 Errors/Day. However, the reliability of the system for the first day is very different.

Unless detailed statistics/models are available, the best estimate of the short-term future behavior is the current behavior. For instance, if a system suffers 24 failures during one day, the best estimate for the next day is that 24 failures will occur (24 errors/day) that correspond to a 1hour MTTF. That means that by default, we could assume that most of the system failures follow an exponential distribution, and hence the following formula could be used to calculate their reliability:

$\mathrm{<mtext>R(t)</mtext>}=e^{\mathrm{-\lambda t}}$

Where λ is the Error Rate and t is the amount of time for which the system reliability is calculated. A key concept of exponential distributions is that the error rate is constant and hence it doesn't change with the time. The following tables show the Probability and Cumulative Density Functions of exponential curves with different values of λ

However, this concept of systems having a constant error rate that doesn't change over time, it's not very common in the real world. For instance, in Hardware components, the error rate evolves with the time in different ways:

• During the early life of the product the error rate starts at high value, but this error rate tends to go down gradually over time.
• After the early life of the product passes (the duration of the early life depends on the product) the error rate tends to be stable, during a time called Useful Time .
• After the useful time is over, the wearout period starts. During that period, the error rate keeps growing until the hardware component suffers a failure.

A canonical example of this behaviour are lightbulbs. In order to minimize the number of failures users suffer with lightbulbs, right after being produced, the manufacturers keep them switched on during a time similar to the early life of the product. By doing this, they guarantee that the lightbulbs that go to the customer hands, got there at a time when the error rate is as low as possible. Any component with early failures is discarded before it can get to the end-users.

You can find a lot of information about reliability and how maths is used for calculating it at [[RELIABILITY-MATHS]]

Although exponential distribution may be a good compromise that could be applied to any software system, there are other distributions that may describe in a more accurate way the idea of having a non constant error rate. For instance, the Weibull distribution is frequently used in reliability analysis [[WEIBULL-BASICS]].

In a Weibull distribution, the error rate can change with the time. The Reliability function is:

That depends on two variables, η and β that define the shape of the distribution function. For a fixed value of η, the failure rate could be constant (β = 1), descending (β < 1) or ascending (β > 1). Please note that combining those three options we could describe the Hardware component failure rate phases.

If we take into account Software Upgrades, there are some interesting analysis about how a sawteeth pattern is observed [[SOFTWARE-RELIABILITY]]. Again, such a curve could be defined by combining different Weibull distributions.

Defect Density

Defect Density is the number of confirmed defects detected in software/component during a defined period of development/operation divided by the size of the software/component.

$\mathrm{<mtext>Defect\; Density</mtext>}=\frac{\text{Number of Confirmed Defects}}{\text{Software Size}}$

The "defects" are usually counted as confirmed and agreed defects (not just reported). For instance, dropped defects are not counted.

The "period" or metrics time frame, might be for one of the following:

• for a duration (say, the first month, the quarter, or the year).
• for each phase of the software life cycle.
• for the whole of the software life cycle, usually known as Life Of Product (LOF) and may comprise time after the software product's release to the market.

The "opportunities for error" (OFE) or sofware "size" is measured in one of the following:

• Source Lines of Code that are usually counted as thousands of Lines Of Code (KLOC)
• Function Points (FP)

In the following chapters both ways of measuring OFE will be studied separately.

Lines of Code

Counting the lines of code (LOC) is way more complex that what it could be initially considered. The major problem for couting lines of code comes from the ambiguity of the operational definition, the actual counting. In the early days of Assembler programming, in which one physical line was the same as one instruction, the LOC definition was clear. With the availability of high-level languages the one-to- one correspondence broke down. Differences between physical lines and instruction statements (or logical lines of code) and differences among languages contribute to the huge variations in counting LOCs. Even within the same language, the methods and algorithms used by different counting tools can cause significant differences in the final counts. Multiple variations were already described by Jones in 1986 such as:

• Count only executable lines.
• Count executable lines plus data definitions.
• Count executable lines, data definitions, and comments.
• Count executable lines, data definitions, comments, and job control language.
• Count lines as physical lines on an input screen.
• Count lines as terminated by logical delimiters.

For instance, next example includes two approaches for coding the same functionality. As the functionality is the same, and it is writing in the same manner, the opportunities for error should be the same, however, the lines of code differ. For instance, if we count all the lines (job control language, comments...), in the first case only one line of code is used whereas in the second case 5 lines of code have been used.

                  for (i=0; i<100; ++i) printf("I love compact coding"); /* what is the number of lines of code in this case? */

                  /* How many lines of code is this? */
                  for (i=0; i<100; ++i)
                  {
                    printf("I am the most productive developer"); 
                  }
                  /* end of for */
                

Some authors have considered LOC not only a less useful way to measure software size but also a harmful thing for sofware economics and productivity. For instance, the paper written by Capers Jones called "A Short History of Lines of Code (LOC) Metrics" [[LOC-HISTORY]] offers a very interesting historical view about the evolution of Software Programming Languages and LOC metrics.

Regardless of the LOC measurements used, when a software product is released to the market for the first time, and when a certain way to measure lines of code is specified, it is relatively easy to state its quality level (projected or actual). However, when enhancements are made and subsequent versions of the product are released, the measurement is more complicated. In order to have a good insight on the product quality is important to follow a two-fold approach:

• Measure the quality of the entire product.
• Measure the quality of the new/changed parts of the product.

The first measure may improve over releases due to aging and defect removal, but that improvement in the overall defect rate may hide problems on the developement/quality process (e.g. new code contains a higher defect density that the "old" code which indicates a problem in the process). In order to be able to calculate defect rate for the new and changed code, the following must be available:

• LOC count: The entire software product as well as the new and changed code of the release must be available.
• Defect tracking: Defects must be tracked to the release origin, i.e. the portion of the code that contains the defects and at what release the portion was added, changed, or enhanced. When calculating the defect rate of the entire product, all defects are used; when calculating the defect rate for the new and changed code, only defects of the release origin of the new and changed code are included.

These tasks are enabled by the practice of change flagging. Specifically, when a new function is added or an enhancement is made to an existing function, the new and changed lines of code are flagged. The change-flagging practice is also important to the developers who deal with problem determination and maintenance. When a defect is reported and the fault zone determined, the developer can determine in which function or enhancement pertaining to what requirements at what release origin the defect was injected. The following is an example on how the overall defect rate and the defect rate for new code is mesaured at IBM according to the book "Metrics and models in software quality engineering" by Stephen H. Kan.

In the first version of a software product 30 Defects were reported by end-users and the software size was 30KLOC. After fixing all the discovered bugs, the team works in a new version that includes 10 new KLOC. End-users report 10 additional defects in this new version that were injected in the new 10KLOC.

The Defect Density of the first version was 1 defect/KLOC.

If we calculate the Defect Density of the second version in the same way, it would be: DD = 10/40 = 0.25 defects/KLOC. If we compare this value, we could conclude that the second version was way better that the first one.

But, this could be misleading as indeed, the second version could include only defects in the new 10KLOC, not in the old ones. We could calculate the same metric but just counting the new Lines of Code. If we do so, the result would be: DD = 10/10 = 1 defect/KLOC which is the same than for the first release.

We could conclude that for end-users, the second version is going to be a significant improvement as the number of defects are going to perceive is smaller both in absolute and relative terms. However, the team has been doing a similar job in terms of defects remaining after releasing the product (defect injection and detection).

It is important to think how useful is this metric from two points of view:

• Drive Quality Improvement: Very important for the development team.
• Meet customer expectations.

From the customer's point of view, the defect rate is not as relevant as the total number of defects that might affect their business. Therefore, a good defect rate target should lead to a release-to-release reduction in the total number of defects, regardless of size. I.e. Not only the defect rate should be reduced but also the total number of defects. If a new release is larger than its predecessors, it means the defect rate goal for the new and changed code has to be significantly better than that of the previous release in order to reduce the total number of defects.

In the example above, from the initial release to the second release the defect rate didn't improve. However, customers experienced a 66% reduction [(30 - 10)/30] in the number of defects because the second release is smaller.

Function Points

As explained in the previous chapter, measuring the opportunities for error through the lines of code has some problems. Counting lines of code is but one way to measure size. Another alternative is the using the function point. In recent years the function point has been gaining acceptance in application development in terms of both productivity (e.g., function points per person-year) and quality (e.g., defects per function point)

A function can be defined as a collection of executable statements that performs a certain task, together with declarations of the formal parameters and local variables manipulated by those statements. The ultimate measure of software productivity is the number of functions a development team can produce given a certain amount of resource, regardless of the size of the software in lines of code. The defect rate metric, ideally, is indexed to the number of functions a software provides. If defects per unit of functions is low, then the software should have better quality even though the defects per KLOC value could be higher — when the functions were implemented by fewer lines of code. Although this approach seems very powerful and promising, from a practical point of view it is very difficult to be used.

The function point metric was originated by Albrecht and his colleagues at IBM in the mid-1970s. The name could be a bit misleading as the technique itself does not count the functions. Instead it tries to measures some aspects that determine the software complexity withouth taking into the differences between programming languages and development styles that change the LOC metric. In order to do so, it takes into account five major components that comprise a software product:

• External Inputs (EIs): Elementary process in which data crosses the boundary from outside to inside. This data may come from a data input screen or another application. The data may be used to maintain one or more internal logical files. The data can be either control information or business information. If the data is control information it does not have to update an internal logical file.
• External Outputs (EOs): Elementary process in which derived data passes across the boundary from inside to outside. Additionally, an EO may update an ILF. The data creates reports or output files sent to other applications. These reports and files are created from one or more internal logical files and external interface file.
• External Inquiries (EQs): Elementary process with both input and output components that result in data retrieval from one or more internal logical files and external interface files. The input process does not update any Internal Logical Files, and the output side does not contain derived data.
• Internal Logical Files (ILFs): A user identifiable group of logically related data that resides entirely within the applications boundary and is maintained through external inputs.
• External Interface Files (EIFs): A user identifiable group of logically related data that is used for reference purposes only. The data resides entirely outside the application and is maintained by another application. The external interface file is an internal logical file for another application.

Following figure provides a graphical example on how all these components work together and how they interact with the end-users.

Apart from being technology independent, this way of identifying the key software functions is very interesting as it is focused on the end-user point view: most of the components are thought from the user’s perspective (not the developers one), hence it works well with use cases.

The number of function points is obtained by the addition of the number of occurrences of those components (each of them weighted by a different factor) multiplied by an adjustment factor chosen based on the software characteristics:

FP = FC x VAF

Where:

• FP: Is the Function Points
• FC: Is the weighted function count
• VAF: Is the adjustment factor that depends on software characteristics

In order to calculate the Function Points, every component is classified in three categories according to its complexity (low/medium/high). A different weight factor is assigned to every component type and category. The following weights are defined for every component and complexity:

• Number of external inputs: 3-4-6.
• Number of external outputs: 4-5-7.
• Number of external inquiries: 3-4-6.
• Number of logical internal files: 7-10-15.
• Number of external interface files: 5-7-10.

When the number of components (classified by complexity) is available, given the previous weighting factors, the Function Counts (FCs) can be calculated based on the following formula:

$\mathrm{FC}=\sum _{j=\mathrm{1,}n=\mathrm{1,}}^{\mathrm{3,5}}{w}_{\mathrm{ij}}{x}_{\mathrm{ij}}$

Where wij are the weighting factors and xij the number of ocurrences of each component in the software. i denotes complexity and j denotes the component type. The following table shows graphically how can this function be calculated easily.

 

FC Calculation

Type	Low Complexity	Mid Complexity	High Complexity	Total
EI	_ x 3 +	_ x 4 +	_ x 6 +	=
EO	_ x 4 +	_ x 5 +	_ x 7 +	=
EQ	_ x 3 +	_ x 4 +	_ x 6 +	=
ILF	_ x 7 +	_ x 10 +	_ x 15 +	=
EIF	_ x 5 +	_ x 7 +	_ x 10 +	=

The complexity classification of each component is based on a set of standards that define complexity in terms of objective guidelines. For instance, for the external output component, if the number of data element types is 20 or more and the number of file types referenced is 2 or more, then complexity is high. If the number of data element types is 5 or fewer and the number of file types referenced is 2 or 3, then complexity is low. The following tables provide the standard categorization where:

• DETs are equivalent to non-repeated fields or attributes.
• RETs are equivalent to mandatory or optional sub-groups.
• FTRs are equivalent to ILFs or EIFs referenced by that transaction.

ILF and EIF Complexity Matrix

RETs	1-19 DETs	20-50 DETs	51+ DETs
1	Low	Low	Medium
2-5	Low	Medium	High
6+	Medium	High	High

EI Complexity Matrix

FTRs	1-4 DETs	5-15 DETs	16+ DETs
0-1	Low	Low	Medium
2	Low	Medium	High
3+	Medium	High	High

EO and EQ Complexity Matrix

FTRss	1-5 DETs	6-19 DETs	20+ DETs
0-1	Low	Low	Medium
2-3	Low	Medium	High
4+	Medium	High	High

In order to calculate the Value Adjustment Factor (VAF), 14 characteristics of the software system must be scored (in a scale from 0 to 5) in terms of their effect on the software. The list of characteristics is:

1. Data communications: How many communication facilities are there to aid in the transfer or exchange of information with the application or system?
2. Distributed data processing: How are distributed data and processing functions handled?
3. Performance: Did the user require response time or throughput?
4. Heavily used configuration: How heavily used is the current hardware platform where the application will be executed?
5. Transaction rate: How frequently are transactions executed daily, weekly, monthly, etc.?
6. On-Line data entry: What percentage of the information is entered On-Line?
7. End-user efficiency: Was the application designed for end-user efficiency?
8. On-Line update: How many ILF’s are updated by On-Line transaction?
9. Complex processing: Does the application have extensive logical or mathematical processing?
10. Reusability: Was the application developed to meet one or many user’s needs?
11. Installation ease: How difficult is conversion and installation?
12. Operational ease: How effective and/or automated are start-up, back-up, and recovery procedures?
13. Multiple sites: Was the application specifically designed, developed, and supported to be installed at multiple sites for multiple organizations?
14. Facilitate change: Was the application specifically designed, developed, and supported to facilitate change?

Once all these characteristics are assessed, they are summed, based on the following formula, to arrive at the value adjustment factor (VAF):

$\mathrm{VAF}=\mathrm{0.65}+\mathrm{0.01}\sum _{i=1}^{14}{c}_{\mathrm{i}}$

Where ci is the score for general system characteristic i.

Over the years the function point metric has gained acceptance as a key productivity measure from a practical point of view. However, the meaning of function point and the derivation algorithm and its rationale may need more research and more theoretical groundwork. Furthemore, function point counting can be time-consuming and expensive, and accurate counting requires certified function point specialists.

Customer satisfaction metrics

Defect Density After a Development Cycle

Defect rate during a development cycle is usually positively correlated with the defect rate in the next phases. For instance, the defect rate after integration testing is usually positively correlated with the defect rate in the field. Higher defect rates found during a phase is an indicator that the software has experienced higher error injection during that phase, unless the higher testing defect rate is due to an extraordinary testing effort (for example, additional testing or a new testing approach that was deemed more effective in detecting defects). The rationale for the positive correlation is simple: Software defect density never follows the uniform distribution. If a piece of code or a product has higher testing defects, it is a result of more effective testing or it is because of higher latent defects in the code. Myers suggested a counterintuitive principle that the more defects found during testing, the more defects will be found later.

This simple metric of defects per KLOC or function point is especially useful to monitor subsequent releases of a product in the same development organization. The development team or the project manager can use the following scenarios to judge the release quality:

1. If the defect rate during testing is the same or lower than that of the previous release (or a similar product), then ask: Does the testing for the current release deteriorate?
   • If the answer is no, the quality perspective is positive.
   • If the answer is yes, you need to do extra testing (e.g., add test cases to increase coverage, blitz test, customer testing, stress testing, etc.).
2. If the defect rate during testing is substantially higher than that of the previous release (or a similar product), then ask: Did we plan for and actually improve testing effectiveness?
   • If the answer is no, the quality perspective is negative. Ironically, the only remedial approach that can be taken at this stage of the life cycle is to do more testing, which will yield even higher defect rates.
   • If the answer is yes, then the quality perspective is the same or positive.

This concept is shown graphically in the next diagram:

Defect Arrival Pattern

Overall defect density during testing is a summary indicator. The pattern of defect arrivals (or for that matter, times between failures) gives more information. Even with the same overall defect rate during testing, different patterns of defect arrivals indicate different quality levels in the field.

Next figure shows two contrasting patterns for both the defect arrival rate and the cumulative defect rate. Data were plotted from 44 weeks before code-freeze until the week prior to code-freeze. In both projects, the overall defect count is the same, however the potential forecast of the quality on the field is quite different. In the first project, during the last weeks, the number of defects reported every week is smaller and is tending to zero. The second project, represented by the charts on the right side, follows the opposite pattern. This obviously indicates that testing started late, the test suite was not sufficient, and that the testing ended prematurely. It's extremely likely that this project, if released as it is, would lead to even more defects in the field.

The objective is always to look for defect arrivals that stabilize at a very low level, or times between failures that are far apart, before ending the testing effort and releasing the software to the field. Such declining patterns of defect arrival during testing are indeed the basic assumption of many software reliability models. The time unit for observing the arrival pattern is usually weeks and occasionally months. For reliability models that require execution time data, the time interval is in units of CPU time.

When we talk about the defect arrival pattern, there are actually three slightly different metrics, which should be looked at simultaneously:

• The defect arrivals (defects reported) during the testing phase by time interval (e.g. week). These are the raw number of arrivals, not all of which are valid defects.
• The pattern of valid defect arrivals when problem determination is done on the reported problems. This is the true defect pattern.
• The pattern of defect backlog overtime. This metric is needed because development organizations cannot investigate and fix all reported problems immediately. This metric is a workload statement as well as a quality statement. If the defect backlog is large at the end of the development cycle and a lot of fixes have yet to be integrated into the system, the stability of the system (hence its quality) will be affected. Retesting (regression test) is needed to ensure that targeted product quality levels are reached.

Defect Removal Metrics

We have just introduced an interesting concept. Detecting a defect doesn't mean it is going to be automatically removed. This can happen because of many different reasons:

• Lack of time: The solution of the defect requires a lot of time to be properly fixed (detect root, fix it, ensure no other functionality breaks because of the fix, etc.)
• Lack of bandwidth: The team is focused on implementing features so no immediate attention is put on fixing that defect.
• Lack of importance: The defect is minor or not important enough to become a priority for the team.
• Lack of knowdlege: The defect is well-known but the root of the defect is not known yet (remember a defect is just a symptom).

A metric intended to distinguish between defect detection and defect removal is the Defect Removal Pattern, that describes the evolution of the number of defects removed over the time. This metric can be calculated per unit of time or per phase of the project (e.g. iteration).

Some related metrics are the "average time to detect a defect" (which provides an indication about how good the process is about detecting defects) and the "average time to fix a defect" which is an indiator of how good the process is with respect to fixing defects once they have been detected. These metrics are key as we should remember that the later a defect is detected and fixed, the more expensive it is.

Defect Backlog / Burndown

A burndown chart is a graphical representation of the amount of work left to be done vs. the time. It's typically used in Agile methodologies to check sprint evolution, measure team speed, etc. In those cases, the amount work is always decreasing as the tasks for the sprint are identified before the sprint starts.

An equivalent concept is the Defect Backlog / Burndown. In that case the graphic does not represent the amount of work to be done but the amount of unfixed defects. The curve can go down if no more defects are found and the remaining ones are fixed, or go up if the number of detected defects overpace the fix rate.

In the example above, we can see that the red line shows the number of cumulative defects (fixed or unfixed), the green one shows the number of fixed ones, whereas the black line shows the "delta" between detected and fixed defects (i.e. the size of the defect backlog).

Ideally, the defect backlog count should be zero before releasing a product. But in big product that is nearly impossible. This requires product managers to play with the four key aspects of any software project: resources, scope, time and quality. In particular, the following actions could be done:

• Increase the number of resources fixing defects: The idea is adding more resources to work on fixing defects. However, we should be extremely cautious about this as adding resources late to a project can lead to additional delays.
• Reduce the product scope: don't let more features being added at a given point of time (no more features 1 month before the target release date). This leads to more resoures focused on fixing defects and fewer bugs injected by new features landing on the project.
• Postpone the release date: Ask for more time so more time can be spent fixing defects.
• Reduce the target quality by not paying attention to any bug but just to the critical ones (e.g. blocker defects).

One approach that is used sometimes to make the defect backlog go to zero is using triage meetings to determine which defects are blockers and which ones are not. The idea is that the closer the release date is, the more difficult is to consider a bug as a blocker for the release. However, having a clear set of guidelines about what is a blocker and what is not, is also very helpful, for instance, Mozilla used a particualr one for FirefoxOS

Defect Removal Effectiveness

Defect removal effectiveness can be defined as follows:

It provides a measure of the percentage of defects removed in one phase with regards to the overall number of defects in the code when entering into that phase. As the total number of latent defects in the product at any given phase is not known, the denominator of the metric can only be approximated which is usually done through:

The metric can be calculated for the entire development process, for the front end (before code integration), and for each phase. It is called early defect removal

The higher the value of the metric, the more effective the development process and the fewer the defects escape to the next phase or to the field. This metric is a key concept of the defect removal model for software development.

For instance, if during the development of a product 80 bugs were found and fixed, but there were still 20 defects latent that were found by the customers when the product hit the field, the DRE would be:

DRE = 80 / (80 + 20) = 80%

In average, 80 of every 100 defects were removed.

Another view of this metric is depicted in the following Figure.

It shows how are defects injected, detected and repair during a project phase. Based on it, another way to calculate the defect removal efficiency can be found:

The following table is an example in which the data about when are errrors injected and detected in a software project is provided.

With that data, the DRE could be calculated for different phases of the software development process. Some examples are shown below:

              During the First Iteration the total numbers of defect injected was 25.

              The meaning of the value "5" in the intersection of "Iteration 1"
              row and column is that during that phase, only 5 defects were
              removed.

              The meaning of the value 10 in the intersetion of row "Iteration
              2" and column "Iteration 1" is that during Iteration 2, 10
              defects that were originated in Iteration 1 were removed. 

              Equally, the meaning of the value 15 in the intersetion of row
              "Iteration 2" and column "Iteration 2" is that during Iteration
              2, 15 defects that were also originated in Iteration 2 were
              removed.

              We could calculate the DRE of all the different iterations quite
              easily:

              * Iteration 1: DRE = 5/25 = 20%
              * Iteration 2: DRE = (10+15) / [(25+25) - 5] = 25/45 = 55%;
              * Iteration 3: DRE = (5+5+10) / [(25+25+10) - (5+25)] = 20/30 = 66%;
              * Iteration 4: DRE = (5+5+0+5) / [(25+25+10+5) - (5+25+20)] = 15/15 = 100%;
            

The following table describes for each of the software development process phases the most important sources of defect injection and removal.

Management and Planning

A successful SCM implementation requires careful planning and management. This, in turn, requires an understanding of the organizational context for, and the constraints placed on, the design and implementation of the SCM process.

Some aspects that should be decided during this activity are:

• The types of documents to be managed and a document-naming scheme.
• Who takes responsibility for the CM procedures and creation of baselines?
• Policies for change control and version management.
• Tools to be used and process linked to their usage.

Configuration Identification

The software configuration identification activity identifies items to be controlled, establishes identification schemes for the items and their versions, and establishes the tools and techniques to be used in acquiring and managing controlled items. These activities provide the basis for the other SCM activities.

Configuration Item: A configuration item is any possible part of the development or delivery of a system or product that it's necessary to identify, produce, store, use and change individually. Many people associate configuration item with a source code file, but configuration items are not limited to that, many other items could be identified and managed such as:

• System data files
• System build files and scripts
• Requirements, Interface, Design specifications
• Test plans, procedures, data sets and results
• User documentation
• Compilers, Linkers, Debuggers
• Shell scripts
• Other related support tools

For each configuration item, additional information apart from the item itself is controlled by the SCM. As it is data about data, it is called metadata. Every configuration item must have a unique identification that is sometimes also called label. Additionally, metadata may include additional information such as:

• Name
• Version
• Status
• Date
• Location
• ...

A first step in controlling change is to identify the software items to be controlled. This involves understanding the software configuration within the context of the system configuration, selecting software configuration items, developing a strategy for labelling software items and describing their relationships, and identifying the baselines to be used.

Software Configuration: A software configuration is the set of functional and physical characteristics of software as set forth in the technical documentation or achieved in a product.

Selecting Configuration Items: It is an important process in which a balance must be achieved between providing adequate visibility for project control purposes and providing a manageable number of controlled items. The items of a configuration should include all the items that are part of a given software release.

Defining relationships and interfaces between the various configuration items is key as it also affects other SCM activities such as software building or assessing the impact of suggested changes. The identification or labelling scheme used should support the need to evolve software items and their relationships (e.g. configuration item X requires version A of configuration item Y).

Identifying the baselines is another critical task of SCM Identification. A software baseline is a set of software configuration items formally designated and fixed at a specific time during the software life cycle. The term is also used to refer to a particular version of a software configuration item that has been agreed on. In either case, the baseline can only be changed through formal change control procedures. A baseline, together with all approved changes to the baseline, represents the current approved configuration.

Configuration Change Control

The software is subject to continuous changes that are coming from different sources:

• Users that have new needs
• Developers that identify issues on the software
• Market forces that identify new business needs and opportunities

Change Control takes care of keeping track of these changes and ensures that they are implemented in a controlled manner.

The most important activity from a Change Control point of view is the definition of how are changes made:

• Can any developer change any configuration items?
• Do developers need to raise an issue or a Change Request before changing a configuration item?
• Which configurations or baselines developers can modify? E.g. some baselines should be read-only (tags), some others may be modifiable only after a Change Request has been made (branches) and some others may be modifiable with no restriction.
• Do changes need a third-party approval before they are added to the SCM system?
• How are other developers notified about changes?
• What is the link to the Issue Tracker system?
• Is there any need to test anything or build the system before accepting a change?

In short, the key thing is having a clear working flow, you can find an example about the FirefoxOS flow. the process for making changes is clear, it is also important to specify how the revision history of configuration items is going to be kept, how other developers are going to be notified about those changes:

• Maintaining baselines
• Processing changes
• Developing change report forms
• Controlling release of the product

Configuration Status Accounting

Configuration status accounting main target is recording and reporting of information needed for effective management of the software configuration.

The information that should be available is diverse:

• Which are the different available baselines.
• Which is the approved configuration.
• Which are the issues raised for every configuration.
• Which is the status of all the issues and changes.

In order to provide and control all this information a good tool support is needed. This could be part of the Configuration Item Management system or another independent tool that is integrated with it.

Reported information can be used by various organizational and project elements, including the development team, the maintenance team, project management, and software quality activities. Reporting can take the form of ad hoc queries to answer specific questions or the periodic production of predesigned reports. Some information produced by the status accounting activity during the course of the life cycle might become quality assurance records.

In addition to reporting the current status of the configuration, the information obtained by this system can serve as a basis for various measurements of interest to management, development, and SCM. Examples include the number of change requests per configuration item and the average time needed to implement a change request, defect arrival pattern per release/component...

Configuration Audting

The purpose of configuration audits is to ensure that the software product has been built according to specified requirements (Functional Configuration Audit, FCA), to determine whether all the items identified as a part of CI are present in the product baseline (Physical Configuration Audit, PCA), and whether defined SCM activities are being properly applied and controlled (SCM system audit or in-process audit). A representative from management, the QA department, or the customer usually performs such audits. The auditor should have competent knowledge of both SCM activities and the project.

The author should check the product is complete, consistent (e.g. "Are all the correct versions of files used in this current release?"), that no outstanding issues exist (e.g. "There are no critical defects or CRs") and that the product has passed all the required tests to ensure its quality.

The output of the audit should specify whether the product's performance requirements have been achieved by the product design and the product design has been accurately documented in the configuration documentation.

In order to properly perform this activity is important to:

• Define the audit schedule and procedures.
• Identify who will perform the audits.
• Do the audits on the established baselines.
• Generate audit reports.

Release Build, Management and Delivery

The term "release" is used to refer to a software configuration that is distributed outside of the development team. This includes internal releases as well as distribution to end-users. When different versions of software are available for different platform configurations it is frequently necessary to create multiple releases for delivery.

Building the release:

In order to release a software product, the configuration items must be combined, packaged with the right configuration and in most of the cases built into an executable program that can be installed by the customers. Build instructions ensure that the proper build steps are taken and in the correct sequence. In addition to building software for new releases, it is usually also necessary for SCM to have the capability to reproduce previous releases for recovery, testing, maintenance, or additional release purposes.

Software is built using particular versions of supporting tools, such as compilers. It might be necessary to rebuild an exact copy of a previously built software configuration item. In this case, the supporting tools and associated build instructions need to be under SCM control to ensure availability of the correct versions of the tools (i.e. not only source code evolve, but also the tools we use).

A tool capability is useful for selecting the correct versions of software items for a given target environment and for automating the process of building the software from the selected versions and appropriate configuration data. For large projects with parallel development or distributed development environments, this tool capability is necessary. Most software engineering environments provide this capability.

Release Management:

Software release management encompasses the identification, packaging, and delivery of the elements of a product, for example, executable program, documentation, release notes, and configuration data.

Given that product changes can occur on a continuing basis, one concern for release management is determining when to issue a release. Some aspects to take such a decision are the severity of the problems addressed by the release and the measurements of the fault densities of prior releases.

The packaging task must identify which product items are to be delivered, and then select the correct variants of those items, given the intended application of the product. The information documenting the physical contents of a release is known as a version description document. The release notes typically describe new capabilities, known problems, and platform requirements necessary for proper product operation. The package to be released also contains installation or upgrading instructions. The latter can be complicated because some current users might have versions that are several releases old.

Finally, in some cases, the release management activity might need to track the distribution of the product to various customers or target systems. An example would be a case where the supplier was required to notify a customer of newly reported problems. A tool capability is needed for supporting these release management functions. It is useful to have a connection with the tool capability supporting the issue tracker in order to map release contents to the issues that have been received. This tool capability might also maintain information on various target platforms and on various customer environments.

Summary

Introduction

This chapter provides a set of best practices or patterns that should be used in SCM. There are multiple tools that can be used for SCM. Some of them focus on the configuration identification and change control, some others pay special attention to the auditing and accounting and some others are focused on the build and release part. In most of the cases different tools are required and what is important is all the tools are properly integrated. For instance, a typical situation is using a tool for managing the source code (e.g. Subversion or git), another one for keeping track of the issues, defects or releases (e.g. Redmine or Bugzilla), another one for Agile Management (e.g Trello) and maybe another one for Continous Integration (e.g. Travis). In such a multi-tool environment it is important to ensure that the changes on the configuration items can be linked to the issues and releases in the issue tracker and the agile manaagement tool.

It is important to stress that there are multiple different paradigms for managing the source code, being the most important distinction whether the system is centralized or distributed. Linus Torvalds (who was the creator of git) gave an interesting speech in the Google Tech Talk event in which he compared both approaches [[LINUS-SCM-GOOGLE]].

The patterns described in this chapter try to be generic enough so they can be applied in centralized and distributed systems. However some of them may also apply to one of them. Additionally, it is important to stress that usually, a distributed system can be configured to work in a centralized way. Additinally, during the last years distributed systems have proliferated and it seems they have become the de-facto standard for SCM.

Configuration Identification

Best Practices and Patterns

Tips for branching

Before Git was used, branches were used with a lot of care care since merging in other SCM systems such as SVN was very difficult. Merging is the process by which two configuration items are combined into a new one. Depending on the amount of configuration items to be combined and on the type of changes done in them, and on the SCM system used, merging can be a very difficult operation.

Branches are created to save some work by allowing developers to work in independent features in an independent manner. However, that may end up in some times in spending extra time doing a difficult merge task. The reason why Git is so successful nowadays is that it has simplified the way merges are done and hence has enabled developers to create and work on separate branches.

However, easy merging, does not mean branches should be used without care. For instance, in general, overcomplicated structures where branches are created from branches different to master continuously (arborescent approach) should be avoided.

Branching works better when you integrate with the origin of the branch as quickly as possible.

Best Practice 1: Simplify the branching model.

Althoug branching is cheap in systems such as Git that should not be an excuse for creating too complex tree structures diverging from the master branch. Ask developers to branch from the master branch that is the "home codeline" in which you merge all of your development on, except in special circumstances. Branching always from master reduces merging and synchronization effort by requiring fewer transitive change propagations.

It is also important that the expected branches are planned in advance and that a branch diagram is used. Having a diagram is of huge help to the development as it allows at a glance to have a clear understanding of the different branches available and the relationship across them. There are many tools for getting such a diagram automatically.

You can see below an example of such a diagram:

Best Practice 2: Create specific development branches for every feature you implement

Like shown in the previous diagram (branches Story A and Story B), for every feature to be added or for every bug you fix you should create a separate branch so you can work in a isolated and independent manner.

Best Practice 3: Development branches should be short-lived.

More information about when should a development branch be merged will be provided in the following sections, but by having a look at the diagram is easy understand than the later we merge, the more difficult it will be as branches will have diverged more.

Best Practice 4: When development branches must live for a long time, relatively frequent intermediate merges should be done.

When you create a develop branch and it's going to take a long time before you can merge your changes to master branch, try to sync with master frequently so you can avoid your working branch to diverge from master. The more time you wait, the more difficult the merge would be.

Best Practice 5: Branch Customer Releases.

When a new software version to users, the "usual" situation is that the team must work at least in two versions in parallel:

• A new version with new features (that is developed in the master branch)
• The released version, where typically only bugfixes should be added. This is usually done in a "release" branch. Please note that this bugfixes should also land on the master branch if that branch is also affected.

Due to that, when a version is released to customers a release branch should be created. In this way bugfixing can be done on the release branch without exposing the customer to new feature work in progress on the mainline.

The typical workflow for customer releases is:

• A release branch (e.g. v1.0) is created based on the master branch when the team thinks the software is ready for release (say, a 1.0 release). In that mmoment the content and history of both branches (v1.0 and master) will be the same.
• The team continues to work in both branches in parallel. Depending on the release strategy the product could be released at this moment to all the users, maybe to some early/beta users or maybe just for another testing round.
• In any case, if bugs are discovered in either version, fixes are ported back and forth as necessary. Usually, as time passes, only critical fixes will be landed in the release branch, that eventually will be frozen as new releases supersede it.
• Fixing bugs that affect both branches might be done using different strategies. A recommended one is to land all the code always in the master branch and then uplift or cherrypick those changes that affect the release to the release branch. Obviously, as time passes and branches diverge there might be some bugs that affect only the release branch, in that case, a direct merge on the release branch could be done.

Best Practice 6: Branch long-lived parallel efforts.

Long lived parallel efforts that multiple people will be working on should be done in independent branches. Imagine you want to experiment with a new feature and you know that before having something that can be merged in the master branch a lot of time is going to be needed. In that case, it makes sense to create a specific branch (similar to the release branch) so that the team can work in that feature while others can check your progress.

Best Practice 7: Be always flexible, there may be some very strong reasons for breaking these "rules".

These are just a set of recommendations, but there are different ways to work with branches and all of them are right and wrong at the same time as it is impossible to have a perfect framework. For instance, some authors [[SUCCESSFUL-GIT-BRANCHING]] promote the idea of having an integration branch called develop and with an infinite lifetime (as master) as shown in the following figure:

Continuous Integration and Building

Since many people are making changes in the repository, it is impossible for a developer to be 100% sure that the entire system builds correctly after they integrate their changes in the repository even if they create a local build before and extensively test it.

Continuous Integration (CI) is a software development practice where members of a team commit their work frequently, leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. This approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

Releasing

A release is a version of the product that is made available to its intended customers. External releases are published to end-users whereas internal releases are made available only to developers. The releases are identified by release numbers which are totally independent from the SCM version numbers.

Releases can be also classified in full or partial releases, depending on whether it requires a complete installation or not respectively. Partial releases require a previous full release to be installed.

Release creation involves collecting all files and documentation required to create ystem release. Configuration descriptions have to be written for different hardware and installation scripts have to be written. The specific release must be documented to record exactly what files were used to create it. This allows it to be re-created if necessary

Release planning is concerned with when to issue a system version as a release. The following factors should be taken into account for defining a release strategy:

• Technical Quality of the System: If serious system faults are reported which affect the way in which many customers use the system, it may be necessary to issue a fault repair release. However, minor system faults may be repaired by issuing patches (often distributed over the Internet) that can be applied to the current release of the system.
• Platform Changes: You may have to create a new release of a software application when a new version of the operating system platform is released.
• Lehman's fifth law: This suggests that the increment of functionality that is included in each release is approximately constant. Therefore, if there has been a system release with significant new functionality, then it may have to be followed by a repair release.
• Competition: A new system release may be necessary because a competing product is available.
• Marketing Requirements: The marketing departm ent of an organisation may have made a commitment for releases to be available at a particular date.
• Customer Change Proposals: For customised systems, customers may have made and paid for a specific set of system change proposals and they expect a system release as soon as these have been implemented

Controlling Changes

There is a need to submit continuously changes to the repository. The reasons for checking-in changes are multiple:

• Defects: A defect has been detected and needs to be fixed.
• New Features: New features must be added to the software.
• Improvements: A functionality already existing can be improved.

Even in a continuous integration model, it is important to have the possibility to control the changes that have been done to the configuration items in the repository. Control in this context does not mean approval but traceability. I.e. It is not always needed that someone approves a change, but what is needed is that it is possible to identify for every change committed the reasons for it. Lack of control in the process leads to project failures, confusion and chaos. Using a good control mechanism enables communication, sharing data and efficiency.

Depending on the codeline in which the changes are done, the level of information required and the flow that should be followed for implementing and approving it should be different.

For instance, changes in master should be encouraged rather than discouraged. In order to do so, developers should be free to commit their changes to the repository if:

• The change is inline with the product backlog or requirements he was developing for.
• He raises an issue on the issue tracker describing the reason for the changes and the changes themselves.
• When he performs the commit, he indicates in the commit information the related issue (the one he created).
• After the commit is done, the is sue is either automatically or manually marked as resolved.

Additionally, anybody in the development team should be free to raise additional issues that can be assigned to anybody within the team. Giving freedom to the development team (within some limits) is usually a good idea.

If the changes are going to be applied in a branch that was created based on a commercial release, the process usually follows a more strict control. For instance:

• The developer creates an issue in the issue tracker, this kind of issues are usually called Change Request (CR). A CR typically has the following information:
  • Project name, date, requestor, and priority.
  • Description of the problem.
  • Affected configuration items, branches and releases.
  • Suggested fix.
  • Severity.
  • Log files, screen shots...
• The CR is then analysed by the configuration control board that decides what to do, approve it or reject it.
  • In case it is rejected the initiator could review and create a new version (depending on the reasons for the rejection).
  • In case it is approved, it is assigned to a developer who will implement it and add it to the repository.

Regardless of how "controlled" is the implementation of changes in the repository, the system used should allow some features such as:

• Identify which changes have been implemented between two different baselines.
• Check the status of a particular issue, defect or CR for the different platforms and releases.
• Check if the resolution of a problem can be merged in a branch.

With respect to the tools, there are multiple tools that allow issue tracking. There are commercial ones such as Jira or free ones such as Bugzilla or Redmine. The later is a very powerful as it is quite flexible and can be integrated with the most popular source code management tools such as git and svn. Additionally, other Agile management tools can be used such as Trello.

Additionally, the Source Code Management tools should also have adequate authorization mechanism to ensure the traceability of the changes, i.e. identified who made any particular change in the repository. For instance, SVN offers and authentication mechanism and allows to use others such as LDAP. The important thing is that SVN identifies which user has done which commit in order to trace back a change to the author. SVN also allows the definition of permissions in per-branch or per-configuration item basis, access to some branches may be only allowed to some users.

Criteria 1: Functional and Structural Testing

The main difference between functional and structural testing is the knowledge (or lack of knowledge) about the software internals and hence the related focus:

• Functional testing considers that there is no information about software internals or implementation details. Hence, this type of testing is usally knonw as "Black-Box". This affects both the test defintion as well as the test execution.
• Structural testing considers that the information about the internals of the software is known and should be used for both test definition an execution. This type of testing are usually known as "white-box" testing.

Criteria 2: Coverage vs. Usage based testing

One of the most important decisions that should be taken by the QA (and the whole software and product) team is to decide when to stop testing.

Obviously, an easy (but wrong decision) would be stopping based on the resources, e.g. stop when you run out of time or of money or time. As such a decision would lead to quality problems, we need to find a quality-based criteria to decide when our product has passed enough tests. In order to identify when the product has reached the quality goals, there are different points of view:

• Measure the quality directly by in-use metrics. The issue is that this approach requires actual customer usage.
• Measure the quality indirectly through the execution of a set of tests that provide a good test coverage.

Criteria 3: Test Target

Tests are frequently grouped by where they are added in the software development process, or by the target (element) to be tested.

Criteria 4: Objectives of testing

Although testing has a common set of goals, the targets of testing can be very different. Some examples of type testing based on their goals are listed in this section.

Testing Planning and Preparation

Most of the key decisions about testing are made during this stage. During this phase an overall testing strategy is fixed by making the following decisions:

• Overall objectives and goals, which can be refined into specific goals for specific testing. Some specific goals include reliability for usage-based statistical testing or coverage for various traditional testing techniques.
• Objects to be tested and the specific focus: Functional testing views the software product as a black-box and focuses on testing the external functional behavior; while structural testing views the software product or component as a (transparent) whitebox and focuses on testing the internal implementation details.

As soon as the first models are being generated (for example, usage models, system models, architectural models, etc), they can be used to generate test cases: A test case is a collection of entities and related information that allows a test to be executed or a test run to be performed. The collection of individual test cases that will be run in a test sequence until some stopping criteria are satisfied is called a test suite. IEEE Standard 610 (1990) defines test case as follows:

• A set of test inputs, execution conditions, and ex pected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
• (IEEE Std 829-1983) Documentation specifying inputs, predicted results, and a set of execution conditions for a test item.

According to Ron Patton: "Test cases are the specific inputs that you'll try and the procedures that you'll follow when you test the software."

From a more practical point of view, a test case is composed by:

• Preconditions that should be established before the test is conducted.
• Clear sequence of actions and input data that constitutes the test sequence.
• Expected Result.

On the other hand, a test run, is a dynamic unit of specific test activities in the overall testing sequence on a selected testing object. Each time a static test case is invoked, an individual dynamic test run is created.

One aspect that should be considered when planning the test cases is the sequencing of the individual test cases and the switch-over from one test run to another. Several concerns affect the specific test procedure to be used, including:

• Dependencies among individual test cases. For instance, does a test case require the execution of another test case before?
• Defect detection related sequencing. Many problems can only be effectively detected after others have been discovered and fixed.
• Natural grouping of test cases, such as by functional and structural areas or by usage frequencies, can also be used for test sequencing and to manage parallel testing.

Testing Execution

The most important activities related with test execution are:

• Allocating test time and resources.
• Invoking and running tests, and collecting execution information and measurements.
• Checking testing results and identifying system failures.

One of the critical aspects in order to fulfill the objectives of testing is checking if the result of the test run is successful or not. In order to do so, it must be possible to observe the results of the test and determine whether the expected result was achieved or not.

Is enough with observing the results? In some situations, such as in object-oriented software, the execution of a test-run may have affected the state of an object. That state might also affect in the future the software that has been tested. Due to that, in some situations it is helpful to examine the state of some objects before and after a test is conducted. The reason for that is that only a small percentage of the overall functionality of an object can be observed via the return values.

This may be in conflict with using a "black-box" testing approach, in which only events observable outside can be used to verify the results of a test-run. However, the meaning of observed may be different for different software projects: outside a method? An object? The whole software?

When a failure is observed, it needs to be recorded and tracked until their resolution. In order to allow developers to trace back the failure to the bug causing it, it is important that detailed information about failure observations and the related activities is registered.

But not only failures must be registered, successful executions also need to be recorded as it is very important for regression testing.

In general for every test-run the following information should be gathered:

• Run identification.
• Timing. Start and end time.
• Tester. The tester who attempted the te st run.
• Transactions. Transactions handled by the test run.
• Results. Result of the test run.

Testing Analysis and Follow-up

The results of the testing activities (i.e. the measurement data collected during test execution), together with other data about the testing and the overall environment provide valuable feedback to test execution and other testing and development activities.

Obviously, as a consequence of testing, there are some direct follow-up activities:

• Defect fixing. The development team must repair detected defects.
• Management decisions, such as product release and transition from one development phase or sub-phase to another. For instance, given the results of testing I can decid if my product mature enough for being published.

In order to fix an issue, it is important to follow these steps:

1. Understanding the problem by studying the execution record.
2. Being able to recreate the same problem scenario and observe the same problem.
3. Problem diagnosis too examine what kind of problem it is, where, when and possible causes.
4. Fault locating, to identify the exact location(s) of fault(s).
5. Defect fixing, to fix the located fault(s) by adding, removing, or correcting certain parts of the code. Sometimes, design and requirement changes could also be triggered or propagated from the above changes due to logical linkage among the different software components.

In order to take appropriate management decisions, some analysis can be performed on the overall testing results:

1. Reliability analysis for usage-based testing, which can be used to assess current product reliability. Sometimes, low reliability areas can be identified for focused testing and reliability improvement. In case of this type of testing, it is also very helpful to detect problems non linked to defects, using the PUM metric.
2. Coverage analysis for coverage-based testing, which can be used as a surrogate for reliability and used as the stopping criterion.
3. Overall defect analysis, which can be used to examine defect distribution and to identify high-defect areas for focused remedial actions.

Test Coverage criterion

A test coverage criterion is a rule about how to select tests and when to stop testing. One basic issue in testing research is how to compare the effectiveness of different test coverage criteria. The standard approach is to use the subsumes relationship.

Structural Testing

Structural testing coverage is based on the structure of the source code. The simplest structural testing criterion is every statement coverage, often called C0 coverage.

C0 - Every State Coverage

This criterion is that every statement of the source code should be executed by some test case. The normal approach to achieving C0 coverage is to select test cases until a coverage tool indicates that all statements in the code have been executed.

The pseudocode in the following table implements the triangle problem. The table also shows which lines are executed by which test cases. Note that the first three statements (A, B, and C) can be considered parts of the same node.

Test Cases for Triangle Problem - C0 - Structural Testing

Node	Source	3,4,5	3,5,3	0,1,0	4,4,4
A	read a,b,c	*	*	*	*
B	type="scalene"	*	*	*	*
C	if((a==b) || (b==c) || (a==c)))	*	*	*	*
D	type="isosceles"		*	*	*
E	if((a==b)&&(b==c))	*	*	*	*
F	type="equilateral"				*
G	if((a>=b+c) || (b>=a+c) || (c>=a+b)))	*	*	*	*
H	type="not a triangle"			*
I	if((a<=0>) || (b<=0>) || (c<=0>)))	*	*	*	*
J	type="bad inputs"			*
K	print type	*	*	*	*

By the fourth test case, every statement has been executed. This set of test cases is not the smallest set that would cover every statement. However, finding the smallest test set would often not find a good test set.

C1 - Every Branch Testign

A more thorough test criterion is every-branch testing, which is often called C1 test coverage. In this criterion, the goal is to go both ways out of every decision.

If we model the program of previous table as a control flow graph, this coverage criterion requires covering every arc in the following control flow diagram.

Next table shows the test cases identified with this criteria.

Test Cases for Triangle Problem - C1 Approach - Structural Testing

Arcs	Test Case: (3,4,5)	Test Case: (3,5,3)	Test Case: (0,1,0)	Test Case: (4,4,4)
ABC-D		*	*	*
ABC-E	*
D-E		*	*	*
E-F				*
E-G	*	*	*
F-G				*
G-H			*
G-I	*	*		*
H-I			*
I-J			*
I-K	*	*		*
J-K			*

Every Path Testign

Even more thorough is the every-path testing criterion. A path u is a unique sequence of program nodes that are executed by a test case. In the testing matrix above, there were eight subdomains. Each of these just happens to be a path. In that example, there are sixteen different combinations of T and F. However, eight of those combinations are infeasible paths. That is, there is no test case that could have that combination of T and F for the decisions in the program. It can be exceedingly hard to determine if a path is infeasible or if it is just hard to find a test case that executes that path.

Most programs with loops will have an infinite number of paths. In general, every-path testing is not reasonable.

Next table shows the eight feasible paths in the triangle pseudocode as well as the test cases required for testing all of them.

Test Cases for Triangle Problem - Every Path Approach - Structural Testing

Path	T/F	Test Case	Output
ABCEGIK	FFFF	3,4,5	Scalene
ABCEGHIK	FFTF	3,4,8	Not a triangle
ABCEGIJK	FFTT	0,5,6	Bad inputs
ABCDEGIK	TFFF	5,8,5	Isosceles
ABCDEGHIK	TFTF	3,8,3	Not a triangle
ABCDEGHIJK	TFTT	0,4,0	Bad Inputs
ABCDEFGIK	TTFF	3,3,3	Equilateral
ABCDEFGHIJK	TTTT	0,0,0	Bad Inputs

Multiple Condition Coverage

A multiple-condition testing criterion requires that each primitive relation condition is evaluated both true and false. Additionally, all combinations of T/F for the primitive relations in a condition must be tried. Note that lazy evaluation of expressions will eliminate some combinations. For example, in an "and" of two primitive relations, the second will not be evaluated if the first one is false.

In the pseudocode for the triangle example, there are multiple conditions in each decision statement as displayed in the tables below. Primitives that are not executed because of lazy evaluation are shown with an 'X'.

Test Cases for Triangle Problem - Multiple Condition: Condition if(a==b||b==c||a==c) - Structural Testing

Combination	Possible Test Case	Branch
TXX	3,3,4	ABC-D
FTX	4,3,3	ABC-D
FFT	3,4,3	ABC-D
FFF	3,4,5	ABC-E

Test Cases for Triangle Problem - Multiple Condition: Condition (a==b&&b==c) - Structural Testing

Combination	Possible Test Case	Branch
TT	3,3,3	E-F
TF	3,3,4	E-G
FX	4,3,3	E-G

Test Cases for Triangle Problem - Multiple Condition: Condition (a>=b+c||b>=a+c||c>=a+b) - Structural Testing

Combination	Possible Test Case	Branch
TXX	8,4,3	G-H
FTX	4,8,3	G-H
FFT	4,3,8	G-H
FFF	3,3,3	G-I

Test Cases for Triangle Problem - Multiple Condition: Condition (a<=0||b<=0||c<=0) - Structural Testing

Combination	Possible Test Case	Branch
TXX	0,4,5	I-J
FTX	4,-2,-2	I-J
FFT	5,-4,3	I-J
FFF	3,3,3	I-K

Subdomain Testing

Subdomain testing is the idea of partitioning the input domain into mutually exclusive subdomains and requiring an equal number of test cases from each subdomain. This was basically the idea behind the test matrix. Subdomain testing is more general in that it does not restrict how the subdomains are selected. Generally, if there is a good reason for picking the subdomains, then they may be useful for testing. Additionally, the subdomains from other approaches might be subdivided into smaller subdomains. Theoretical work has shown that subdividing subdomains is only effective if it tends to isolate potential errors into individual subdomains.

Every-statement coverage and every-branch coverage are not subdomain tests. There are not mutually exclusive subdomains related to the execution of different statements or branches. Every-path coverage is a subdomain coverage, since the subdomain of test cases that execute a particular path through a program is mutually exclusive with the subdomain for any other path.

For the triangle problem, we might start with a subdomain for each output. These might be further subdivided into new subdomains based on whether the largest or the bad element is in the first position, second position, or third position (when appropriate). Next table shows the subdomains and test cases for every subdomain.

Test Cases for Triangle Problem - Subdomain Testing

Subdomain	Possible Test Case
Eauilateral	3,3,3
Isosceles first largest	8,5,5
Isosceles second largest	5,8,5
Isosceles third largest	5,5,8
Scalene first largest	5,4,3
Scalene second largest	3,5,4
Scalene third largest	3,4,5
Not a triangle first largest	8,3,3
Not a triangle second largest	3,8,4
Not a triangle third largest	4,3,8
Bad Inputs first largest	4,3,0
Bad Inputs second largest	3,4,0
Bad Inputs third largest	-1,4,5

Data Flow Testing

Data flow testing is testing based on the flow of data through a program. Data flows from where it is defined to where it is used.

A definition of data, or DEF, is when a value is assigned to a variable. For example, with respect to a variable x, nodes containing statements such as input x and x = 2 would both be defiing nodes.

Usage nodes (USE) refer to situations in which a variable is used by the software. Two main kinds of use have been identified:

• The computation use, or C-USE, is when the variable is used in a computation (e.g. it appears on the right-hand side of an assignment statement) such as in print x or a = 2+x. A C-USE is said to occur on the assignment statement.
• The predicate use, or P-USE, is when the variable is used in the condition of a decision statement (e.g. if x>6). A P-USE is assigned to both branches out of the decision statement.

There are also three other types of usage node, which are all, in effect, subclass of the C-USE type:

• O-use: output use - the value of the variable is output to the external environment (for instance, the screen or a printer print(x)).
• L-use: location use - the value of the variable is used, for instance, to determine which position of an array is used (e.g. a[x]).
• I-use: iteration use - the value of the variable is used to control the number of iterations made by a loop (for example: for (int i = 0;i <= x; i++))

A definition free path, or def-free, is a path from a definition of a variable to a use of that variable that does not include another definition of the variable.

Next figure depicts the Control Flow Graph of Triangle Problem and is annotated with the definitions and uses of the variables type, a, b, and c.

More details about the control-flow procedure and examples can be found in the paper "Data Flow Testing - CS-399: Advanced Topics in Computer Science, Mark New (321917)"

Random Testing

Random testing is accomplished by randomly selecting the test cases. This approach has the advantage of being fast and it also eliminates biases of the testers. Additionally, statistical inference is easier when the tests are selected randomly. Often the tests are selected randomly from an operational profile.

For example, for the triangle problem, we could use a random number generator and group each successive set of three numbers as a test set. We would have the additional work of determining the expected output. One problem with this is that the chance of ever generating an equilateral test case would be very small. If it actually happened, we would probably start questioning our pseudo random number generator.

Boundary Testing

Often errors happen at boundaries between domains. In source code, decision statements determine the boundaries. If a decision statement is written as x<1 instead of x<0, the boundary has shifted. If a decision is written x=<1, then the boundary, x=1, is in the true subdomain. In the terminology of boundary testing, we say that the on tests are in the true domain and the off tests are values of x greater than 1 and are in the false domain.

If a decision is written x<1 instead of x=<1, then the boundary, x=1, is now in the false subdomain instead of in the true subdomain.

Boundary testing is aimed at ensuring that the actual boundary between two subdomains is as close as possible to the specified boundary. Thus, test cases are selected on the boundary and off the boundary as close as reasonable to the boundary. The standard boundary test is to do two on tests as far apart as possible and one off test close to the middle of the boundary.

Next figure shows a simple boundary. The arrow indicates that the on tests of the boundary are in the subdomain below the boundary. The two on tests are at the ends of the boundary and the off test is just above the boundary halfway along the boundary.

In the triangle example, for the primitive conditions, a>=b+c or b>=a+c or c >= a + b, we could consider the boundary. Since these are in three variables as a plane in 3D space. The on tests would be two (or more) widely separated tests that have equality - for example, (8,1,7) and (8,7,1). These are both true. The off test would be in the other domain (false) and would be near the middle - for example, (7.9, 4,4).

Goal1: Fully Automated Tests

A test that can be run without any Manual Intervention is a Fully Automated Test. Satisfying this criterion is a prerequisite to meeting many of the other goals. Yes, it is possible to write Fully Automated Tests that don't check the results and that can be run only once. The main() program that runs the code and directs print statements to the console is a good example of such a test.

Goal2: Self-checking Tests

A Self-Checking Test has encoded within it everything that the test needs to verify that the expected outcome is correct. The Test Runner "calls us" only when a test did not pass; as a consequence, a clean test run requires zero manual effort. Many members of the xUnit family provide a Graphical Test Runner (see Test Runner) that uses a green bar to signal that everything is OK; a red bar indicates that a test has failed and warrants further investigation.A

Goal3: Repeatable Tests

A Repeatable Test can be run many times in a row and will produce exactly the same results without any human intervention between runs. Unrepeatable Tests increase the overhead of running tests significantly. This outcome is very undesirable because we want all developers to be able to run the tests very frequently, as often as after every "save". Unrepeatable Tests can be run only once before whoever is running the tests must perform a Manual Intervention. Just as bad are non determinstic Tests that produce different results at different times; they force us to spend lots of time chasing down failing tests. The power of the red bar diminishes significantly when we see it regularly without good reason. All too soon, we begin ignoring the red bar, assuming that it will go away if we wait long enough. Once this happens, we have lost a lot of the value of our automated tests, because the feedback indicating that we have introduced a bug and should fix it right away disappears. The longer we wait, the more effort it takes to find the source of the failing test.

Tests that run only in memory and that use only local variables or fields are usually repeatable without us expending any additional effort. Unrepeatable Tests usually come about because we are using a Shared Fixture of some sort. In such a case, we must ensure that our tests are self-cleaning as well. When cleaning is necessary, the most consistent and foolproof strategy is to use a generic Automated Teardown mechanism. Although it is possible to write teardown code for each test, this approach can result in Erratic Tests when it is not implemented correctly in every test.

Goal4: Simplicity

Coding is a fundamentally difficult activity because we must keep a lot of information in our heads as we work. When we are writing tests, we should stay focused on testing rather than coding of the tests. This means that tests must be simple - simple to read and simple to write. They need to be simple to read and understand because testing the automated tests themselves is a complicated endeavor. They can be tested properly only by introducing the very bugs that they are intended to detect; this is hard to do in an automated way so it is usually done only once (if at all), when the test is first written. For these reasons, we need to rely on our eyes to catch any problems that creep into the tests, and that means we must keep the tests simple enough to read quickly.

Of course, if we are changing the behavior of part of the system, we should expect a small number of tests to be affected by our modifications. We want to Minimize Test Overlap so that only a few tests are affected by any one change. Contrary to popular opinion, having more tests pass through the same code doesn't improve the quality of the code if most of the tests do exactly the same thing.

Tests become complicated for two reasons:

• We try to verify too much functionality in a single test.
• Too large an "expressiveness gap" separates the test scripting language (e.g. Java) and the before/after relationships between domain concepts that we are trying to express in the test.

The tests should be small and test one thing at a time. Keeping tests simple is particularly important during test-driven development because code is written to pass one test at a time and we want each test to introduce only one new bit of behavior. We should strive to Verify One Condition per Test by creating a separate Test Method for each unique combination of pre-test state and input.

The major exception to the mandate to keep Test Methods short occurs with customer tests that express real usage scenarios of the application. Such extended tests offer a useful way to document how a potential user of the software would go about using it; if these interactions involve long sequences of steps, the Test Methods should reflect this reality.

Goal5: Maintainability

Tests should be maintained along with the rest of the software. Testware must be much easier to maintain that production software as otherwise:

• It will slow the development down.
• It will get left behind.
• It will have less value.
• Developers will go back to manual testing.

What is TDD

The steps of test first design (TFD) are overviewed in the UML activity diagram of next figure. The first step is to quickly add a test, basically just enough code to fail. Next you run your tests, often the complete test suite although for sake of speed you may decide to run only a subset, to ensure that the new test does in fact fail. You then update your functional code to make it pass the new tests. The fourth step is to run your tests again. If they fail you need to update your functional code and retest. Once the tests pass the next step is to start over (you may first need to refactor any duplication out of your design as needed, which is what converts TFD into TDD).

Dean Leffingwell describes TDD with this simple formula:

TDD = Refactoring + TFD.

TDD completely turns traditional development around. When you first go to implement a new feature, the first question that you ask is whether the existing design is the best design possible that enables you to implement that functionality. If so, you proceed via a TFD approach. If not, you refactor it locally to change the portion of the design affected by the new feature, enabling you to add that feature as easy as possible. As a result you will always be improving the quality of your design, thereby making it easier to work with in the future.

Instead of writing functional code first and then your testing code as an afterthought, if you write it at all, you instead write your test code before your functional code. Furthermore, you do so in very small steps - one test and a small bit of corresponding functional code at a time. A programmer taking a TDD approach refuses to write a new function until there is first a test that fails because that function isn't present. In fact, they refuse to add even a single line of code until a test exists for it. Once the test is in place they then do the work required to ensure that the test suite now passes (your new code may break several existing tests as well as the new one). This sounds simple in principle, but when you are first learning to take a TDD approach it proves require great discipline because it is easy to "slip" and write functional code without first writing a new test.

An underlying assumption of TDD is that you have a testing framework available to you. Agile software developers often use the xUnit family of open source tools, such as JUnit or VBUnit, although commercial tools are also viable options. Without such tools TDD is virtually impossible. Next figure presents a UML state chart diagram for how people typically work with the xUnit tools (source Keith Ray).

Kent Beck, who popularized TDD, defines two simple rules for TDD (Beck 2003):

• First, you should write new business code only when an automated test has failed.
• Second, you should eliminate any duplic ation that you find.

Beck explains how these two simple rules generate complex individual and group behavior:

• You design organically, with the running code providing feedback between decisions.
• You write your own tests because you can't wait 20 times per day for someone else to write them for you.
• Your development environment must provide rapid response to small changes (e.g you need a fast compiler and regression test suite).
• Your designs must consist of highly cohesive, loosely coupled components (e.g. your design is highly normalized) to make testing easier (this also makes evolution and maintenance of your system easier too).

For developers, the implication is that they need to learn how to write effective unit tests.

TDD also improves documentation

Most programmers don't read the written documentation for a system, instead they prefer to work with the code. And there's nothing wrong with this. When trying to understand a class or operation most programmers will first look for sample code that already invokes it. Well-written unit tests do exactly this - they provide a working specification of your functional code - and as a result unit tests effectively become a significant portion of your technical documentation. The implication is that the expectations of the pro-documentation crowd need to reflect this reality. Similarly, acceptance tests can form an important part of your requirements documentation. This makes a lot of sense when you stop and think about it. Your acceptance tests define exactly what your stakeholders expect of your system, therefore they specify your critical requirements. Your regression test suite, particularly with a test-first approach, effectively becomes detailed executable specifications.

Are tests sufficient documentation? Very likely not, but they do form an important part of it. For example, you are likely to find that you still need user, system overview, operations, and support documentation. You may even find that you require summary documentation overviewing the business process that your system supports. When you approach documentation with an open mind, I suspect that you will find that these two types of tests cover the majority of your documentation needs for developers and business stakeholders. Furthermore, they are an important part of your overall efforts to remain as agile as possible regarding documentation.

Why TDD

A significant advantage of TDD is that it enables you to take small steps when writing software. This is far more productive than attempting to code in large steps. For example, assume you add some new functional code, compile, and test it. Chances are pretty good that your tests will be broken by defects that exist in the new code. It is much easier to find, and then fix, those defects if you've written two new lines of code than two thousand. The implication is that the faster your compiler and regression test suite, the more attractive it is to proceed in smaller and smaller steps. I generally prefer to add a few new lines of functional code, typically less than ten, before I recompile and rerun my tests.

The act of writing a unit test is more an act of design than of verification. It is also more an act of documentation than of verification. The act of writing a unit test closes a remarkable number of feedback loops, the least of which is the one pertaining to verification of function.

The first reaction that many people have to agile techniques is that they're ok for small projects, perhaps involving a handful of people for several months, but that they wouldn't work for "real" projects that are much larger. That's simply not true. Beck (2003) reports working on a Smalltalk system taking a completely test-driven approach which took 4 years and 40 person years of effort, resulting in 250,000 lines of functional code and 250,000 lines of test code. There are 4000 tests running in under 20 minutes, with the full suite being run several times a day. Although there are larger systems out there, it's clear that TDD works for good-sized systems.

A simple example

You are asked to write a code to extract the UK postal area code from any given full UK postcode. Example Input "SS17 7HN", output should be "SS", for input "B43 4RW", output should be "B".

Before starting coding, you need to create the tests and for doing so you need to think about the structure of the software. An obvious approach is creating a class named PostCode with a method that retrieves the postal area code (e.g. areaCode()). In that way, in order to calculate the are code something similar to:

postCode = new PostCode("SS17 THN");

string area = postCode.areaCode();

Hence, the first thing that should be done is developing the test cases for such a solution, as you have been already provided with two examples, a good idea is using them as the test cases:

• Test Case 1: ("SS17 THN") -> "SS"
• Test Case 2: ("B43 4RW") -> "B"

Depending on the programming language, you should build those test cases through the xUnit tool. If you try to execute them, both are going to fail because the class PostCode does not exist yet.

The next step is building an empty class PostCode with a method postalArea that always return the string postcode.

In this case, again the tests are going to fail, as the return value is not the expected one.

In the following iteration we could implement the solution for the first part of the problem, when the postal code is 8 chars long, the area code is composed by the 2 first ones. If we try again to run the tests, the first one ("SS17 THN") will complete successfully whereas the second one will fail. We have added a very reduce functionality, and we have nearly immediately checks that what we has added is correct.

In the following iteration we could implement the solution for the second part of the problem, when the postal code is 7 chars long, the area code is composed by the firs ones. If we try again to run the tests, both will complete successfully. Again, we have just added very few lines code (e.g. could be 2-3), and have checked immediately that what we has added is correct.

Imagine now that you have been told that there are postal codes of 8 digits in which only the first one denotes the area, e.g. "W1Y2 3RD", now you should first create a new test case for that new example. Obviously, it will fail as it will return "W1" instead of "W " but adding the funcionality to implement the new feature should be quite safe as it would be easy to check if in the process of implementing it you are breaking the existing features.

Formal Code Inspection

For historical reasons, formal reviews are usually called inspections. This is due to the work Michael Fagan conducted and presented in his 1976 study at IBM regarding the efficacy of peer reviews. We are going to called them Formal Code Inspection to distinguish them from other types of Code Reviews.

There is always a inspection team that usually consists of four people. One of them plays the role of moderator who should be an expert programmer, but not the author of the program (he does not need to be familiar with the software either).

Moderator duties include:

• Distributing materials for, and scheduling, the inspection session.
• Leading the session.
• Recording all errors found.
• Ensuring that the errors are subsequently corrected.

The rest of the team is the developer of the code, a software architect (could be the architecture of the software) and a Quality Assurance engineer.

The Inspection Agenda is distributed some days in advance of the Inspect Session. Together with the agenda, the moderator distributes the software, specification and any relevant material to the inspection team so they can become familiar with the material before the meeting takes place.

During the review session the moderator ensures that two key activities take place:

1. The programmer describes, statement by statement, the logic of the software. Other participants are free (and encouraged) to raise questions in order to determine whether errors exist. It is likely that the developer himself, instead of the rest of team, is the one that find many of the errors identified during this stage. In other words, the simple act of reading aloud a program to an audience seems to be a remarkably effective error-detection technique.
2. The program is analyzed with respect to checklists of historically common programming errors.

When the session is over, the programmer receives an error list that includes all the errors that have been discovered. Hence, the session is focused on finding defects not fixing them. Despite that, in some occasions, when a problem is discovered, the review team could propose and discuss some design changes. When some of the detected defects require significant changes in the code, the review team could agree to have follow-up meetings in order to review again the code after the changes are implemented.

The list of errors is not only used by the developer in order to fix them; it is also used by moderator to verify if the error checklist could be improved with the results.

The review sessions are typically very dynamic and hence the moderator should be responsible not only for reviewing the code but also to keeping it focused so time is used efficiently (these sessions should be of 90-120 minutes maximum).

This kind of approaches requires of the right attitude, specially from the developer whose work is going to be under scrutiny. He must forget about his ego and think about the process as a way to improve the quality of his work and improve his development skills, as he usually receives a lot of feedback about programming styles, algorithms and techniques. But it is not only the developer but also the rest of the team the ones who could learn by such an open exchange of ideas.

The following diagram describes theis process grafically:

The following tables describes some checklists used in formal code reviews as explained in [[ART-OF-TESTING]].

Walkthrough

A Walkthrough is quite similar to "Formal Code Inspections" as it is also very formal, it is conducted by a team, and it takes place during a pre-scheduled session of 90-120 minutes. However, there is a key difference: the procedure during the meeting. Instead of simply reading the software and use checklists, the participants "play computer", which means that a person that is designated as the tester comes to the meeting with a set of pre-defined test cases for the software. During the meeting, each test case is mentally executed; that is, the test data are "walked through" the logic of the program. The state of the program is monitored on paper or a whiteboard.

The test cases must not be a complete set of test cases, especially because every mental execution of a test case use to take a lot of time. The test cases themselves are not the critical thing; they are just an excuse for questioning the developer about the assumptions and decisions taken.

Although the size of the team is quite similar (three to five), the role of the participants is slightly different. Apart from the author of the software and a moderator, there are two key roles in walkthroughs: a tester role (that is the one responsible for guiding the execution of the test cases) and a secretary that writes down all the errors found. Additionally, other participants are welcome, typically experience programmers.

Over the shoulder Review

The two formal approaches described formerly, are good, and help to detect many defects. Additionally, they provide extra metrics and information about the effectiveness of the reviews themselves. However, this require a lot of effort, and consumes a lot of extra developer time. Many studies during the last yeasr have shown that there are other less formal methods that could achieve similar results but requiring less training and time.

The first one we are going to study is over-the-shoulder reviews. This is the most common and informal of code reviews. An over-the-shoulder review is just that: a developer standing over the developer's computer while the author walks the reviewer through a set of code changes.

Typically the author "drives" the review by sitting at the computer opening various files, pointing out the changes and explaining why it was done that way. Multiple tools can be used by the developer and it's usual to move back and forth between files.

If the reviewer sees something wrong, they can take different actions, such as doing a little of "pair-programming" while the developer implements the fix or just take note of the issue to be solved offline.

With cooperation tools such as videoconferencing, desktop sharing and so on, it is possible to perform this kind of reviews remotely but obviously, they are not so effective as the greatest asset of this technique is the closeness between developers and the easyness to take ad-hoc actions taken the opportunity of being together.

The key advantage of this approach is its simplicity: no special training is required and can be done at any time without any preparation. It also encourages human interaction and encourages people to cooperate. Reviewers tend to be more verbose and brave when speaking than when they need to record their reviews in a system such as DataBase.

Of course it has some drawbacks. The first one is that due to its informal nature, it is really difficult to be enforced, i.e. there is no way (document, tool, etc...) to check if such a review has been conducted. The second one is that, as the author is the one leading the whole process he might omit parts of the code. The third one is the lack of traceability to check that the detected defects have been properly addressed.

The following diagram describes this process grafically

Offline Reviews

This is the second-most common form of informal code review, and the technique preferred by most open-source projects. Here, whole files, or changes are packaged up (ZIP file, URL, Pull Request, etc...) by the author and sent to reviewers via e-mail or any other tool. Reviewers examine the files offline, ask questions and discuss with the author and other developers, and suggest changes.

Collecting the files to be reviewed was formerly a difficul task but nowadays, with Source Code Management systems such as Git, it is extremely easy to identify the files that the developer has modified and hence the changes he wants to merge into the main repository.

But SCM tools have helped not only to identifying the changes made by the developer, but also in other multiple areas such as:

• Sending E-mail notifications: Request for review, review done, comments need to be addressed, etc...
• Recording review comments: Things to be changed, result of the review, whether the changes have been implemented or not, etc. This is key for having a way to enforce reviews.
• Combined display: Allow developers and reviewers to easily check the differences between files allowing different views.
• Discussion: Sometimes it's needed some type of discussion between the developer and the reviewer in order to undertand a bit more the code, clarify reasons behind some decisions, etc.

Obviously, the main advantage with respect to over-the-shoulder reviews is that it can work perfectly with developers that are not based in the same place, either across a building or across an ocean. Additionally, by using this technique is extremely easy to allow multiple reviewers to review the code in parallel, in many cases, if the reviews are done in an SCM system, even anyone with access to the SCM could comment in the review, even if he/she is not a reviewer.

The main disadvantage with an over-the-shoulder review is that it takes longer as it usually requires different interactions, this could be especially painful if people are in different timezones.

In general, we could say that offline code reviews, done properly integrated in an SCM gets a good balance between speed, effectiveness and traceability.

The following diagram describes this process grafically

Pair Programming

Pair Programming it is a development process that incorporates continuous code review in the development process itself. It consists in two developers writing code at a single terminal with only one developer typing at a time and continuous free-form discussion and review.

Studies of pair-programming have shown it to be very effective at both finding bugs and promoting knowledge transfer. However, having the reviewing developer so involved in the development itself is seen by many people as a risk to be biased: it's going to be more difficult for him to go a step back and critique the code from a fresh point of view. However, it could be argued that deep knowledge and understanding also provides him the capabiltiy to provide more effetive comments.

The key difference with the other techniques mentioned above is that introducing this way of working affects not only how QA Activities are performed but also development ones (i.e. you could combine all the other review techniques with different ways of developing code). Adopting this way of working requires evaluating properly how are developers going to work in such an environment and the time required for working in this way.

Code Review Techniques: Summary

Each of the types of review is useful in its own way. Offline reviews strike a balance between time invested and ease of implementation. In any case, and any kind of code review is better than nothing, but it should be also acknowledged that code reviews are not enough to guarantee the quality of a final product.

Deffensive Programming

Deffensive Programing consists in including in the software as many checks as possible, even if they are redundant (e.g. checks made by callers and callees). Sometimes it's said, "maybe they don't help, but they don't harm either".

The problem with this way of working, is that, in some cases, it ends-up adding a lot of redundancy "just in case", which means adding unnecessary complexity and increasing software size. The bigger and more complex a software is, the easier defects can affect it.

The ideas behind deffensive software, are interesting, but in order to make these ideas have a possitive effect, a more systematic approach should be followed.

Contract Concept

A contract, in the real world, is an agreement between two parties in which each party expect some benefits from the contract if they meet some obligations. Both are linked, i.e. if the obligations are not met by any of the parties, there is no guarantee the benefits will happen. Those benefits and obligations are clearly documented so that there are no misunderstanding between the parties.

Imagine a courier company that has a express service within Madrid city. That express service can be only done if the customer meets some conditios (e.g. the package is within the limits, the address is valid and in Madrid, the user pays...). If the customer meets this conditions, he gets the benefit of the package being delivered in 4 hours. If the customer does not meet them, there is no guarantee he can get the express deliver benefits. The following table shows the obligations/benefits of this example:

One important remark, is that when a contract is exhaustive, there is a guarantee that all the obligations are related to the benefits. This is also called the "No hidden clause" rule. This does not mean that the contract could not refer to external laws, best practices, regulation... it only means they do not need to be explicitly stated. For instance, in case the courier fails to meet their obligations, it is highly likely a law establishes a compensation to the customer.

Contracts in Software

It is easy to understand how the concept of contracts in the real world could be extrapolated to software development. In software every task can be split in multiple sub-tasks, the idea of sub-tasks is similar of contracting something to a company. I create a function, module, etc... that handles this part that is essential to meet the complete task.

           task is
           do
             subtask1:
             subtask2:
             subtask3:
           end
          

If all the subtasks are completed correctly, the task will be also finished successfully. If there is a contract between the task and the subtasks, the task will have some guarantees about the completion. Subtasks in software developmentare typically functions, object methods...

Please also think about the Spotify way of working in which they created an architecture that manage every team to deliver different parts of Spotify client independently. It is quite similar, they have divided the main task (the Spotify client) in multiple subtasks (the components of the architecture). If all the components behave properly, the final task will be working properly too.

Design By Contract

Design by Contract (DbC) is based on the definition of formal, precise and verifiable interface specifications for every software component. These specifications extend the ordinary definition of abstract types with preconditions, postconditions and invariants. Those specifications are also known as contracts.

A software contract could be defined as the set of three different things:

• Preconditions: A certain conditions to be guaranteed on entry by any client module that calls it. It is an obligation for the client module and a benefit for the supplier (no need to handle cases outside of the precondition)
• Postconditions: Gurantee a certain property on exit. This is an obligation for the supplier and a benefit for the client.
• Class-invariant: Guarantee that certain properties are not going to be changed on exit.

This could be formalized as three questions developers must try to solve when implementing a function:

• What does contract expect?
• What does contract gurantee?
• What does contract maintain?

Using Design By Contract

The ideal environment for Design by Contract is one in which the language developers use has support for it in a native way. Unfortunately not too many of them support this capability, being Eiffel the most known one. For those languages, the contract is part of the function definition. For instance, see an Eiffel example below:

          class ACCOUNT create
            make
            feature
              ... Attributes as before:
                  balance , minimum_balance , owner , open ...
              deposit (sum: INTEGER) is
                    -- Deposit sum into the account.
                require
                  sum >= 0
                do
                  add (sum)
                ensure
                  balance = old balance + sum
                end
          

In programming languages with no direct support, in most of the cases assertions are used as a way to implement DbC techniques. There are libraries that try to simplify the process of defining these assertions. An assertion is a predicate used to indicate that if the software is a correct status, the predicate should be always true at that place. If an assertion evaluates to false, that should mean that the software is in a wrong status (e.g. the contract has been broken).

Of course, the functions can still do some checkings, but only for conditions that are not part of the contract. The idea of DbC is removing any duplication and minimizing the amount of code necessary to check that the contract is met.

Fault Tolerance and Failure Containment

There is no practical way to guarantee that a given software has no bugs. It doesn't matter how good are our tools, methodologies and engineers... It doesn't matter how deep inspections and testings are either. In many cases the presence of bugs is tolerated as something "natural", but there are some systems were the reliability and security requirements are so important that extra measures should be taken to mitigate the consequences of undetected bugs.

When a system has extreme reliability requirements, fault tolerant solutions should be put in place. The idea behind fault tolerant solutions consists in breaking the bug/failure cause/effect relationship. A result of this is a increase of the reliability as reliability is inversely proportional to the frequency of failures. These techniques are usually expensive as they typically require redundancy of sort so they are only in systems that require it.

For instance, the software used in the flight control systems is one example of software with very extreme requirements about failures. The report [[CHALLENGES-FAULT-TOLERANT-SYSTEMS]] provides more details about the challenges that this kind of systems pose to software developers.

However, in some situations, failures cannot be prevented and hence reliability cannot be improved. However, there are ways to minimize the consequences of failures with the target of maximizing safety. It is important we don't confuse reliability with safety: for instance a medical system could not be 100% reliable but it should be 100% safe. The techniques intended to increase system safety are called failure containment techniques.

In this chapter we are going to study both type of techniques.

Fault Tolerance

Fault Tolerance techniques are used to tolerate software faults and prevent system failures from occurring when a fault occurs. These kind of techniques are used in software that is very sensitive to failures such as aerospacial software, nuclear power, healthcare...

Single Version Software Environments (No Redundancy)

In this case, only one instance of the software exists, and it tries to detect faults and recover from them without the need to replicate the software. This kind of techniques are really difficult to be implemented, as it has been studied that efficient fault tolerant systems require some kind of redundancy as we will see in next section.

Multiple Version Software Environments (Redundancy)

Redundancy in real world activities is the best way to increase reliability: multiple engines in a plane, lights in a car... For instance, the NASA performed a research to calculate the possibility of survival in a mission depending of the amount of redundant equipment in the spacecraft and the results demonstrated that the survival chances are extremely dependent on redundancy as show in the figure below.

The same is true for software systems but with some caveats. Redundancy in software only works if the redundant system works when the original one fails: this is normal in hardware systems but not in software. If I have two identical software systems and the first one fails, it is extremely likely the second one fails too. Due to this is important that redundant software systems are uncorrelated. Designing uncorrelated systems usually requires two teams, working isolated, with different techniques... This means duplicating at least the development cost.

In these systems, those multiple instances of the software developed independently can work in different configurations: N-Version programming (NVP), Recovery Blocks (RcB), N self checking programming (NSCP)...

Software Fault Tolerance

1. Recovery blocks: Use repeated executions (or redundancy over time) as the basic mechanism for fault tolerance. The software includes a set of "recovery points" in which the status is recorded so that they could be used as fallbacks in case something goes wrong. When a piece of code is executed, a "test acceptance" is internally executed, if the result is OK, a new "recovery point" is set-up, if the result is not acceptable, then the software returns to the previous "recovery point" and an alternative to the faulty code is enacted. This process continues until the "acceptance test" is passed or no more alternatives are available, which leads to a failure. Some key characteristics about this scheme that is depicted in :
   • It is a backward error recovery technique: when an error occurs, appropriate actions are taken to react but no preventing action is taken.
   • It is a "serial technique" in which the same functionality (the recovery block) is never executed in parallel.
   • The "acceptance test" algorithm is the critical part to success as well as the availability of recovery blocks designed in different ways to the original code.

   

2. NVP (N-version programming):

   This technique uses parallel redundancy, where N copies, each of a different version, of codes fulfilling the same functionality are running in parallel with the same inputs. When all of those N-copies have completed the operation, an adjudication process (decision unit) takes place to determine (based in a more or less complex vote) the output.

   Some key characteristics about this scheme that is depicted in

   • It is a forward error recovery technique: preventive actions are taken. Even if no error occurs the same functionality is executed multiple times.
   • It is a "parallel technique" in which the same functionality is always executed in parallel by different versions of the same functionality.
   • The "decision unit" algorithm is the critical part to success as well as the availability of different versions of the same code designed in different ways.

   

   Obviously a wide range of different variants of those systems have been proposed based in multiple combinations of them [[COST-EFFECTIVE-FAULT-TOLERANCE]] and multiple comparisons between the performance are also available [[PERFORMANCE-RB-NVP-SCOP]].

What do you think are the key advantages and disadvantages of the two fault tolerance techniques described (Recovery Blocks & N-Version)? Exercise 3: Recovery Blocks vs. N-Version

Types of recovery

Error Recovery is the key part in fault tolerant systems. However, although the most important one, it's the last step in a series of 4 parts:

1. Error Detection: This consists in the ability to detect that the Software is an erronous state, for instance, via asserions as it has been studied in this unit.
2. Error Diagnosis: Assess the causes for the error situation in which the Software has fallen.
3. Error Containment: Before trying to correct the error, we should stop the error propagation, so further damages will not happen.
4. Error Recovery: Consists in replacing the erroneous state with an error-free state.

Depending on how the new error-free state is calculated we could distinguish two approaches:

• Backward Recovery: In this mechanisms the system tries to go back to a previously saved state that the system knows it is correct. Recoding or saving these states is called checkpointing. Some systems instead of recording the states completely keep deltas with respect to previous states, which is know as differential checkpointing. Recovery blocks is one example of this type of technique.
• Forward Recovery: In this approach, the system tries to find a new state from which the system can continue the operation. This state can be calculated by Error Compensation. Error Compensation relies on redundancy based algorithms in which the redundancy system provides a set of potential results from which a compensation is executed to derive an answer deemed as acceptable. An example of this technique is N-Version programming. This techniques are more complex and requires more resources so they are mostly used when the system is critical with respect to delays (i.e. there is no time for a backward recovery).