4. When should Testing stop?
"When to stop testing" is one of the most difficult questions to a test engineer.
The following are few of the common Test Stop criteria:
1. All the high priority bugs are fixed.
2. The rate at which bugs are found is too small.
3. The testing budget is exhausted.
4. The project duration is completed.
5. The risk in the project is under acceptable limit.
5. Verification Strategies
5.1.What is ‘Verification’?
"Am I building the product right?" Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase.
What is the importance of the Verification Phase?
Verification process helps in detecting defects early, and preventing their leakage downstream. Thus, the higher cost of later detection and rework is eliminated.
5.1.1 Review
A process or meeting during which a work product, or set of work products, is presented to project personnel, managers, users, customers, or other interested parties for comment or approval.
The main goal of reviews is to find defects. Reviews are a good compliment to testing to help assure quality. A few purposes’ of SQA reviews can be as follows:
• Assure the quality of deliverable’s before the project moves to the next stage.
• Once a deliverable has been reviewed, revised as required, and approved, it can be used as a basis for the next stage in the life cycle.
5.1.2.What are the various types of reviews?
Types of reviews include Management Reviews, Technical Reviews, Inspections, Walkthroughs and Audits.
Management Reviews
Management reviews are performed by those directly responsible for the system in order to monitor progress, determine status of plans and schedules, confirm requirements and their system allocation.
Therefore the main objectives of Management Reviews can be categorized as follows:
• Validate from a management perspective that the project is making progress according to the project plan.
• Ensure that deliverables are ready for management approvals.
• Resolve issues that require management’s attention.
• Identify any project bottlenecks.
• Keeping project in Control.
The participants of the review play the roles of Decision-Maker, Review Leader, Recorder, Management Staff, and Technical Staff.
Technical Reviews
Technical reviews confirm that product Conforms to specifications, adheres to regulations, standards, guidelines, plans, changes are properly implemented, changes affect only those system areas identified by the change specification.
The main objectives of Technical Reviews can be categorized as follows:
• Ensure that the software confirms to the organization standards.
• Ensure that any changes in the development procedures (design, coding, testing) are implemented per the organization pre-defined standards.
In technical reviews, the following Software products are reviewed
• Software requirements specification
• Software design description
• Software test documentation
• Software user documentation
• Installation procedure
• Release notes
The participants of the review play the roles of Decision-maker, Review leader, Recorder, Technical staff.
What is Requirement Review?
A process or meeting during which the requirements for a system, hardware item, or software item are presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include system requirements review, software requirements review.
• Product management leads Requirement Review. Members from every affected department participates in the review
What is Design Review?
A process or meeting during which a system, hardware, or software design is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Types include critical design review, preliminary design review, and system design review.
• QA team member leads design review. Members from development team and QA team participate in the review.
What is Code Review?
A meeting at which software code is presented to project personnel, managers, users, customers, or other interested parties for comment or approval.
• QA team member (In case the QA Team is only involved in Black Box Testing, then the Development team lead chairs the review team) leads code review. Members from development team and QA team participate in the review.
5.2 Walkthrough
A static analysis technique in which a designer or programmer leads members of the development team and other interested parties through a segment of documentation or code, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems.
The objectives of Walkthrough can be summarized as follows:
• Detect errors early.
• Ensure (re)established standards are followed:
• Train and exchange technical information among project teams which participate in the walkthrough.
• Increase the quality of the project, thereby improving morale of the team members.
The participants in Walkthroughs assume one or more of the following roles:
a) Walk-through leader
b) Recorder
c) Author
d) Team member
Input to the walk-through shall include the following:
a) A statement of objectives for the walk-through
b) The software product being examined
c) Standards that are in effect for the acquisition, supply, development, operation, and/or maintenance of the software product
Input to the walk-through may also include the following:
5.3 Inspection
A static analysis technique that relies on visual examination of development products to detect errors, violations of development standards, and other problems. Types include code inspection; design inspection, Architectural inspections, Test ware inspections etc.
The participants in Inspections assume one or more of the following roles:
a) Inspection leader
b) Recorder
c) Reader
d) Author
e) Inspector
All participants in the review are inspectors. The author shall not act as inspection leader and should not act as reader or recorder. Other roles may be shared among the team members. Individual participants may act in more than one role.
Input to the inspection shall include the following:
a) A statement of objectives for the inspection
b) The software product to be inspected
c) Documented inspection procedure
d) Inspection reporting forms
e) Current anomalies or issues list
Input to the inspection may also include the following:
f) Inspection checklists
g) Any regulations, standards, guidelines, plans, and procedures against which the software product is to be inspected
6.Testing Types and Techniques
Testing types
Testing types refer to different approaches towards testing a computer program, system or product. The two types of testing are black box testing and white box testing, which would both be discussed in detail in this chapter. Another type, termed as gray box testing or hybrid testing is evolving presently and it combines the features of the two types.
Testing Techniques
Testing techniques refer to different methods of testing particular features a computer program, system or product. Each testing type has its own testing techniques while some techniques combine the feature of both types. Some techniques are
• Error and anomaly detection technique
• Interface checking
• Physical units checking
• Loop testing ( Discussed in detail in this chapter)
• Basis Path testing/McCabe’s cyclomatic number( Discussed in detail in this chapter)
• Control structure testing( Discussed in detail in this chapter)
• Error Guessing( Discussed in detail in this chapter)
• Boundary Value analysis ( Discussed in detail in this chapter)
• Graph based testing( Discussed in detail in this chapter)
• Equivalence partitioning( Discussed in detail in this chapter)
• Instrumentation based testing
• Random testing
• Domain testing
Difference between Testing Types and Testing Techniques
Testing types deal with what aspect of the computer software would be tested, while testing techniques deal with how a specific part of the software would be tested.
That is, testing types mean whether we are testing the function or the structure of the software. In other words, we may test each function of the software to see if it is operational or we may test the internal components of the software to check if its internal workings are according to specification.
On the other hand, ‘Testing technique’ means what methods or ways would be applied or calculations would be done to test a particular feature of a software (Sometimes we test the interfaces, sometimes we test the segments, sometimes loops etc.)
How to Choose a Black Box or White Box Test?
White box testing is concerned only with testing the software product; it cannot guarantee that the complete specification has been implemented. Black box testing is concerned only with testing the specification; it cannot guarantee that all parts of the implementation have been tested. Thus black box testing is testing against the specification and will discover faults of omission, indicating that part of the specification has not been fulfilled. White box testing is testing against the implementation and will discover faults of commission, indicating that part of the implementation is faulty. In order to completely test a software product both black and white box testing are required.
White box testing is much more expensive (In terms of resources and time) than black box testing. It requires the source code to be produced before the tests can be planned and is much more laborious in the determination of suitable input data and the determination if the software is or is not correct. It is advised to start test planning with a black box testing approach as soon as the specification is available. White box tests are to be planned as soon as the Low Level Design (LLD) is complete. The Low Level Design will address all the algorithms and coding style. The paths should then be checked against the black box test plan and any additional required test cases should be determined and applied.
7.1 White Box Testing
What is WBT?
White box testing involves looking at the structure of the code. When you know the internal structure of a product, tests can be conducted to ensure that the internal operations performed according to the specification. And all internal components have been adequately exercised. In other word WBT tends to involve the coverage of the specification in the code.
Code coverage is defined in six types as listed below.
• Segment coverage – Each segment of code b/w control structure is executed at least once.
• Branch Coverage or Node Testing – Each branch in the code is taken in each possible direction at least once.
• Compound Condition Coverage – When there are multiple conditions, you must test not only each direction but also each possible combinations of conditions, which is usually done by using a ‘Truth Table’
• Basis Path Testing – Each independent path through the code is taken in a pre-determined order. This point will further be discussed in other section.
• Data Flow Testing (DFT) – In this approach you track the specific variables through each possible calculation, thus defining the set of intermediate paths through the code i.e., those based on each piece of code chosen to be tracked. Even though the paths are considered independent, dependencies across multiple paths are not really tested for by this approach. DFT tends to reflect dependencies but it is mainly through sequences of data manipulation. This approach tends to uncover bugs like variables used but not initialize, or declared but not used, and so on.
• Path Testing – Path testing is where all possible paths through the code are defined and covered. This testing is extremely laborious and time consuming.
• Loop Testing – In addition top above measures, there are testing strategies based on loop testing. These strategies relate to testing single loops, concatenated loops, and nested loops. Loops are fairly simple to test unless dependencies exist among the loop or b/w a loop and the code it contains.
What do we do in WBT?
In WBT, we use the control structure of the procedural design to derive test cases. Using WBT methods a tester can derive the test cases that
• Guarantee that all independent paths within a module have been exercised at least once.
• Exercise all logical decisions on their true and false values.
• Execute all loops at their boundaries and within their operational bounds
• Exercise internal data structures to ensure their validity.
White box testing (WBT) is also called Structural or Glass box testing.
Limitations
1.Unfortunately in WBT, exhaustive testing of a code presents certain logistical problems. Even for small programs, the number of possible logical paths can be very large.
2. Exhaustive WBT is impossible for large software systems. But that doesn’t mean WBT should be considered as impractical. Limited WBT in which a limited no. of important logical paths are selected and exercised and important data structures are probed for validity, is both practical and WBT.
It is suggested that white and black box testing techniques can be coupled to provide an approach that that validates the software interface selectively ensuring the correction of internal working of the software.
7.1.1 Basis Path Testing
The Basis path method enables to derive a logical complexity measure of a procedural design and use this measure as a guide for defining a basis set of execution paths. Test Cases derived to exercise the basis set are guaranteed to execute every statement in the program at least one time during testing.
7.1.2 Flow Graph Notation
The flow graph depicts logical control flow using a diagrammatic notation. Each structured construct has a corresponding flow graph symbol.
7.1.3 Cyclomatic Complexity
Cyclomatic complexity is a software metric that provides a quantitative measure of the logical complexity of a program. When used in the context of a basis path testing method, the value computed for Cyclomatic complexity defines the number for independent paths in the basis set of a program and provides us an upper bound for the number of tests that must be conducted to ensure that all statements have been executed at least once.
An independent path is any path through the program that introduces at least one new set of processing statements or a new condition.
Computing Cyclomatic Complexity
Cyclomatic complexity has a foundation in graph theory and provides us with extremely useful software metric. Complexity is computed in one of the three ways:
1. The number of regions of the flow graph corresponds to the Cyclomatic complexity.
2. Cyclomatic complexity, V(G), for a flow graph, G is defined as
V (G) = E-N+2
Where E, is the number of flow graph edges, N is the number of flow graph nodes.
3. Cyclomatic complexity, V (G) for a flow graph, G is also defined as:
V (G) = P+1
Where P is the number of predicate nodes contained in the flow graph G.
7.1.4 Graph Matrices
The procedure for deriving the flow graph and even determining a set of basis paths is amenable to mechanization. To develop a software tool that assists in basis path testing, a data structure, called a graph matrix can be quite useful.
A Graph Matrix is a square matrix whose size is equal to the number of nodes on the flow graph. Each row and column corresponds to an identified node, and matrix entries correspond to connections between nodes.
7.1.5 Control Structure Testing
Described below are some of the variations of Control Structure Testing.
Condition Testing
Condition testing is a test case design method that exercises the logical conditions contained in a program module.
Data Flow Testing
The data flow testing method selects test paths of a program according to the locations of definitions and uses of variables in the program.
7.1.6 Loop Testing
Loop Testing is a white box testing technique that focuses exclusively on the validity of loop constructs. Four classes of loops can be defined: Simple loops, Concatenated loops, nested loops, and unstructured loops.
Simple Loops
The following sets of tests can be applied to simple loops, where ‘n’ is the maximum number of allowable passes through the loop.
1. Skip the loop entirely.
2. Only one pass through the loop.
3. Two passes through the loop.
4. ‘m’ passes through the loop where m
5. n-1, n, n+1 passes through the loop.
Nested Loops
If we extend the test approach from simple loops to nested loops, the number of possible tests would grow geometrically as the level of nesting increases.
1. Start at the innermost loop. Set all other loops to minimum values.
2. Conduct simple loop tests for the innermost loop while holding the outer loops at their minimum iteration parameter values. Add other tests for out-of-range or exclude values.
3. Work outward, conducting tests for the next loop, but keep all other outer loops at minimum values and other nested loops to “typical” values.
4. Continue until all loops have been tested.
Concatenated Loops
Concatenated loops can be tested using the approach defined for simple loops, if each of the loops is independent of the other. However, if two loops are concatenated and the loop counter for loop 1 is used as the initial value for loop 2, then the loops are not independent.
Unstructured Loops
Whenever possible, this class of loops should be redesigned to reflect the use of the structured programming constructs.
8. Black Box Testing
Black box is a test design method. Black box testing treats the system as a "black-box", so it doesn't explicitly use Knowledge of the internal structure. Or in other words the Test engineer need not know the internal working of the “Black box”.
It focuses on the functionality part of the module.
Some people like to call black box testing as behavioral, functional, opaque-box, and closed-box.
Advantages of Black Box Testing
- Tester can be non-technical.
- This testing is most likely to find those bugs as the user would find.
- Testing helps to identify the vagueness and contradiction in functional specifications.
- Test cases can be designed as soon as the functional specifications are complete
Disadvantages of Black Box Testing
- Chances of having repetition of tests that are already done by programmer.
- The test inputs needs to be from large sample space.
- It is difficult to identify all possible inputs in limited testing time. So writing test cases is slow and difficult
Chances of having unidentified paths during this testing
8.2.1 Graph Based Testing Methods
Software testing begins by creating a graph of important objects and their relationships and then devising a series of tests that will cover the graph so that each objects and their relationships and then devising a series of tests that will cover the graph so that each object and relationship is exercised and error is uncovered.
8.2.2 Error Guessing
Error Guessing comes with experience with the technology and the project. Error Guessing is the art of guessing where errors can be hidden. There are no specific tools and techniques for this, but you can write test cases depending on the situation: Either when reading the functional documents or when you are testing and find an error that you have not documented.
8.2.3 Boundary Value Analysis
Boundary Value Analysis (BVA) is a test data selection technique (Functional Testing technique) where the extreme values are chosen. Boundary values include maximum, minimum, just inside/outside boundaries, typical values, and error values. The hope is that, if a system works correctly for these special values then it will work correctly for all values in between.
Extends equivalence partitioning
Test both sides of each boundary
Look at output boundaries for test cases too Test min, min-1, max, max+1, typical values
BVA focuses on the boundary of the input space to identify test cases
Rational is that errors tend to occur near the extreme values of an input variable
There are two ways to generalize the BVA techniques:
1. By the number of variables
o For n variables: BVA yields 4n + 1 test cases.
2. By the kinds of ranges
o Generalizing ranges depends on the nature or type of variables
NextDate has a variable Month and the range could be defined as {Jan, Feb, …Dec}
Min = Jan, Min +1 = Feb, etc.
Triangle had a declared range of {1, 20,000}
Boolean variables have extreme values True and False but there is no clear choice for the remaining three values
Advantages of Boundary Value Analysis
1.Robustness Testing - Boundary Value Analysis plus values that go beyond the limits
2.Min - 1, Min, Min +1, Nom, Max -1, Max, Max +1
3.Forces attention to exception handling
4.For strongly typed languages robust testing results in run-time errors that abort normal execution
Limitations of Boundary Value Analysis
BVA works best when the program is a function of several independent variables that represent bounded physical quantities
1. Independent Variables
o NextDate test cases derived from BVA would be inadequate: focusing on the boundary would not leave emphasis on February or leap years
o Dependencies exist with NextDate's Day, Month and Year
o Test cases derived without consideration of the function
2. Physical Quantities
o An example of physical variables being tested, telephone numbers - what faults might be revealed by numbers of 000-0000, 000-0001, 555-5555, 999-9998, 999-9999?
8.2.4 Equivalence Partitioning
Equivalence partitioning is a black box testing method that divides the input domain of a program into classes of data from which test cases can be derived.
EP can be defined according to the following guidelines:
1. If an input condition specifies a range, one valid and one two invalid classes are defined.
2. If an input condition requires a specific value, one valid and two invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, one valid and one invalid equivalence class is defined.
4. If an input condition is Boolean, one valid and one invalid class is defined.
8.2.5 Comparison Testing
There are situations where independent versions of software be developed for critical applications, even when only a single version will be used in the delivered computer based system. It is these independent versions which form the basis of a black box testing technique called Comparison testing or back-to-back testing.
8.2.6 Orthogonal Array Testing
The Orthogonal Array Testing Strategy (OATS) is a systematic, statistical way of testing pair-wise interactions by deriving a suitable small set of test cases (from a large number of possibilities).
Posts
