Black Box Testing
Diagram.
Black-Box Testing is a method of testing software that tests
the functionality of an application as opposed to its internal structures or
workings .Specific knowledge of the application's code/internal structure and
programming knowledge in general is not required. Test cases are built around
specifications and requirements, i.e., what the application is supposed to do.
It uses external descriptions of the software, including specifications,
requirements, and design to derive test cases. These tests can be functional or
non-functional, though usually functional. The test designer selects valid and
invalid inputs and determines the correct output. There is no knowledge of the
test object's internal structure.
This method of test can
be applied to all levels of software testing: unit, integration, functional, system and acceptance. It typically comprises most if not
all testing at higher levels, but can also dominate unit testing as well.
Test Design Techniques
Typical black-box test
design techniques include:
- Decision table testing
- All-pairs testing
- State transition
tables
- Equivalence
partitioning
- Boundary value
analysis
Hacking
In penetration testing, black-box testing refers to
a methodology where an ethical hacker has no knowledge of the
system being attacked. The goal of a black-box penetration test is to simulate
an external hacking or cyber warfare attack.
Decision Table
Decision tables are a precise yet compact way to model
complicated logic
Decision tables,
like flowcharts and if-then-else and switch-case statements, associate conditions
with actions to perform, but in many cases do so in a more elegant way.
In the 1960s and 1970s a
range of "decision table based" languages such as Filetab were popular for business
programming.
Structure
A decision table is
typically divided into four quadrants, as shown below.
The
four quadrants
|
|
Conditions
|
Condition alternatives
|
Actions
|
Action entries
|
Each decision
corresponds to a variable, relation or predicate whose possible values are
listed among the condition alternatives. Each action is a procedure or
operation to perform, and the entries specify whether (or in what order) the
action is to be performed for the set of condition alternatives the entry
corresponds to. Many decision tables include in their condition alternatives
the don't care symbol,
a hyphen. Using don't cares can simplify decision tables, especially when a
given condition has little influence on the actions to be performed. In some
cases, entire conditions thought to be important initially are found to be
irrelevant when none of the conditions influence which actions are performed.
Aside from the basic
four quadrant structure, decision tables vary widely in the way the condition
alternatives and action entries are represented. Some decision tables use simple
true/false values to represent the alternatives to a condition (akin to
if-then-else), other tables may use numbered alternatives (akin to
switch-case), and some tables even use fuzzy logic or probabilistic representations
for condition alternatives. In a similar way, action entries can
simply represent whether an action is to be performed (check the actions to
perform), or in more advanced decision tables, the sequencing of actions to
perform (number the actions to perform).
Decision table can be
used if the combination of conditions are given. In decision table conditions
are known as causes and serinal numbers of conditions are known as business
rule.
Example
The limited-entry
decision table is the simplest to describe. The condition alternatives are
simple boolean values, and the action entries are check-marks, representing
which of the actions in a given column are to be performed.
A technical support
company writes a decision table to diagnose printer problems based upon
symptoms described to them over the phone from their clients.
The following is a balanced
decision table.
Printer
Troubleshooter
|
|||||||||
Rules
|
|||||||||
Conditions
|
Printer does not print
|
Y
|
Y
|
Y
|
Y
|
N
|
N
|
N
|
N
|
A red light is flashing
|
Y
|
Y
|
N
|
N
|
Y
|
Y
|
N
|
N
|
|
Printer is unrecognized
|
Y
|
N
|
Y
|
N
|
Y
|
N
|
Y
|
N
|
|
Actions
|
Check the power cable
|
X
|
|||||||
Check the printer-computer cable
|
X
|
X
|
|||||||
Ensure printer software is
installed
|
X
|
X
|
X
|
X
|
|||||
Check/replace ink
|
X
|
X
|
X
|
X
|
|||||
Check for paper jam
|
X
|
X
|
|||||||
Of course, this is just
a simple example (and it does not necessarily correspond to the reality of
printer troubleshooting), but even so, it demonstrates how decision tables can
scale to several conditions with many possibilities.
Software Engineering
Benefits
Decision Tables,
especially when coupled with the use of a domain-specific
language, allow developers and policy experts to work from the same
artifacts, the Decision Tables themselves.
Tools to render
traditional into decision tables can also be used as a debugging tool
Decision Tables have
proven to be easier to understand and review than code, and have been used extensively
and successfully to produce specifications for complex systems.
Program Embedded Decision
Tables
Decision tables can be,
and often are, embedded within computer programs and used to 'drive' the logic
of the program. A simple example might be a lookup table containing a range of possible
input values and a function pointer to
the section of code to process that input.
Static
Decision Table
|
|
Input
|
Function Pointer
|
'1'
|
Function 1 (initialize)
|
'2'
|
Function 2 (process 2)
|
'9'
|
Function 9 (terminate)
|
Multiple conditions can
be coded for in similar manner to encapsulate the entire program logic in the
form of an 'executable' decision table or control table.
All-Pairs Testing
All-pairs testing or pairwise testing is
a combinatorial software testing method that, for each pair
of input parameters to a system (typically, a software algorithm), tests all possible discrete
combinations of those parameters. Using carefully chosen test
vectors, this can be done much faster than an exhaustive search of
all combinations of all parameters, by "parallelizing" the tests of
parameter pairs. The number of tests is typically O(nm), where n and m are
the number of possibilities for each of the two parameters with the most
choices.
The reasoning behind
all-pairs testing is this: the simplest bugs in a program are generally
triggered by a single input parameter. The next simplest category of bugs
consists of those dependent on interactions between pairs of parameters, which
can be caught with all-pairs testing.[1] Bugs involving interactions
between three or more parameters are progressively less common[2], whilst at the same time being
progressively more expensive to find by exhaustive testing, which has as its
limit the exhaustive testing of all possible inputs.
Many testing methods
regard all-pairs testing of a system or subsystem as a reasonable cost-benefit
compromise between often computationally infeasible higher-order combinatorial
testing methods, and less exhaustive methods which fail to exercise all
possible pairs of parameters. Because no testing technique can find all bugs,
all-pairs testing is typically used together with other quality assurance techniques such as unit testing, symbolic execution, fuzz testing, and code review.
State Transition Table
In automata theory and sequential logic, a state transition
table is a table showing what state (or states in the case of a nondeterministic
finite automaton) a finite semiautomaton or finite state machine will
move to, based on the current state and other inputs. A state table is
essentially a truth table in
which some of the inputs are the current state, and the outputs include the
next state, along with other outputs.
A state table is one of
many ways to specify a state machine, other ways being a state diagram, and a characteristic
equation.
Common forms
One-Dimensional State Tables
Also called characteristic
tables, single-dimension state tables are much more like truth tables than
the two-dimensional versions. Inputs are usually placed on the left, and
separated from the outputs, which are on the right. The outputs will represent
the next state of the machine. Here's a simple example of a state machine with
two states, and two combinatorial inputs:
A
|
B
|
Current
State
|
Next
State
|
Output
|
0
|
0
|
S1
|
S2
|
1
|
0
|
0
|
S2
|
S1
|
0
|
0
|
1
|
S1
|
S2
|
0
|
0
|
1
|
S2
|
S2
|
1
|
1
|
0
|
S1
|
S1
|
1
|
1
|
0
|
S2
|
S1
|
1
|
1
|
1
|
S1
|
S1
|
1
|
1
|
1
|
S2
|
S2
|
0
|
S1 and S2 would
most likely represent the single bits 0 and 1, since a single bit can only have
two states.
Two-Dimensional State Tables
State transition tables
are typically two-dimensional tables. There are two common forms for arranging
them.
- The vertical (or horizontal) dimension indicates current
states, the horizontal (or vertical) dimension indicates events, and the
cells (row/column intersections) in the table contain the next state if an
event happens (and possibly the action linked to this state transition).
State
Transition Table
|
||||
Events
State |
E1
|
E2
|
...
|
En
|
S1
|
-
|
Ay/Sj
|
...
|
-
|
S2
|
-
|
-
|
...
|
Ax/Si
|
...
|
...
|
...
|
...
|
...
|
Sm
|
Az/Sk
|
-
|
...
|
-
|
(S: state, E: event, A:
action, -: illegal transition)
- The vertical (or horizontal) dimension indicates
current states, the horizontal (or vertical) dimension indicates next
states, and the row/column intersections contain the event which will lead
to a particular next state.
State
Transition Table
|
||||
next
current |
S1
|
S2
|
...
|
Sm
|
S1
|
Ay/Ej
|
-
|
...
|
-
|
S2
|
-
|
-
|
...
|
Ax/Ei
|
...
|
...
|
...
|
...
|
...
|
Sm
|
-
|
Az/Ek
|
...
|
-
|
(S: state, E: event, A:
action, -: impossible transition)
Example
An example of a state
transition table for a machine M together with the
corresponding state diagram is given below.
|
State
Diagram
|
|||||||||||||
All the possible inputs
to the machine are enumerated across the columns of the table. All the possible
states are enumerated across the rows. From the state transition table given
above, it is easy to see that if the machine is in S1 (the
first row), and the next input is character 1, the machine will
stay in S1. If a character 0 arrives,
the machine will transition to S2 as can be seen
from the second column. In the diagram this is denoted by the arrow from S1 to S2 labeled
with a 0.
For a nondeterministic
finite automaton (NFA), a new input may cause the machine to be
in more than one state, hence its non-determinism. This is denoted in a state
transition table by a pair of curly braces { } with the set of
all target states between them. An example is given below.
State
Transition Table for an NFA
|
|||
Input
State |
1
|
0
|
Ε
|
S1
|
S1
|
{
S2, S3 }
|
Φ
|
S2
|
S2
|
S1
|
Φ
|
S3
|
S2
|
S1
|
S1
|
Here, a nondeterministic
machine in the state S1 reading an input of 0 will
cause it to be in two states at the same time, the states S2 and S3.
The last column defines the legal transition of states of the special
character, ε. This special character allows the NFA to move to a different
state when given no input. In state S3, the NFA may move
to S1 without consuming an input character. The two
cases above make the finite automaton described non-deterministic.
Transformations from/to State Diagram
It is possible to draw
a state diagram from
the table. A sequence of easy to follow steps is given below:
1. Draw the circles
to represent the states given.
2. For each of the
states, scan across the corresponding row and draw an arrow to the destination
state(s). There can be multiple arrows for an input character if the automaton
is an NFA.
3. Designate a
state as the start state. The start
state is given in the formal definition of the automaton.
4. Designate one or
more states as accept state. This is
also given in the formal definition.
Equivalence Partitioning
Equivalence partitioning is a software testing technique that divides the
input data of a software unit into partitions of data from which test cases can
be derived. In principle, test cases are designed to cover each partition at
least once. This technique tries to define test cases that uncover classes of
errors, thereby reducing the total number of test cases that must be developed.
In rare cases
equivalence partitioning is also applied to outputs of a software component,
typically it is applied to the inputs of a tested component. The equivalence
partitions are usually derived from the requirements specification for input
attributes that influence the processing of the test object. An input has
certain ranges which are valid and other ranges which are invalid. Invalid data
here does not mean that the data is incorrect, it means that this data lies
outside of specific partition. This may be best explained by the example of a
function which takes a parameter "month". The valid range for the
month is 1 to 12, representing January to December. This valid range is called
a partition. In this example there are two further partitions of invalid
ranges. The first invalid partition would be <= 0 and the second invalid
partition would be >= 13.
...
-2 -1 0 1 .............. 12 13 14 15 .....
--------------|-------------------|---------------------
invalid partition
1 valid partition invalid
partition 2
The testing theory
related to equivalence partitioning says that only one test case of each partition
is needed to evaluate the behavior of the program for the related partition. In
other words it is sufficient to select one test case out of each partition to
check the behavior of the program. To use more or even all test cases of a
partition will not find new faults in the program. The values within one
partition are considered to be "equivalent". Thus the number of test
cases can be reduced considerably.
An additional effect of
applying this technique is that you also find the so called "dirty"
test cases. An inexperienced tester may be tempted to use as test cases the
input data 1 to 12 for the month and forget to select some out of the invalid
partitions. This would lead to a huge number of unnecessary test cases on the
one hand, and a lack of test cases for the dirty ranges on the other hand.
The tendency is to
relate equivalence partitioning to so called black box testing which is strictly
checking a software component at its interface, without consideration of
internal structures of the software. But having a closer look at the subject
there are cases where it applies to grey box testing as well. Imagine an
interface to a component which has a valid range between 1 and 12 like the
example above. However internally the function may have a differentiation of
values between 1 and 6 and the values between 7 and 12. Depending upon the input
value the software internally will run through different paths to perform
slightly different actions. Regarding the input and output interfaces to the
component this difference will not be noticed, however in your grey-box testing
you would like to make sure that both paths are examined. To achieve this it is
necessary to introduce additional equivalence partitions which would not be
needed for black-box testing. For this example this would be:
...
-2 -1 0 1 ..... 6 7 ..... 12 13 14 15 .....
--------------|---------|----------|---------------------
invalid partition
1 P1 P2 invalid
partition 2
valid
partitions
To check for the
expected results you would need to evaluate some internal intermediate values
rather than the output interface. It is not necessary that we should use
multiple values from each partition. In the above scenario we can take -2 from
invalid partition 1, 6 from valid partition, and 15 from invalid partition 2.
Equivalence partitioning
is not a stand alone method to determine test cases. It has to be supplemented
by boundary value
analysis. Having determined the partitions of possible inputs the
method of boundary value analysis has to be applied to select the most
effective test cases out of these partitions.
Boundary-Value Analysis
Boundary value analysis is a software testing technique in which tests
are designed to include representatives of boundary values. Values on the edge
of an equivalence
partition or at the smallest value on either side of an edge. The values
could be either input or output ranges of a software component. Since these
boundaries are common locations for errors that result in software faults they
are frequently exercised in test cases.
Applying Boundary
Value Analysis
The expected input and
output values should be extracted from the component specification. The input
and output values to the software component are then grouped into sets with
identifiable boundaries. Each set, or partition, contains values that are
expected to be processed by the component in the same way. Partitioning of test
data ranges is explained in the equivalence
partitioning test case design technique. It is important to
consider both valid and invalid partitions when designing test cases.
For an example, if the
input values were months of the year expressed as integers, the input parameter
'month' might have the following partitions:
...
-2 -1 0 1 .............. 12 13 14 15 .....
---------------|-----------------|---------------------
invalid partition
1 valid partition invalid
partition 2
The boundaries are the
values on and around the beginning and end of a partition. If possible test
cases should be created to generate inputs or outputs that will fall on and to
either side of each boundary. This would result in three cases per boundary.
The test cases on each side of a boundary should be in the smallest increment
possible for the component under test. In the example above there are boundary
values at 0,1,2 and 11,12,13. If the input values were defined as decimal data
type with 2 decimal places then the smallest increment would be the 0.01.
Where a boundary value falls within the invalid
partition the test case is designed to ensure the software component handles
the value in a
controlled manner. Boundary value analysis
can be used throughout the testing cycle and is equally applicable at all
testing phases.
After determining the
necessary test cases with equivalence partitioning and subsequent
boundary value analysis, it
is necessary to define the combination of the test cases when there are
multiple inputs to a software component
No comments:
Post a Comment