Software Verification and Validation

Software testing is used in association with verification and validation:

· Verification: Have we built the software right? (i.e., does it match the specification).

· Validation: Have we built the right software? (i.e., is this what the customer wants).

The terms verification and validation are commonly used interchangeably in the industry; it is also common to see these two terms incorrectly defined. According to the IEEE Standard Glossary of Software Engineering Terminology:

Verification is the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase.

Use – To identify defects in the product early in the life cycle

 

Validation is the process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.

 

Verification & Method of Verification

Verification

Verification is a quality assurance process intended to check that a product, service, or system (or portion thereof, or set thereof) meets a set of initial design requirements, specifications, and regulations. In the development phase, verification procedures involve performing special tests to model or simulate a portion, or the entirety, of a product, service or system, then performing a review or analysis of the modeling results. In the post-development phase, verification procedures involve regularly repeating tests devised specifically to ensure that the product, service, or system continues to meet the initial design requirements, specifications, and regulations as time progresses It is a process that is used to evaluate whether a product, service, or system complies with regulations, specifications, or conditions imposed at the start of a development phase. Verification can be in development, scale-up, or production. This is often an internal process.

 

Validation & Level of Validation

Validation

Validation is a quality control process intended to check that development and verification procedures for a product, service, or system (or portion thereof, or set thereof) result in a product, service, or system (or portion thereof, or set thereof) that meets initial requirements, specifications, and regulations. For a new development flow or verification flow, validation procedures may involve modeling either flow and using simulations to predict faults or gaps that might lead to invalid or incomplete verification or development of a product, service, or system (or portion thereof, or set thereof). A set of validation requirements, specifications, and regulations may then be used as a basis for qualifying a development flow or verification flow for a product, service, or system (or portion thereof, or set thereof). Additional validation procedures also include those that are designed specifically to ensure that modifications made to an existing qualified development flow or verification flow will have the effect of producing a product, service, or system (or portion thereof, or set thereof) that meets the initial design requirements, specifications, and regulations; these validations help to keep the flow qualified. It is a process of establishing evidence that provides a high degree of assurance that a product, service, or system accomplishes its intended requirements. This often involves acceptance of fitness for purpose with end users and other product stakeholders. This is often an external process.

It is sometimes said that validation can be expressed by the query "Are you building the right thing?" and verification by "Are you building it right?"  "Building the right thing" refers back to the user's needs, while "building it right" checks that the specifications are correctly implemented by the system. In some contexts, it is required to have written requirements for both as well as formal procedures or protocols for determining compliance

 

 

Level of Validation

A level of validation, for the purposes of this guidance, is a numeric code indicating the degree of confidence in the data. These levels provide some commonality among data collected and quality controlled by different agencies, and help ensure that all data have received a comparable level of validation. Various data validation "levels" that apply to air quality and meteorological data have been defined by Mueller and Watson. Basically, four levels of data validation have been defined:

·   Level 0 data validation is essentially raw data obtained directly from the data acquisition systems in the field. Level 0 data have been reduced and possibly reformatted, but are unedited and unreviewed. These data have not received any adjustments for known biases or problems that may have been identified during preventive maintenance checks or audits. These data should be used to monitor the instrument operations on a frequent basis (e.g., daily), but should not be used for regulatory purposes until they receive at least Level 1 validation.

·   Level 1 data validation involves quantitative and qualitative reviews for accuracy, completeness, and internal consistency. Quantitative checks are performed by software screening programs (see Section 8.7.3.2) and qualitative checks are performed by meteorologists or trained personnel who manually review the data for outliers and problems. Quality control flags, consisting of numbers or letters, are assigned to each datum to indicate its quality. A list of suggested quality control codes is given in Table 8-3. Data are only considered at Level 1 after final audit reports have been issued and any adjustments, changes, or modifications to the data have been made.

·   Level 2 data validation involves comparisons with other independent data sets. This includes, for example, intercom paring collocated measurements or making comparisons with other upper-air measurement systems.

·    Level 3 validation involves a more detailed analysis when inconsistencies in analysis and modeling results are found to be caused by measurement errors.

 

Validation Procedures

All necessary supporting material, such as audit reports and any site logs, should be readily available for the level 1 validation. Access to a daily weather archive should be provided for use in relating suspect data with to local and regional meteorological conditions. Any problem data, such as data flagged in an audit, should be corrected prior to the level 1 data validation. The validation procedures described in the following include screening, manual review, and comparison.

Table Suggested quality control (QC) codes for meteorological data Code Meaning Description
Code	Meaning	Description
0	Valid	Observations that were judged accurate within the performance limits of the instrument Observations that required additional processing because the original values were suspect, invalid, or missing.
1	Estimated	Estimated data may be computed from patterns or trends in the data (e.g., via interpolation), or they may be based on the meteorological judgment of the reviewer.
2	Calibration applied	Observations that were corrected using a known, measured  quantity (e.g., instrument offsets measured during audits)
3	Unassigned	Reserved for future use
4	Unassigned	Reserved for future use
5	Unassigned	Reserved for future use
6	Failed automatic QC check	Observations that were flagged with this QC code did not pass screening criteria set in automatic QC software.
7	Suspect	Observations that, in the judgment of the reviewer, were in error because their values violated reasonable physical criteria or did not exhibit reasonable consistency, but a specific cause of the problem was not identified (e.g., excessive wind shear in an adiabatic boundary layer). Additional review using other, independent data sets (Level 2 validation) should be performed to determine the final validity of suspect observations.
8	Invalid	Observations that were  judged inaccurate or in error,  and the cause  of the inaccuracy or error was known (e.g., wind contaminated by ground clutter or a temperature lapse rate that exceeded the auto convective  lapse rate). Besides the QC flag signifying invalid data, the data values themselves should be assigned invalid indicators.
9	Missing	Observations data were not collected.

 

Data Screening

Screening procedures generally include comparisons of measured values to upper and lower limits; these may be physical limits, such as an instrument threshold, or may be established based on experience or historical data. Other types of procedures employed in screening include assessments based on the rate of change of a variable (in these data that change too rapidly or not at all are flagged as suspect) and assessments based on known physical principles relating two or more variables (e.g., the dew point should never exceed the dry-bulb temperature).

Screening may be regarded as an iterative process in which range checks and other screening criteria are revised as necessary based on experience. For example, an initial QA pass of a data set using default criteria may flag values which upon further investigation are determined to be valid for the particular site. In such cases, one or more follow-up QA passes using revised criteria may be necessary to clearly segregate valid and invalid data. Suggested screening criteria are listed in Table 8-4. Data which fail the screening test should be flagged for further investigation.

 

 

Manual Review

The manual review should result in a decision to accept or reject data flagged by the screening process. In addition, manual review may help to identify outliers that were missed by screening. This review should be performed by someone with the necessary training in meteorological monitoring.

In the typical manual review, data should be scanned to determine if the reported values are reasonable and in the proper format. Periods of missing data should be noted and investigated. Data should also be evaluated for temporal consistency. This is particularly useful for identifying outliers in hourly data. Outliers should be reviewed with reference to local meteorological conditions. Data are considered to be at Level 1 validation following the manual review and can be used for modeling and analysis.

Comparison Program

After the data have passed through the screening program, they should be evaluated in a comparison program. Randomly selected values should be manually compared with other available, reliable data (such as, data obtained from the nearest National Weather Service observing station). At least one hour out of every 10 days should be randomly selected. To account for hour-to-hour variability and the spatial displacement of the NWS station, a block of several hours may be more desirable. All data selected should be checked against corresponding measurements at the nearby station(s). In addition, monthly average values should be compared with climatological normals, as determined by the National Weather Service from records over a 30-year period. If discrepancies are found which can not be explained by the geographic difference in the measurement locations or by regional climatic variations, the data should be flagged as questionable.

 

Table Suggested Data Screening Criteria
Variable	Screening Criteria: flag data if the value
Wind Speed	- is less than zero or greater than 25 m/s - does not vary by more than 0.1 m/s for 3 consecutive   hours - does not vary by more than 0.5 m/s for 12 consecutive   hours
Wind Direction	- is less than zero or greater than 360 - is less tan zero or greater than 1 degree for more than 3   consecutive hours - does not vary by more than 10 degrees for 18 consecutive   hours
Temperature	- is greater than the local record high - is less than the local record low   (The above  limit could be applied on a monthly basis) - is greater than a 5 °C change from the previous hour - does not vary by more than 0.5 °C form 12 consecutive   hours
Temperature Difference	- is greater than 0.1 °C/m during the daytime - is less than -0.1 °C/m during the night time - is greater than 5.0 °C or less tan -3.0 °C
Dew Point Temperature	- is greater than the  ambient temperature for the given time   period - is greater than a 5 °C change form the previous hour - does not vary by more than 0.5 °C for 12 consecutive hours - equals the ambeint temperature for 12 consecutive hours
Precipitation	- is greater than 25 mm in one hour - is greater than 100 mm in 24 hours - is less than 50 mm in three moths   (The above values can be adjusted based on local climate)
Pressure	- is greater than 1060 mb (sea level) - is less than 940 mb (sea level)   (The above values should be adjusted for elevations other   than sea level.) - changes by more than 6 mb in three hours
Radiation	- is greater than zero at night - is greater than the maximum possible for th date and   latitude.

 

Further Evaluations

Any data which are flagged by the screening program or the comparison program should be evaluated by personnel with meteorological expertise. Decisions must be made to either accept the flagged data, or discard and replace it with back-up or interpolated data, or data from a nearby representative monitoring station (see Section 1). Any changes in the data due to the validation process should be documented as to the reasons for the change. If problems in the monitoring system are identified, corrective actions should also be documented. Any edited data should continue to be flagged so that its reliability can be considered in the interpretation of the results of any modeling analysis which employs the data.

 

 

Validation Methods

Allowed Character Checks

Checks that ascertain that only expected characters are present in a field. For example a numeric field may only allow the digits 0-9, the decimal point and perhaps a minus sign or commas. A text field such as a personal name might disallow characters such as< and >, as they could be evidence of a markup based security attack. An e-mail address might require exactly one @ sign and various other structural details. Regular expressions effective ways of implementing such checks. (See also data type checks below)

Batch Totals

Checks for missing records. Numerical fields may be added together for all records in a batch. The batch total is entered and the computer checks that the total is correct, e.g., add the 'Total Cost' field of a number of transactions together.

Cardinality Check

Checks that record has a valid number of related records. For example if Contact record classified as a Customer it must have at least one associated Order (Cardinality > 0). If order does not exist for a "customer" record then it must be either changed to "seed" or the order must be created. This type of rule can be complicated by additional conditions. For example if contact record in Payroll database is marked as "former employee", then this record must not have any associated salary payments after the date on which employee left organization (Cardinality = 0).

Check Digits

Used for numerical data. An extra digit is added to a number which is calculated from the digits. The computer checks this calculation when data are entered. For example the last digit of an ISBN for a book is a check digit calculated modulus

Consistency Checks

Checks fields to ensure data in these fields corresponds, e.g., If Title = "Mr.", then Gender = "M".

Control Totals

This is a total done on one or more numeric fields which appears in every record. This is a meaningful total, e.g., add the total payment for a number of Customers.

Cross-System Consistency Checks

Compares data in different systems to ensure it is consistent, e.g., The address for the customer with the same id is the same in both systems. The data may be represented differently in different systems and may need to be transformed to a common format to be compared, e.g., one system may store customer name in a single Name field as 'Doe, John Q', while another in three different fields: First_Name (John), Last_Name (Doe) and Middle_Name (Quality); to compare the two, the validation engine would have to transform data from the second system to match the data from the first, for example, using SQL: Last_Name || ', ' || First_Name || substr(Middle_Name, 1, 1) would convert the data from the second system to look like the data from the first 'Doe, John Q'

Data Type Checks

Checks the data type of the input and give an error message if the input data does not match with the chosen data type, e.g., In an input box accepting numeric data, if the letter 'O' was typed instead of the number zero, an error message would appear.

File Existence Check

Checks that a file with a specified name exists. This check is essential for programs that use file handling.

Format or Picture Check

Checks that the data is in a specified format (template), e.g., dates have to be in the format DD/MM/YYYY.

Regular expressions should be considered for this type of validation.

Hash Totals

This is just a batch total done on one or more numeric fields which appears in every record. This is a meaningless total, e.g., add the Telephone Numbers together for a number of Customers.

Limit Check

Unlike range checks, data is checked for one limit only, upper OR lower, e.g., data should not be greater than 2 (<=2).

Logic Check

Checks that an input does not yield a logical error, e.g., an input value should not be 0 when there will be a number that divides it somewhere in a program.

Presence Check

Checks that important data are actually present and have not been missed out, e.g., customers may be required to have their telephone numbers listed.

Range Check

Checks that the data lie within a specified range of values, e.g., the month of a person's date of birth should lie between 1 and 12.

Referential Integrity

In modern Relational database values in two tables can be linked through foreign key and primary key. If values in the primary key field are not constrained by database internal mechanism, then they should be validated. Validation of the foreign key field checks that referencing table must always refer to a valid row in the referenced table.

Spelling and Grammar Check

Looks for spelling and grammatical errors.

Uniqueness Check

Checks that each value is unique. This can be applied to several fields (i.e. Address, First Name, Last Name).

Table Look Up Check

A table look up check takes the entered data item and compares it to a valid list of entries that are stored in a database table.