Standards arise when groups recognise the benefits of doing things in ways developed over time. Often companies create standards as part of their working practices, impacting products or processes and provide the company that developed them with a commercial advantage. In business, the ‘good guys’ are not always incentivised to share better, safer practices, and it takes maturing of the industry to gather individuals to compile standards that could be used in the interest of the industry.
Individuals and organisations use benchmarks to compare progress (against others or previous results). Competitors will pay a neutral party for mutually beneficial benchmarks e.g. salary surveys, allowing managers to know general achievements or measures.
Regulations arise where business or individual self-interest does not take public risks or well-being into account. The way safety-critical software is tested, is specified, regulated and checked for compliance. Where business aspects are regulated – e.g. the National Credit Act in South Africa – software testing must often confirm that applications (and the companies building them) comply.
Meeting regulatory requirements can take a large percentage of company development (and testing) budgets. Understandably, organisations are not always keen to comply, but compliance is mandatory with penalties for non-compliance.
Frameworks guide organisations in the right direction, as opposed to forcing them to meet requirements. Frameworks help people to know what the next steps are for consistent and efficient progress and to measure and improve processes.
Standards include standard terms and vocabulary (IEEE Std 610.12-1990 IEEE Standard Glossary of Software Engineering Terminology) and testing-related terminology in BS 7925-1: Glossary of terms used in software testing by the British Computer Society.
Standardising terminology and a glossary of terms used in a large corporate pays large dividends. New staff members can read and quickly grasp concepts, definitions, acronyms and abbreviations used in the organisation. These benefits are even greater when whole industries and countries use the same meanings and terminology.
The IEEE 829 of 1998, Standard for Software Testing Documentation explains 8 different testing documents. This test plan is used around the world. Figure 1 shows a test methodology which uses this document against a testing process and provides support for populating them with relevant information (reproduced from Test and Data Services Top Practice™ Manual: Introduction to DREAM Detailed Requirements Extraction and Management™).
Figure 1: Support for Test Process and Documents (© 2005 Test and Data Services (Pty) Ltd)
Figure 2 shows selected document relationships using the IEEE Standard (reproduced from Test and Data Services Top Practice™ Manual: Introduction to DREAM Detailed Requirements Extraction and Management™). Figures 1 and 2 illustrate ways that the IEEE 829 standard can be used by software testers.
The BS 7925-2: Standard for Software Component Testing describes how to conduct structured testing techniques.
Figure 2: Test Documentation Relationships (© Test and Data Services (Pty) Ltd)
Standards support a common approach and guarantee a minimum acceptable practice. On the negative side, they may stifle creativity, remain unread, be misinterpreted, be poorly applied and policed, and may age. At best they are supportive, continually improved and reflect the best-known practise (an aid to continuous, controlled and communicated improvements).
Work that follows standards is auditable and repeatable. Effectiveness and efficiency may improve, particularly if the standard is up-to-date, of high quality, and interpreted correctly. Standards can and have brought industries forward as a group.
ISO 9126: This standard defines quality by six characteristics and sub-characteristics:
Quality is measured in a number of dimensions and this standard is one where testers have often expanded, changed, or redefined terms to suit their needs. This illustrates that some areas may still be rapidly advancing while others have stabilised.
Where there could be a high cost of failure – where health, safety, lives, money, reputation, efficiencies or time can be at risk, regulators step in.
Electric-Programmable Sector:
Airborne Systems DO-178B: "Software Considerations in Airborne Systems and Equipment Certification". In this standard, “Trustworthiness of software is an absolute concept independent of the verification process used.”
Railways Industry: EN 50128 Software for railway control and protection systems.
Energy (Nuclear Power) Industry: IEC 880 Software safety systems in nuclear power stations.
Defence Industry:
Common factors with the standards mentioned are that they all address risks in industries where these might have severe consequences if not mitigated. I believe computer systems are critical in most businesses. Therefore, good software and testing processes are becoming increasingly important.
The standards take a view that good software development processes implies a good outcome. Some standards reflect that training of practitioners to correctly interpret and apply standards is becoming increasingly important.
The standards regulators realise that not everything can be measured during development and testing processes, and measures are done at key checkpoints in the individual processes. Compliance to standards is difficult to monitor, resulting in too many after-the-event discoveries leading to undue losses. Ideally, each industry requires processes which include regular audits and feedback.
Many of the industries divide the risks within the work into required integrity categories. Testing types or approaches are matched to each category, with more testing applied the higher the required integrity. For many safety critical systems, whitebox testing including code coverage percentages is mandatory at the higher integrity levels.
Each industry’s integrity levels and corresponding approaches do not necessarily match the levels and approaches in other industries.
Some industries realised that classical statistical techniques cannot be applied to software testing. Instead, an assemblage of testing best practices is used to ensure ‘an extremely low probability’ of failure.
Traceable, auditable testing is often required of third party companies to assess evidence of compliance to each industry standard.
Benchmarks are a more positive aid to better software quality and testing. Successful, measured achievements are shared between industry participants who use this information to set improvement goals. These are industry-dependent and specific benchmarks may vary over time whereas others may be part of entrenched industry practice.
Software or general quality frameworks on the one hand, such as Total Quality Management (TQM), standards from the International Standards Organisation (ISO), Software Process Improvement (SPI), Capability Maturity Model (CMM) and CMMI help build general and software quality respectively.
Frameworks such as Test Maturity Model (TMM) and Test Process Improvement (TPI®), on the other hand, are designed to improve testing initiatives.
Most of the frameworks suggest progressive stages through which quality increases. Each framework begins with almost no prerequisites, advances through better definition of processes, passes through some kind of quantification by metrics, and ends with optimisation.
Figure 3: Various Quality and Testing Models or Frameworks
Figure 3 illustrates the need to understand the differences and similarities between models and to choose the model or framework best suited to your company needs.
The presence in the industry of many standards, regulations, benchmarks and frameworks for software quality suggest a maturing of the industry. Consolidation of these quality items should continue as the industry matures further.
Wayne Mallinson
info@testfocus.co.za
ISO Standards: http://www.iso.com/
IEC Standards: http://www.iec.ch/
MISRA Standards: http://www.misra.org.uk/
IEEE Standards: http://standards.ieee.org/
DO Standards: http://www.rtca.org/
Def Stan Standards: http://www.dstan.mod.uk/
American National Standards Institute (ANSI): http://www.ansi.org/
British Standards (BS): http://www.bsi-global.com/
The reproduction, adaptation or broadcast, without permission, of any articles
or photographs published in this
publication, is forbidden and copyright
is expressly reserved for Test and Data Services (Pty) Ltd.
Opinions expressed by contributors are their own and do not necessarily express the opinions of the Publisher, Editor or members of the Editorial Panel.