There is an important aspect of testing that is on the upswing. Typical testing (good testing, that is) is based on prioritising the scope of testing work by risk and effort. The risk factor is normally associated with functional requirements, and addresses the risk to the project implementing the application. This risk is used to determine which tests to test first, and most. We always take the users' perspective when testing, but do we consider the high-level managers' perspective? Seldom do we actually address the risk that is faced by the organisation as a whole, or the system as a whole; and in this lack of vision there is a great danger. Organisations and systems can fail due to lack of adequate testing to address these high-level requirements.
Just as functional and non-functional requirements are difficult to solicit and test, so are Regulatory requirements. The difference is that - instead of just failing to deliver an important function, or having a server fail due to untested load issues - failure to comply with Regulatory requirements may lead to legal action or even worse, the loss of your business.
In order to give the reader some insight into the magnitude of this issue, I will address some of the aspects that I generally refer to as Regulatory Testing. These issues are not in any order of criticality.
Apart from the functional requirements, which requirements address the vision or mission of the organisation or the business objectives themselves? We also get so engrossed in the deliverables of the project (or blinded by technology) that we fail to see that the true business processes are not being supported, or even addressed by requirements. A good method to addressing this issue is to understand all of the business workflows, and ensure that these are being tested. Special attention must be given to workflows that cross business silos. There is way too much silo-based testing; each business area will look after its own segment of the system (in fact, there is often no test environment that represents the whole business end-to-end system). Larger organisations are faced with the difficulties of middleware testing, and the ownership of this testing - as the middleware often forms the backbone of the IT system - but often there is no single person who is responsible for the testing of the middleware.
It is the responsibility of the board to ensure that IT systems are delivered and maintained as cheaply as possible. By effective use of reviews and inspections defects can be prevented, rework can be reduced, and maintenance costs reduced! These are savings that can be achieved by mature testing techniques (both verification and validation).
Business requirements seldom address privacy, accessibility, and security issues, yet these are so often in international headlines today.
Privacy requirements cover the information security of your clients. How secret is their balance? How secret is their credit card information? How secret is your clients' financial information? Do we test using client records, and then not dispose of this information in the same manner that production data is handled? How happy would your clients be if they discovered that you were testing with their information? Would they sue you if they knew? (Please rather spend the effort to create fictitious test data, than spend the effort to cover up the fact that production data is being used to generate the test data).
How many web applications, ATMs, or internet banking platforms address accessibility? South Africa has a high level of illiteracy, which means that these people are unable to read/understand on-screen instructions. Do you cater for visually disadvantaged clients (how often do you see the buttons and options on an ATM align, yet this is a problem even if you are not visually impaired)? The Australian Olympic Committee had legal action taken against them for lack of accessibility on their web site, and they lost - see www.tomw.net.au/2001/bat2001.html.
What I have found very worrying is the lack of knowledge of industry-specific standards; these will impact the whole business sector, but remain untested due to business competition. Organisations will develop applications that are on the outer fringes of a specific industry, but will not check the industry standards. If, for instance, an application for passenger reservations is being developed, few project teams will bother to check the standards addressing tourism or travel regulations - the assumption is made that the Business Analyst will have included all of this information.
Adherence to industry standards is well understood by organisations that develop aircraft control systems, or software for medical devices. The reason for paying such close attention in these industries is because of the potential danger to life, and the level of control being exercised by the relevant bodies. But who would bother to check financial accounting standards when developing payment systems? These defects are simply fixed in production or propagated into the public systems. Would this picture change if there was a real possibility that somebody would sue you if you failed to test this, or create the associated requirements?
I think that just because we do not write software that is life critical, does not give us an excuse to ignore industry standards. As competition grows, clients will move away to development teams who have knowledge of the various industries, and who know how to apply testing to each specific industry.
It appears strange to address corporate governance and software testing, as the testing is such a low-level issue when compared to the high level issues that are addressed by the boards of companies. However, guidelines and standards such as ITIL, COBIT, and ISO 17799, and the Sarbanes-Oxley act in the USA, are suggesting that the board is responsible for implementing structures that address IT issues, and the associated software testing.
ITIL is a guideline to help organisations address IT Service Management. COBIT specifically addresses IT infrastructure, and an important part of this is the risk of change to the existing systems. Such risk must be addressed through a thorough testing of the change itself (prior to release into production), testing that addresses impacted areas, and then testing the system as a whole. Such risk is addressed through regression testing, but how many organisations have a reliable regression testing strategy in place? I think not too many.
Companies need to set up regression testing strategies that will give guidance on which features will be tested, and which will not. This must not be a fixed set of tests, but must be a suite (of test sets) that addresses different business cycles (end of day, end of month, end of year, billing cycles, etc.), which should be run as appropriate cycles are required (as long as there has been change - do not run tests if there has been no change to the system at all). These tests should be automated, as they seldom change, and are the ideal area to start test automation initiatives.
ISO 17799 and BS 7799-2 are standards that address information security. How can such an issue not be on the minds of members of a board? Currently there is a very high interest in security issues. Recently 3 million credit cards were compromised worldwide. Virus attacks and hacking are on the rise (even worse is the fraud being conducted by internal employees through knowledge of system vulnerabilities). Yet I have met few Product/Project Managers who actually know who conducts the security testing on the systems for which they are responsible. There are 128 detailed checkpoints for system security, yet I have seldom seen security requirements that have any detail. Is your organisation aware of security constraints? Do you know who is testing your security? I would like to suggest that if you do not know these people, now is the time to meet them (I do hope that somebody is testing security in your organisation).
Quality is not the responsibility of the Test Department, but should start at the top. The old testing saying 'you cannot test quality into a system, you engineer it in' is just as true today as it was 10 years ago. Quality starts with the board. Testing measures the presence of quality, and does not provide it. Are boards mandating quality requirements, and ensuring that testing is providing that vision or measure of quality? I have found many testers who are made responsible for critical failures in production, but when investigated, there is no evidence that there was any reference in system/project documentation to quality criteria. I have been testing for years, but have not mastered the art of mind reading (I do not think I ever will). If board members are leaving the definition of quality to the testers, they are in grave danger of producing inferior systems and processes, and - quite frankly - of jeopardising their own business (I do believe that you should be able to measure the quality of your mission and vision statements). The board is responsible for meeting all governance or regulatory requirements.
If you write software, are you responsible for ensuring that it is the correct product, that it was developed using the correct methods, and that it will not cause external systems to fail? Can you be held legally responsible for ensuring the correct implementation?
I have already discussed the possibility that an organisation can be held legally responsible for failure to test software before release into production. One must seriously consider an instance whereby a software development company undertakes to write software for another party - the requirement for this software was to create additional market growth - then the development house fails to adequately test the software, and as a result of this the client lost potential revenue. Would there be a basis for legal action? This is a question that I have thought about often, and I have a strong feeling that there could be grounds for legal action (but please remember that I am not a legal expert).
Not all is doom and gloom. I recommend that most of these issues can be resolved in a very practical manner. The higher level managers prioritise their risks (and vulnerabilities), take the top 10, and list them in a check sheet. It is important that a formal risk analysis (likelihood vs. impact) be conducted so that a quantified approach is used (numbers are normally better than gut feel). These items on the checklist should form the basis of reviews (even better - inspections), or headings on templates to ensure that these issues are addressed. Each time one of these items is not addressed, this should be logged as a defect. Draw up monthly statistics, and report to the board. At the same time draw metrics from production defect logs (use the same top 10 list), and once again report to the board. The gaps are the areas that require immediate operational attention.