What is Exploratory Testing?

by James Bach

This first appeared on www.StickyMinds.com as a column feature

Exploratory software testing is a powerful and fun approach to testing. In some situations, it can be orders of magnitude more productive than scripted testing. I haven't found a tester yet who didn't, at least unconsciously, perform exploratory testing at one time or another. Yet few of us study this approach, and it doesn't get much respect in our field. It's high time we stop the denial, and publicly recognize the exploratory approach for what it is: scientific thinking in real-time. Friends, that's a good thing.

Concurrent Test Design and Execution

The plainest definition of exploratory testing is test design and test execution at the same time. This is the opposite of scripted testing (predefined test procedures, whether manual or automated). Exploratory tests, unlike scripted tests, are not defined in advance and carried out precisely according to plan. This may sound like a straightforward distinction, but in practice it's murky. That's because "defined" is a spectrum. Even an otherwise elaborately defined test procedure will leave many interesting details (such as how quickly to type on the keyboard, or what kinds of behavior to recognize as a failure) to the discretion of the tester. Likewise, even a free-form exploratory test session will involve tacit constraints or mandates about what parts of the product to test, or what strategies to use. A good exploratory tester will write down test ideas and use them in later test cycles. Such notes sometimes look a lot like test scripts, even if they aren't. Exploratory testing is sometimes confused with "ad hoc" testing. Ad hoc testing normally refers to a process of improvised, impromptu bug searching. By definition, anyone can do ad hoc testing. The term "exploratory testing"--coined by Cem Kaner, in Testing Computer Software-- refers to a sophisticated, thoughtful approach to ad hoc testing. In the last decade, James Whittaker (at Florida Tech), Cem Kaner and I have worked to identify the skills and techniques of excellent exploratory testing. For one example of a fully defined and articulated process of exploratory testing, see the General Functionality and Stability Test Procedure for Microsoft's Windows 2000 Compatibility Certification program. This document is publicly available on Microsoft's web site, or on my site at http://www.satisfice.com/tools/procedure.pdf.

Balancing Exploratory Testing With Scripted Testing

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle, or when we haven't yet had the opportunity to create those tests. If we are running scripted tests, and new information comes to light that suggests a better test strategy, we may switch to an exploratory mode (as in the case of discovering a new failure that requires investigation). Conversely, we take a more scripted approach when there is little uncertainty about how we want to test, new tests are relatively unimportant, the need for efficiency and reliability in executing those tests is worth the effort of scripting, and when we are prepared to pay the cost of documenting and maintaining tests.

The results of exploratory testing aren't necessarily radically different than those of scripted testing, and the two approaches to testing are fully compatible. Companies such as Nortel and Microsoft commonly use both approaches on the same project. Still there are many important differences between the two approaches.

Why Do Exploratory Testing?

Recurring themes in the management of an effective exploratory test cycle are tester, test strategy, test reporting and test mission. The scripted approach to testing attempts to mechanize the test process by taking test ideas out of a test designer's head and putting them on paper. There's a lot of value in that way of testing. But exploratory testers take the view that writing down test scripts and following them tends to disrupt the intellectual processes that make testers able to find important problems quickly. The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time. That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test. In the rapid testing classes at Satisfice, Inc., we have equipment that watches testers invent tests in real-time. When the instructor makes a new suggestion for what to test, or provides new information to the testers about the product, we observe and measure how a roomful of exploratory testers reacts to that information. Free from the encumbrance of pre-documentation, they immediately incorporate new ideas into their tests.

Scripting has its place. I can imagine testing situations where efficiency and repeatability are so important that we should script or automate them. For example, in the case where a test platform is only intermittently available, such as a client-server project where there are only a few configured servers available and they must be shared by testing and development. The logistics of such a situation may dictate that we script tests carefully in advance to get the most out of every second of limited test execution time. Exploratory testing is especially useful in complex testing situations, when little is known about the product, or as part of preparing a set of scripted tests. The basic rule is this: exploratory testing is called for any time the next test you should perform is not obvious, or when you want to go beyond the obvious. In my experience, that's most of the time.

Basic Guidelines and Checklist for Website Testing

How to Write a Bug Report

See this ::-

Testing Life Cycle

Testing Life Cycle

Test Plan Preparation

The software test plan is the primary means by which software testers communicate to the product development team what they intend to do. The purpose of the software test plan is to prescribe the scope, approach, resource, and schedule of the testing activities. To identify the items being tested, the features to be tested, the testing tasks to be preformed, the personnel responsible for each task, and the risks associated with the plan.

The test plan is simply a by-product of the detailed planning process that’s undertaken to create it. It’s the planning that matters, not the resulting documents. The ultimate goal of the test planning process is communicating the software test team’s intent, its expectations, and its understanding of the testing that’s to be performed.

The following are the important topics, which helps in preparation of Test plan.

· High-Level Expectations

The first topics to address in the planning process are the ones that define the test team’s high-level expectations. They are fundamental topics that must be agreed to, by everyone on the project team, but they are often overlooked. They might be considered “too obvious” and assumed to be understood by everyone, but a good tester knows never to assume anything.

· People, Places and Things

Test plan needs to identify the people working on the project, what they do, and how to contact them. The test team will likely work with all of them and knowing who they are and how to contact them is very important.

Similarly, where documents are stored, where the software can be downloaded from, where the test tools are located, and so on need to be identified.

· Inter-Group Responsibilities

Inter-Group responsibilities identify tasks and deliverables that potentially affect the test effort. The test team’s work is driven by many other functional groups – programmers, project manages, technical writers, and so on. If the responsibilities aren’t planned out, the project, specifically the testing, can become a worst or resulting in important tasks been forgotten.

· Test phases

To plan the test phases, the test team will look at the proposed development model and decide whether unique phases, or stages, of testing should be performed over the course of the project. The test planning process should identify each proposed test phase and make each phase known to the project team. This process often helps the entire team from and understands the overall development model.

· Test strategy

The test strategy describes the approach that the test team will use to test the software both overall and in each phase. Deciding on the strategy is a complex task- one that needs to be made by very experienced testers because it can determine the successes or failure of the test effort.

· Bug Reporting

Exactly what process will be used to manage the bugs needs to be planned so that each and every bug is tracked, from when it’s found to when it’s fixed – and never, ever forgotten.

· Metrics and Statistics

Metrics and statistics are the means by which the progress and the success of the project, and the testing, are tracked. The test planning process should identify exactly what information will be gathered, what decisions will be made with them, and who will be responsible for collecting them.

· Risks and Issues

A common and very useful part of test planning is to identify potential problem or risky areas of the project – ones that could have an impact on the test effort.

Test Case Design

The test case design specification refines the test approach and identifies the features to be covered by the design and its associated tests. It also identifies the test cases and test procedures, if any, required to accomplish the testing and specifics the feature pass or fail criteria. The purpose of the test design specification is to organize and describe the testing needs to be performed on a specific feature.

The following topics address this purpose and should be part of the test design specification that is created:

· Test case ID or identification

A unique identifier that can be used to reference and locate the test design specification the specification should also reference the overall test plan and contain pointers to any other plans or specifications that it references.

· Test Case Description

It is a description of the software feature covered by the test design specification for example, “ the addition function of calculator,” “font size selection and display in word pad,” and “video card configuration testing of quick time.”

· Test case procedure

It is a description of the general approach that will be used to test the features. It should expand on the approach, if any, listed in the test plan, describe the technique to be used, and explain how the results will be verified.

· Test case Input or Test Data

It is the input the data to be tested using the test case. The input may be in any form. Different inputs can be tried for the same test case and test the data entered is correct or not.

· Expected result

It describes exactly what constitutes a pass and a fail of the tested feature. Which is expected to get from the given input.

Test Execution and Test Log Preparation

After test case design, each and every test case is checked and actual result obtained. After getting actual result, with the expected column in the design stage is compared, if both the actual and expected are same, then the test is passed otherwise it will be treated as failed.

Now the test log is prepared, which consists of entire data that were recorded, whether the test failed or passed. It records each and every test case so that it will be useful at the time of revision.

Example

Test case ID	Test case Description	Test status/ result
Sys_xyz_01	Checking the login window	Fail
Sys_xyz_02	Checking the main window	True

Defect Tracking

A defect can be defined in one or two ways. From the producer's viewpoint, a defect is a deviation from specifications, whether missing, wrong, etc. From the Customer's viewpoint, a defect is any that causes customer dissatisfaction, whether in the requirements or not, this is known as "fit for use". It is critical that defects identified at each stage of the project life cycle be tracked to resolution.

Defects are recorded for following major purposes:

· To correct the defect

· To report status of the application

· To gather statistics used to develop defect expectations in future applications

· To improve the software development process

Most project teams utilize some type of tool to support the defect tracking process. This tool could be as simple as a white board or a table created and maintained in a word processor or one of the more robust tools available today, on the market, such as Mercury's Test Director etc. Tools marketed for this purpose usually come with some number of customizable fields for tracking project specific data in addition to the basics. They also provide advanced features such as standard and ad-hoc reporting, e-mail notification to developers and/or testers when a problem is assigned to them, and graphing capabilities.

At a minimum, the tool selected should support the recording and communication significant information about a defect. For example, a defect log could include:

· Defect ID number

· Descriptive defect name and type

· Source of defect -test case or other source

· Defect severity

· Defect priority

· Defect status (e.g. open, fixed, closed, user error, design, and so on) -more robust tools provide a status history for the defect

· Date and time tracking for either the most recent status change, or for each change in the status history

· Detailed description, including the steps necessary to reproduce the defect

· Component or program where defect was found

· Screen prints, logs, etc. that will aid the developer in resolution process

· Stage of origination

· Person assigned to research and/or correct the defect

Severity versus Priority

The severity of a defect should be assigned objectively by the test team based on pre defined severity descriptions. For example a "severity one" defects maybe defined as one that causes data corruption, a system crash, security violations, etc. In large project, it may also be necessary to assign a priority to the defect, which determines the order in which defects should be fixed. The priority assigned to a defect is usually more subjective based upon input from users regarding which defects are most important to them, and therefore should be fixed first.

It is recommended that severity levels be defined at the start of the project so that they intently assigned and understood by the team. This foresight can help test teams avoid the common disagreements with development teams about the criticality of a defect.

Some general principles

· The primary goal is to prevent defects. Wherever this is not possible or practical, the goals are to both find the defect as quickly as possible and minimize the impact of the defect.

· The defect management process, like the entire software development process, should be risk driven, i.e., strategies, priorities and resources should be based on an assessment of the risk and the degree to which the expected impact of risk can be reduced.

· Defect measurement should be integrated into the development process and be used by the project team to improve the development process. In other words, information on defects should be captured at the source as a natural by-product of doing the job. People unrelated to the project or system should not do it.

· As much as possible, the capture and analysis of the information should be automated. There should be a document, which includes a list of tools, which have defect management capabilities and can be used to automate some of the defect management processes.

· Defect information should be used to improve the process. This, in fact, is the primary reason for gathering defect information.

· Imperfect or flawed processes cause most defects. Thus, to prevent defects, the process must be altered.

The Defect Management Process

The key elements of a defect management process are as follows.

· Defect prevention

· Deliverable base-lining

· Defect discovery/defect naming

· Defect resolution

· Process improvement

· Management reporting

16

Test Reports

A final test report should be prepared at the conclusion of each test activity. This might include

· Individual Project Test Report (e.g., a single software system)

· Integration Test Report

· System Test Report

· Acceptance Test Report

The test reports are designed to document the results of testing as defined in the test plan. Without a well-developed test plan, which has been executed in accordance with its criteria, it is difficult to develop a meaningful test report.

It is designed to accomplish three objectives:

· Define the scope of testing - normally a brief recap of the test plan;

· Present the results of testing; and

· Draw conclusions and make recommendations based on those results

The test report may be a combination of electronic data and hard copy. For example, if the function test matrix is maintained electronically, there is no reason to print that, as the paper report will summarize that data, draws the appropriate conclusions, and present recommendations.

The test report has one immediate and three long-term purposes. The immediate purpose is to provide information to the customers of the software system so that they can determine whether the system is ready for production: and if so, to assess the potential consequences and initiate appropriate actions to minimize those consequences.

The first of the three long-term uses is for the project to trace problems in the event the application malfunctions in production. Knowing which functions have been correctly tested and which ones still contain defects can assist in taking corrective action.

The second long-term purpose is to use the data to analyze the rework process for making changes to prevent defects from occurring in the future. Accumulating the results of many test reports to identify which components of the rework process are detect-prone does this. These defect-prone components identify tasks/steps that, if improved, could eliminate or minimize the occurrence of high-frequency defects.

The third long-term purpose is to show what was accomplished.

Individual Project Test Report

These reports focus on individual projects (e.g., software system). When different testers test individual projects, they should prepare a report on their results.

Integration Test Report

Integration testing tests the interfaces between individual projects. A good test plan will identify the interfaces and institute test conditions that will validate interfaces. Given this, the interface report follows the same format as the individual Project Test report, except that the conditions tested are the interfaces.

System Test Report

A system test plan standard that identified the objectives of testing, what was to be tested, how it was to be tested and when tests should occur. The System Test report should present the results of executing that test plan. If this is maintained electronically, it need only be referenced, not included in the report.

Acceptance Test Report

There are two primary objectives for testing. The first is to ensure that the system as implemented meets the real operating needs of the user or customer. If the defined requirements are those true needs, the testing should have accomplished this objective. The second objective is to ensure that the software system can operate in the real-world user environment, which includes people skills and attitudes, time pressures, changing business conditions, and so forth.

Eight Interim Reports:

1. Functional Testing Status

2. Functions Working Timeline

3. Expected verses Actual Defects Detected Timeline

4. Defects Detected verses Corrected Gap Timeline

5. Average Age of Detected Defects by Type

6. Defect Distribution

7. Relative Defect Distribution

8. Testing Action

1. Functional Testing Status Report

This report will show percentages of the functions, which have been:

· Fully Tested

· Tested With Open Defects

· Not Tested

2. Functions Working Timeline report

This report will show the actual plan to have all functions working verses the current status of functions working. An ideal format could be a line graph.

3. Expected verses Actual Defects Detected report

This report will provide an analysis between the number of defects being generated against the expected number of defects expected from the planning stage

4. Defects Detected verses Corrected Gap report

This report, ideally in a line graph format, will show the number of defects uncovered verses the number of defects being corrected and accepted by the testing group. If the gap grows too large, the project may not be ready when originally planned.

5. Average Age Detected Defects by Type report

This report will show the average outstanding defects by type (severity 1, severity 2, etc.). In the planning stage, it is benefic determine the acceptable open days by defect type.

6. Defect Distribution report

This report will show the defect distribution by function or module. It can also include items such as numbers of tests completed.

7. Relative Defect Distribution report

This report will take the previous report (Defect Distribution) and normalize the level of defects. An example would be one application might be more in depth than another, and would probably have a higher level of defects. However, when normalized over the number of functions or lines of code, would show a more accurate level of defects.

8. Testing action report

This report can show many different things, including possible shortfalls in testing. Examples of data to show might be number of severity defects, tests that are behind schedule, and other information that would present an accurate testing picture