In computing, software engineering, and software testing, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed. The use of oracles involves comparing the output(s) of the system under test, for a given test-case input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden. Additional work on different kinds of oracles was explored by Elaine Weyuker.
Oracles often operate separately from the system under test. However, method postconditions are part of the system under test, as automated oracles in design by contract models. Determining the correct output for a given input (and a set of program or system states) is known as the oracle problem or test oracle problem, which is a much harder problem than it seems, and involves working with problems related to controllability and observability.
Categories
A research literature survey covering 1978 to 2012 found several potential categories of test oracles.
Specified
These oracles are typically associated with formalized approaches to software modeling and software code construction. They are connected to formal specification, model-based design which may be used to generate test oracles, state transition specification for which oracles can be derived to aid model-based testing and protocol conformance testing, and design by contract for which the equivalent test oracle is an assertion.
Specified Test Oracles have a number of challenges. Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behavior.
Derived
A derived test oracle differentiates correct and incorrect behavior by using information derived from artifacts of the system. These may include documentation, system execution results and characteristics of versions of the system under test. Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions.
A pseudo-oracle falls into the category of derived test oracle. A pseudo-oracle, as defined by Weyuker, is a separately written program which can take the same input as the program or system under test so that their outputs may be compared to understand if there might be a problem to investigate.
A partial oracle is a hybrid between specified test oracle and derived test oracle. It specifies important (but not complete) properties of the system under test. For example, metamorphic testing exploits such properties, called metamorphic relations, across multiple executions of the system.
Implicit
An implicit test oracle relies on implied information and assumptions. For example, there may be some implied conclusion from a program crash, i.e. unwanted behavior - an oracle to determine that there may be a problem. There are a number of ways to search and test for unwanted behavior, whether some call it negative testing, where there are specialized subsets such as fuzzing.
There are limitations in implicit test oracles - as they rely on implied conclusions and assumptions. For example, a program or process crash may not be a priority issue if the system is a fault-tolerant system and so operating under a form of self-healing/self-management. Implicit test oracles may be susceptible to false positives due to environment dependencies. Property based testing relies on implicit oracles.