What are Test Oracles?

Sometimes you are tasked with testing a product or feature that has a lack of documentation and requirements. When we are lacking this information it can be tough to work out if what we are seeing is expected. You really need a clear way to recognise problems with your application.

Test Oracles are a mechanism that we can use to work out some truths in the product or feature. We compare the output from the application, to the oracles we have gathered.

Why do we need Oracles?

There are many reasons why we need oracles when testing. We need orcales so we are able to determine if a test has passed or failed. When speaking with developers we need a way to clearly show what the issues are and why the test might have failed. The opposite is also true, if challenged, we need a way to show why the test has passed.

Even so called ‘complete specifications’ will have information missing that could be important to the application, especially if the requirements were created before any code has been written. This is because we have a limited amount of information before creating the application, we always gather more information as we build, test and explore.

Researching and finding oracles also helps us learn about the application, we get a deep understanding on what we are expecting the application to do, this should make spotting issues easier.

How can we find Test Oracles?

It is important to look around the business and assess what information you have available to you. Use your analytical and researching skills to hunt down important information and extract it, so you can use this as your oracle.

  • **Intranets: **In most organisations there are internal systems, such as Confluence. Intranets are usually full of detailed information that can be used as oracles.
  • **Bug Tracking Tool: **Such as JIRA, previous issues or stories can be identified and we can view things like acceptance criteria, summaries.
  • **Communication Tools: **Yammer, Flowdock etc.
  • Help Section: The ‘help’ section should be a really important place to look, as this should be detailed enough that users are able to follow and work out how the application is used. If the help documentation is yet to be written, talk to the team responsible for creating the help pages, how are they finding out the information on what the new product or feature will be doing?
  • Talk to Different Teams: Have other teams worked on this section of the application before? Do they have knowledge in the area you are about to add or refactor?

Using the James Bach and Michael Boltons’ Mnemonic FEW HICCUPS

Created by James Bach and Michael Bolton – see article links below

In Michael Boltons article “Testing without a Map” he explains the mnemonic HICCUPS. He has since added to this mnemonic to include FEW. The idea is that a product should be consistent with the following:

  • Familiarity. Expect the system to be inconsistent with patterns of familiar problems.

  • Explainability. Expect a system to be understandable to the degree that we can articulately explain its behaviour to ourselves and others.

  • World. We expect the product to be consistent with things that we know about or can observe in the world.

  • History: Expect the present version of the system to be consistent with past versions of it.

  • **Image: **Expect the system to be consistent with an image that the organization wants to project, with its brand, or with its reputation.

  • **Comparable Products: **Expect the system to be consistent with systems that are in some way comparable. This includes other products in the same product line; competitive products, services, or systems; or products that are not in the same category but which process the same data; or alternative processes or algorithms.

  • **Claims: **Consider that the system should be consistent with things important people say about it, whether in writing (references specifications, design documents, manuals…) or in conversation (meetings, public announcements, lunchroom conversations…).

  • Users’ Desires: Believe that the system should be consistent with ideas about what reasonable users might want.

  • **Product: **Expect each element of the system (or product) to be consistent with comparable elements in the same system.

  • Purpose: Expect the system to be consistent with the explicit and implicit uses to which people might put it.

  • **Statutes and Standards: **Expect a system to be consistent with relevant statutes, acts, laws, regulations, or standards.


Further Information on Few Hiccups

Few Hiccups – Michael Bolton

Testing Without a Map – Michael Bolton

Other Articles on Oracles

Examples of Test Oracles – Cem Kaner

Heuristics and Oracles – Katrina Clokie

Testing with Heuristics and Oracles – Mike Talks