mirror of
https://github.com/classilla/tenfourfox.git
synced 2024-07-06 19:29:27 +00:00
73 lines
2.8 KiB
Markdown
73 lines
2.8 KiB
Markdown
Some testing scenarios are intrinsically difficult to automate and
|
|
require a human to run the test and check the pass condition.
|
|
|
|
## When to Write Manual Tests
|
|
|
|
Whenever possible it's best to write a fully automated test. For a
|
|
browser vendor it's possible to run an automated test hundreds of
|
|
times a day, but manual tests are likely to be run a handful of times
|
|
a year. This makes them significantly less useful for catching
|
|
regressions than automated tests.
|
|
|
|
However, there are certain scenarios in which this is not yet
|
|
possible. For example:
|
|
|
|
* Tests that require interaction with browser security UI (e.g. a test
|
|
in which a user refuses a geolocation permissions grant)
|
|
|
|
* Tests that require interaction with the underlying OS e.g. tests for
|
|
drag and drop from the desktop onto the browser
|
|
|
|
* Tests that require non-default browser configuration e.g. images
|
|
disabled
|
|
|
|
* Tests that require interaction with the physical environment
|
|
e.g. tests that the vibration API causes the device to vibrate or
|
|
that various sensor APIs respond in the expected way.
|
|
|
|
There are also some rare cases where it isn't possible to write a layout
|
|
test as a reftest, and a manual test must be written instead.
|
|
|
|
## Requirements for a Manual Test
|
|
|
|
Manual tests are distinguished by their filename; all manual tests
|
|
have filenames of the form `name-manual.ext` i.e. a `-manual`
|
|
suffix after the main filename but before the extension.
|
|
|
|
Manual tests must be fully
|
|
[self-describing](test-style-guielines.html#self-describing-tests). It
|
|
is particularly important for these tests that it is easy to determine
|
|
the result from the information presented to the tester, because a
|
|
tester may have hundreds of tests to get through, and little
|
|
understanding of the features that they are testing. Therefore
|
|
minimalism is a virtue. An ideal self-describing test will have:
|
|
|
|
* Step-by-step instructions for performing the test (if required)
|
|
|
|
* A clear statement of the test outcome (if it can be automatically
|
|
determined after some setup) or of how to determine the outcome.
|
|
|
|
Any information other than this (e.g. quotes from the spec) should be
|
|
avoided.
|
|
|
|
## Using testharness.js for Manual Tests
|
|
|
|
A convenient way to present the results of a test that can have the
|
|
result determined by script after some manual setup steps is to use
|
|
testharness.js to determine and present the result. In this case one
|
|
must pass `{explicit_timeout: true}` in a call to `setup()` in order
|
|
to disable the automatic timeout of the test. For example:
|
|
|
|
```html
|
|
<!doctype html>
|
|
<title>Manual click on button triggers onclick handler</title>
|
|
<script src="/resources/testharness.js"></script>
|
|
<script src="/resources/testharnessreport.js"></script>
|
|
<script>
|
|
setup({explicit_timeout: true})
|
|
</script>
|
|
<p>Click on the button below. If a "PASS" result appears the test
|
|
passes, otherwise it fails</p>
|
|
<button onclick="done()">Click Here</button>
|
|
```
|