mirror of
https://github.com/classilla/tenfourfox.git
synced 2024-10-22 02:25:05 +00:00
107 lines
4.7 KiB
ReStructuredText
107 lines
4.7 KiB
ReStructuredText
|
wptrunner Design
|
||
|
================
|
||
|
|
||
|
The design of wptrunner is intended to meet the following
|
||
|
requirements:
|
||
|
|
||
|
* Possible to run tests from W3C web-platform-tests.
|
||
|
|
||
|
* Tests should be run as fast as possible. In particular it should
|
||
|
not be necessary to restart the browser between tests, or similar.
|
||
|
|
||
|
* As far as possible, the tests should run in a "normal" browser and
|
||
|
browsing context. In particular many tests assume that they are
|
||
|
running in a top-level browsing context, so we must avoid the use
|
||
|
of an ``iframe`` test container.
|
||
|
|
||
|
* It must be possible to deal with all kinds of behaviour of the
|
||
|
browser runder test, for example, crashing, hanging, etc.
|
||
|
|
||
|
* It should be possible to add support for new platforms and browsers
|
||
|
with minimal code changes.
|
||
|
|
||
|
* It must be possible to run tests in parallel to further improve
|
||
|
performance.
|
||
|
|
||
|
* Test output must be in a machine readable form.
|
||
|
|
||
|
Architecture
|
||
|
------------
|
||
|
|
||
|
In order to meet the above requirements, wptrunner is designed to
|
||
|
push as much of the test scheduling as possible into the harness. This
|
||
|
allows the harness to monitor the state of the browser and perform
|
||
|
appropriate action if it gets into an unwanted state e.g. kill the
|
||
|
browser if it appears to be hung.
|
||
|
|
||
|
The harness will typically communicate with the browser via some remote
|
||
|
control protocol such as WebDriver. However for browsers where no such
|
||
|
protocol is supported, other implementation strategies are possible,
|
||
|
typically at the expense of speed.
|
||
|
|
||
|
The overall architecture of wptrunner is shown in the diagram below:
|
||
|
|
||
|
.. image:: architecture.svg
|
||
|
|
||
|
The main entry point to the code is :py:func:`run_tests` in
|
||
|
``wptrunner.py``. This is responsible for setting up the test
|
||
|
environment, loading the list of tests to be executed, and invoking
|
||
|
the remainder of the code to actually execute some tests.
|
||
|
|
||
|
The test environment is encapsulated in the
|
||
|
:py:class:`TestEnvironment` class. This defers to code in
|
||
|
``web-platform-tests`` which actually starts the required servers to
|
||
|
run the tests.
|
||
|
|
||
|
The set of tests to run is defined by the
|
||
|
:py:class:`TestLoader`. This is constructed with a
|
||
|
:py:class:`TestFilter` (not shown), which takes any filter arguments
|
||
|
from the command line to restrict the set of tests that will be
|
||
|
run. The :py:class:`TestLoader` reads both the ``web-platform-tests``
|
||
|
JSON manifest and the expectation data stored in ini files and
|
||
|
produces a :py:class:`multiprocessing.Queue` of tests to run, and
|
||
|
their expected results.
|
||
|
|
||
|
Actually running the tests happens through the
|
||
|
:py:class:`ManagerGroup` object. This takes the :py:class:`Queue` of
|
||
|
tests to be run and starts a :py:class:`testrunner.TestRunnerManager` for each
|
||
|
instance of the browser under test that will be started. These
|
||
|
:py:class:`TestRunnerManager` instances are each started in their own
|
||
|
thread.
|
||
|
|
||
|
A :py:class:`TestRunnerManager` coordinates starting the product under
|
||
|
test, and outputting results from the test. In the case that the test
|
||
|
has timed out or the browser has crashed, it has to restart the
|
||
|
browser to ensure the test run can continue. The functionality for
|
||
|
initialising the browser under test, and probing its state
|
||
|
(e.g. whether the process is still alive) is implemented through a
|
||
|
:py:class:`Browser` object. An implementation of this class must be
|
||
|
provided for each product that is supported.
|
||
|
|
||
|
The functionality for actually running the tests is provided by a
|
||
|
:py:class:`TestRunner` object. :py:class:`TestRunner` instances are
|
||
|
run in their own child process created with the
|
||
|
:py:mod:`multiprocessing` module. This allows them to run concurrently
|
||
|
and to be killed and restarted as required. Communication between the
|
||
|
:py:class:`TestRunnerManager` and the :py:class:`TestRunner` is
|
||
|
provided by a pair of queues, one for sending messages in each
|
||
|
direction. In particular test results are sent from the
|
||
|
:py:class:`TestRunner` to the :py:class:`TestRunnerManager` using one
|
||
|
of these queues.
|
||
|
|
||
|
The :py:class:`TestRunner` object is generic in that the same
|
||
|
:py:class:`TestRunner` is used regardless of the product under
|
||
|
test. However the details of how to run the test may vary greatly with
|
||
|
the product since different products support different remote control
|
||
|
protocols (or none at all). These protocol-specific parts are placed
|
||
|
in the :py:class:`Executor` object. There is typically a different
|
||
|
:py:class:`Executor` class for each combination of control protocol
|
||
|
and test type. The :py:class:`TestRunner` is responsible for pulling
|
||
|
each test off the :py:class:`Queue` of tests and passing it down to
|
||
|
the :py:class:`Executor`.
|
||
|
|
||
|
The executor often requires access to details of the particular
|
||
|
browser instance that it is testing so that it knows e.g. which port
|
||
|
to connect to to send commands to the browser. These details are
|
||
|
encapsulated in the :py:class:`ExecutorBrowser` class.
|