mirror of
https://github.com/classilla/tenfourfox.git
synced 2024-10-26 13:27:27 +00:00
239 lines
8.9 KiB
ReStructuredText
239 lines
8.9 KiB
ReStructuredText
Getting Started
|
|
===============
|
|
|
|
Installing wptrunner
|
|
--------------------
|
|
|
|
The easiest way to install wptrunner is into a virtualenv, using pip::
|
|
|
|
virtualenv wptrunner
|
|
cd wptrunner
|
|
source bin/activate
|
|
pip install wptrunner
|
|
|
|
This will install the base dependencies for wptrunner, but not any
|
|
extra dependencies required to test against specific browsers. In
|
|
order to do this you must use use the extra requirements files in
|
|
``$VIRTUAL_ENV/requirements/requirements_browser.txt``. For example,
|
|
in order to test against Firefox you would have to run::
|
|
|
|
pip install -r requirements/requirements_firefox.txt
|
|
|
|
If you intend to work on the code, the ``-e`` option to pip should be
|
|
used in combination with a source checkout i.e. inside a virtual
|
|
environment created as above::
|
|
|
|
git clone https://github.com/w3c/wptrunner.git
|
|
cd wptrunner
|
|
pip install -e ./
|
|
|
|
In addition to the dependencies installed by pip, wptrunner requires
|
|
a copy of the web-platform-tests repository. This can be located
|
|
anywhere on the filesystem, but the easiest option is to put it
|
|
under the same parent directory as the wptrunner checkout::
|
|
|
|
git clone https://github.com/w3c/web-platform-tests.git
|
|
|
|
It is also necessary to generate a web-platform-tests ``MANIFEST.json``
|
|
file. It's recommended to also put that under the same parent directory as
|
|
the wptrunner checkout, in a directory named ``meta``::
|
|
|
|
mkdir meta
|
|
cd web-platform-tests
|
|
python manifest --path ../meta/MANIFEST.json
|
|
|
|
The ``MANIFEST.json`` file needs to be regenerated each time the
|
|
web-platform-tests checkout is updated. To aid with the update process
|
|
there is a tool called ``wptupdate``, which is described in
|
|
:ref:`wptupdate-label`.
|
|
|
|
Running the Tests
|
|
-----------------
|
|
|
|
A test run is started using the ``wptrunner`` command. The command
|
|
takes multiple options, of which the following are most significant:
|
|
|
|
``--product`` (defaults to `firefox`)
|
|
The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
|
|
|
|
``--binary`` (required if product is `firefox` or `servo`)
|
|
The path to a binary file for the product (browser) to test against.
|
|
|
|
``--webdriver-binary`` (required if product is `chrome`)
|
|
The path to a `*driver` binary; e.g., a `chromedriver` binary.
|
|
|
|
``--certutil-binary`` (required if product is `firefox` [#]_)
|
|
The path to a `certutil` binary (for tests that must be run over https).
|
|
|
|
``--metadata`` (required only when not `using default paths`_)
|
|
The path to a directory containing test metadata. [#]_
|
|
|
|
``--tests`` (required only when not `using default paths`_)
|
|
The path to a directory containing a web-platform-tests checkout.
|
|
|
|
``--prefs-root`` (required only when testing a Firefox binary)
|
|
The path to a directory containing Firefox test-harness preferences. [#]_
|
|
|
|
``--config`` (should default to `wptrunner.default.ini`)
|
|
The path to the config (ini) file.
|
|
|
|
.. [#] The ``--certutil-binary`` option is required when the product is
|
|
``firefox`` unless ``--ssl-type=none`` is specified.
|
|
|
|
.. [#] The ``--metadata`` path is to a directory that contains:
|
|
|
|
* a ``MANIFEST.json`` file (the web-platform-tests documentation has
|
|
instructions on generating this file)
|
|
* (optionally) any expectation files (see :ref:`wptupdate-label`)
|
|
|
|
.. [#] Example ``--prefs-root`` value: ``~/mozilla-central/testing/profiles``.
|
|
|
|
There are also a variety of other command-line options available; use
|
|
``--help`` to list them.
|
|
|
|
The following examples show how to start wptrunner with various options.
|
|
|
|
------------------
|
|
Starting wptrunner
|
|
------------------
|
|
|
|
The examples below assume the following directory layout,
|
|
though no specific folder structure is required::
|
|
|
|
~/testtwf/wptrunner # wptrunner checkout
|
|
~/testtwf/web-platform-tests # web-platform-tests checkout
|
|
~/testtwf/meta # metadata
|
|
|
|
To test a Firefox Nightly build in an OS X environment, you might start
|
|
wptrunner using something similar to the following example::
|
|
|
|
wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
|
|
--binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
|
|
--certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
|
|
--prefs-root=~/mozilla-central/testing/profiles
|
|
|
|
|
|
And to test a Chromium build in an OS X environment, you might start
|
|
wptrunner using something similar to the following example::
|
|
|
|
wptrunner --metadata=~/testtwf/meta/ --tests=~/testtwf/web-platform-tests/ \
|
|
--binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
|
|
--webdriver-binary=/usr/local/bin/chromedriver --product=chrome
|
|
|
|
--------------------
|
|
Running test subsets
|
|
--------------------
|
|
|
|
To restrict a test run just to tests in a particular web-platform-tests
|
|
subdirectory, specify the directory name in the positional arguments after
|
|
the options; for example, run just the tests in the `dom` subdirectory::
|
|
|
|
wptrunner --metadata=~/testtwf/meta --tests=~/testtwf/web-platform-tests/ \
|
|
--binary=/path/to/firefox --certutil-binary=/path/to/certutil \
|
|
--prefs-root=/path/to/testing/profiles \
|
|
dom
|
|
|
|
-------------------
|
|
Running in parallel
|
|
-------------------
|
|
|
|
To speed up the testing process, use the ``--processes`` option to have
|
|
wptrunner run multiple browser instances in parallel. For example, to
|
|
have wptrunner attempt to run tests against with six browser instances
|
|
in parallel, specify ``--processes=6``. But note that behaviour in this
|
|
mode is necessarily less deterministic than with ``--processes=1`` (the
|
|
default), so there may be more noise in the test results.
|
|
|
|
-------------------
|
|
Using default paths
|
|
-------------------
|
|
|
|
The (otherwise-required) ``--tests`` and ``--metadata`` command-line
|
|
options/flags be omitted if any configuration file is found that
|
|
contains a section specifying the ``tests`` and ``metadata`` keys.
|
|
|
|
See the `Configuration File`_ section for more information about
|
|
configuration files, including information about their expected
|
|
locations.
|
|
|
|
The content of the ``wptrunner.default.ini`` default configuration file
|
|
makes wptrunner look for tests (that is, a web-platform-tests checkout)
|
|
as a subdirectory of the current directory named ``tests``, and for
|
|
metadata files in a subdirectory of the current directory named ``meta``.
|
|
|
|
Output
|
|
------
|
|
|
|
wptrunner uses the :py:mod:`mozlog` package for output. This
|
|
structures events such as test results or log messages as JSON objects
|
|
that can then be fed to other tools for interpretation. More details
|
|
about the message format are given in the
|
|
:py:mod:`mozlog` documentation.
|
|
|
|
By default the raw JSON messages are dumped to stdout. This is
|
|
convenient for piping into other tools, but not ideal for humans
|
|
reading the output. :py:mod:`mozlog` comes with several other
|
|
formatters, which are accessible through command line options. The
|
|
general format of these options is ``--log-name=dest``, where ``name``
|
|
is the name of the format and ``dest`` is a path to a destination
|
|
file, or ``-`` for stdout. The raw JSON data is written by the ``raw``
|
|
formatter so, the default setup corresponds to ``--log-raw=-``.
|
|
|
|
A reasonable output format for humans is provided as ``mach``. So in
|
|
order to output the full raw log to a file and a human-readable
|
|
summary to stdout, one might pass the options::
|
|
|
|
--log-raw=output.log --log-mach=-
|
|
|
|
Configuration File
|
|
------------------
|
|
|
|
wptrunner uses a ``.ini`` file to control some configuration
|
|
sections. The file has three sections; ``[products]``,
|
|
``[manifest:default]`` and ``[web-platform-tests]``.
|
|
|
|
``[products]`` is used to
|
|
define the set of available products. By default this section is empty
|
|
which means that all the products distributed with wptrunner are
|
|
enabled (although their dependencies may not be installed). The set
|
|
of enabled products can be set by using the product name as the
|
|
key. For built in products the value is empty. It is also possible to
|
|
provide the path to a script implementing the browser functionality
|
|
e.g.::
|
|
|
|
[products]
|
|
chrome =
|
|
netscape4 = path/to/netscape.py
|
|
|
|
``[manifest:default]`` specifies the default paths for the tests and metadata,
|
|
relative to the config file. For example::
|
|
|
|
[manifest:default]
|
|
tests = ~/testtwf/web-platform-tests
|
|
metadata = ~/testtwf/meta
|
|
|
|
|
|
``[web-platform-tests]`` is used to set the properties of the upstream
|
|
repository when updating the paths. ``remote_url`` specifies the git
|
|
url to pull from; ``branch`` the branch to sync against and
|
|
``sync_path`` the local path, relative to the configuration file, to
|
|
use when checking out the tests e.g.::
|
|
|
|
[web-platform-tests]
|
|
remote_url = https://github.com/w3c/web-platform-tests.git
|
|
branch = master
|
|
sync_path = sync
|
|
|
|
A configuration file must contain all the above fields; falling back
|
|
to the default values for unspecified fields is not yet supported.
|
|
|
|
The ``wptrunner`` and ``wptupdate`` commands will use configuration
|
|
files in the following order:
|
|
|
|
* Any path supplied with a ``--config`` flag to the command.
|
|
|
|
* A file called ``wptrunner.ini`` in the current directory
|
|
|
|
* The default configuration file (``wptrunner.default.ini`` in the
|
|
source directory)
|