mirror of
https://github.com/classilla/tenfourfox.git
synced 2024-10-03 06:55:41 +00:00
430 lines
15 KiB
ReStructuredText
430 lines
15 KiB
ReStructuredText
|
.. _experiments_manifests:
|
||
|
|
||
|
=====================
|
||
|
Experiments Manifests
|
||
|
=====================
|
||
|
|
||
|
*Experiments Manifests* are documents that describe the set of active
|
||
|
experiments a client may run.
|
||
|
|
||
|
*Experiments Manifests* are fetched periodically by clients. When
|
||
|
fetched, clients look at the experiments within the manifest and
|
||
|
determine which experiments are applicable. If an experiment is
|
||
|
applicable, the client may download and start the experiment.
|
||
|
|
||
|
Manifest Format
|
||
|
===============
|
||
|
|
||
|
Manifests are JSON documents where the main element is an object.
|
||
|
|
||
|
The *schema* of the object is versioned and defined by the presence
|
||
|
of a top-level ``version`` property, whose integer value is the
|
||
|
schema version used by that manifest. Each version is documented
|
||
|
in the sections below.
|
||
|
|
||
|
Version 1
|
||
|
---------
|
||
|
|
||
|
Version 1 is the original manifest format.
|
||
|
|
||
|
The following properties may exist in the root object:
|
||
|
|
||
|
experiments
|
||
|
An array of objects describing candidate experiments. The format of
|
||
|
these objects is documented below.
|
||
|
|
||
|
An array is used to create an explicit priority of experiments.
|
||
|
Experiments listed at the beginning of the array take priority over
|
||
|
experiments that follow.
|
||
|
|
||
|
Experiments Objects
|
||
|
^^^^^^^^^^^^^^^^^^^
|
||
|
|
||
|
Each object in the ``experiments`` array may contain the following
|
||
|
properties:
|
||
|
|
||
|
id
|
||
|
(required) String identifier of this experiment. The identifier should
|
||
|
be treated as opaque by clients. It is used to uniquely identify an
|
||
|
experiment for all of time.
|
||
|
|
||
|
xpiURL
|
||
|
(required) String URL of the XPI that implements this experiment.
|
||
|
|
||
|
If the experiment is activated, the client will download and install this
|
||
|
XPI.
|
||
|
|
||
|
xpiHash
|
||
|
(required) String hash of the XPI that implements this experiment.
|
||
|
|
||
|
The value is composed of a hash identifier followed by a colon
|
||
|
followed by the hash value. e.g.
|
||
|
`sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`
|
||
|
are the two supported hashing mechanisms. The hash value is the hex
|
||
|
encoding of the binary hash.
|
||
|
|
||
|
When the client downloads the XPI for the experiment, it should compare
|
||
|
the hash of that XPI against this value. If the hashes don't match,
|
||
|
the client should not install the XPI.
|
||
|
|
||
|
Clients may also use this hash as a means of determining when an
|
||
|
experiment's XPI has changed and should be refreshed.
|
||
|
|
||
|
startTime
|
||
|
Integer seconds since UNIX epoch that this experiment should
|
||
|
start. Clients should not start an experiment if *now()* is less than
|
||
|
this value.
|
||
|
|
||
|
maxStartTime
|
||
|
(optional) Integer seconds since UNIX epoch after which this experiment
|
||
|
should no longer start.
|
||
|
|
||
|
Some experiments may wish to impose hard deadlines after which no new
|
||
|
clients should activate the experiment. This property may be used to
|
||
|
facilitate that.
|
||
|
|
||
|
endTime
|
||
|
Integer seconds since UNIX epoch after which this experiment
|
||
|
should no longer run. Clients should cease an experiment when the current
|
||
|
time is beyond this value.
|
||
|
|
||
|
maxActiveSeconds
|
||
|
Integer seconds defining the max wall time this experiment should be
|
||
|
active for.
|
||
|
|
||
|
The client should deactivate the experiment this many seconds after
|
||
|
initial activation.
|
||
|
|
||
|
This value only involves wall time, not browser activity or session time.
|
||
|
|
||
|
appName
|
||
|
Array of application names this experiment should run on.
|
||
|
|
||
|
An application name comes from ``nsIXULAppInfo.name``. It is a value
|
||
|
like ``Firefox``, ``Fennec``, or `B2G`.
|
||
|
|
||
|
The client should compare its application name against the members of
|
||
|
this array. If a match is found, the experiment is applicable.
|
||
|
|
||
|
minVersion
|
||
|
(optional) String version number of the minimum application version this
|
||
|
experiment should run on.
|
||
|
|
||
|
A version number is something like ``27.0.0`` or ``28``.
|
||
|
|
||
|
The client should compare its version number to this value. If the client's
|
||
|
version is greater or equal to this version (using a version-aware comparison
|
||
|
function), the experiment is applicable.
|
||
|
|
||
|
If this is not specified, there is no lower bound to versions this
|
||
|
experiment should run on.
|
||
|
|
||
|
maxVersion
|
||
|
(optional) String version number of the maximum application version this
|
||
|
experiment should run on.
|
||
|
|
||
|
This is similar to ``minVersion`` except it sets the upper bound for
|
||
|
application versions.
|
||
|
|
||
|
If the client's version is less than or equal to this version, the
|
||
|
experiment is applicable.
|
||
|
|
||
|
If this is not specified, there is no upper bound to versions this
|
||
|
experiment should run on.
|
||
|
|
||
|
version
|
||
|
(optional) Array of application versions this experiment should run on.
|
||
|
|
||
|
This is similar to ``minVersion`` and ``maxVersion`` except only a
|
||
|
whitelisted set of specific versions are allowed.
|
||
|
|
||
|
The client should compare its version to members of this array. If a match
|
||
|
is found, the experiment is applicable.
|
||
|
|
||
|
minBuildID
|
||
|
(optional) String minimum Build ID this experiment should run on.
|
||
|
|
||
|
Build IDs are values like ``201402261424``.
|
||
|
|
||
|
The client should perform a string comparison of its Build ID against this
|
||
|
value. If its value is greater than or equal to this value, the experiment
|
||
|
is applicable.
|
||
|
|
||
|
maxBuildID
|
||
|
(optional) String maximum Build ID this experiment should run on.
|
||
|
|
||
|
This is similar to ``minBuildID`` except it sets the upper bound
|
||
|
for Build IDs.
|
||
|
|
||
|
The client should perform a string comparison of its Build ID against
|
||
|
this value. If its value is less than or equal to this value, the
|
||
|
experiment is applicable.
|
||
|
|
||
|
buildIDs
|
||
|
(optional) Array of Build IDs this experiment should run on.
|
||
|
|
||
|
This is similar to ``minBuildID`` and ``maxBuildID`` except only a
|
||
|
whitelisted set of Build IDs are considered.
|
||
|
|
||
|
The client should compare its Build ID to members of this array. If a
|
||
|
match is found, the experiment is applicable.
|
||
|
|
||
|
os
|
||
|
(optional) Array of operating system identifiers this experiment should
|
||
|
run on.
|
||
|
|
||
|
Values for this array come from ``nsIXULRuntime.OS``.
|
||
|
|
||
|
The client will compare its operating system identifier to members
|
||
|
of this array. If a match is found, the experiment is applicable to the
|
||
|
client.
|
||
|
|
||
|
channel
|
||
|
(optional) Array of release channel identifiers this experiment should run
|
||
|
on.
|
||
|
|
||
|
The client will compare its channel to members of this array. If a match
|
||
|
is found, the experiment is applicable.
|
||
|
|
||
|
If this property is not defined, the client should assume the experiment
|
||
|
is to run on all channels.
|
||
|
|
||
|
locale
|
||
|
(optional) Array of locale identifiers this experiment should run on.
|
||
|
|
||
|
A locale identifier is a string like ``en-US`` or ``zh-CN`` and is
|
||
|
obtained by looking at
|
||
|
``nsIXULChromeRegistry.getSelectedLocale("global")``.
|
||
|
|
||
|
The client should compare its locale identifier to members of this array.
|
||
|
If a match is found, the experiment is applicable.
|
||
|
|
||
|
If this property is not defined, the client should assume the experiment
|
||
|
is to run on all locales.
|
||
|
|
||
|
sample
|
||
|
(optional) Decimal number indicating the sampling rate for this experiment.
|
||
|
|
||
|
This will contain a value between ``0.0`` and ``1.0``. The client should
|
||
|
generate a random decimal between ``0.0`` and ``1.0``. If the randomly
|
||
|
generated number is less than or equal to the value of this field, the
|
||
|
experiment is applicable.
|
||
|
|
||
|
disabled
|
||
|
(optional) Boolean value indicating whether an experiment is disabled.
|
||
|
|
||
|
Normally, experiments are deactivated after a certain time has passed or
|
||
|
after the experiment itself determines it no longer needs to run (perhaps
|
||
|
it collected sufficient data already).
|
||
|
|
||
|
This property serves as a backup mechanism to remotely disable an
|
||
|
experiment before it was scheduled to be disabled. It can be used to
|
||
|
kill experiments that are found to be doing wrong or bad things or that
|
||
|
aren't useful.
|
||
|
|
||
|
If this property is not defined or is false, the client should assume
|
||
|
the experiment is active and a candidate for activation.
|
||
|
|
||
|
frozen
|
||
|
(optional) Boolean value indicating this experiment is frozen and no
|
||
|
longer accepting new enrollments.
|
||
|
|
||
|
If a client sees a true value in this field, it should not attempt to
|
||
|
activate an experiment.
|
||
|
|
||
|
jsfilter
|
||
|
(optional) JavaScript code that will be evaluated to determine experiment
|
||
|
applicability.
|
||
|
|
||
|
This property contains the string representation of JavaScript code that
|
||
|
will be evaluated in a sandboxed environment using JavaScript's
|
||
|
``eval()``.
|
||
|
|
||
|
The string is expected to contain the definition of a JavaScript function
|
||
|
``filter(context)``. This function receives as its argument an object
|
||
|
holding application state. See the section below for the definition of
|
||
|
this object.
|
||
|
|
||
|
The purpose of this property is to allow experiments to define complex
|
||
|
rules and logic for evaluating experiment applicability in a manner
|
||
|
that is privacy conscious and doesn't require the transmission of
|
||
|
excessive data.
|
||
|
|
||
|
The return value of this filter indicates whether the experiment is
|
||
|
applicable. Functions should return true if the experiment is
|
||
|
applicable.
|
||
|
|
||
|
If an experiment is not applicable, they should throw an Error whose
|
||
|
message contains the reason the experiment is not applicable. This
|
||
|
message may be logged and sent to remote servers, so it should not
|
||
|
contain private or otherwise sensitive data that wouldn't normally
|
||
|
be submitted.
|
||
|
|
||
|
If a falsey (or undefined) value is returned, the client should
|
||
|
assume the experiment is not applicable.
|
||
|
|
||
|
If this property is not defined, the client does not consider a custom
|
||
|
JavaScript filter function when determining whether an experiment is
|
||
|
applicable.
|
||
|
|
||
|
JavaScript Filter Context Objects
|
||
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
|
||
|
The object passed to a ``jsfilter`` ``filter()`` function contains the
|
||
|
following properties:
|
||
|
|
||
|
healthReportSubmissionEnabled
|
||
|
This property contains a boolean indicating whether Firefox Health
|
||
|
Report has its data submission flag enabled (whether Firefox Health
|
||
|
Report is sending data to remote servers).
|
||
|
|
||
|
healthReportPayload
|
||
|
This property contains the current Firefox Health Report payload.
|
||
|
|
||
|
The payload format is documented at :ref:`healthreport_dataformat`.
|
||
|
|
||
|
telemetryPayload
|
||
|
This property contains the current Telemetry payload.
|
||
|
|
||
|
The evaluation sandbox for the JavaScript filters may be destroyed
|
||
|
immediately after ``filter()`` returns. This function should not assume
|
||
|
async code will finish.
|
||
|
|
||
|
Experiment Applicability and Client Behavior
|
||
|
============================================
|
||
|
|
||
|
The point of an experiment manifest is to define which experiments are
|
||
|
available and where and how to run them. This section explains those
|
||
|
rules in more detail.
|
||
|
|
||
|
Many of the properties in *Experiment Objects* are related to determining
|
||
|
whether an experiment should run on a given client. This evaluation is
|
||
|
performed client side.
|
||
|
|
||
|
1. Multiple conditions in an experiment
|
||
|
---------------------------------------
|
||
|
|
||
|
If multiple conditions are defined for an experiment, the client should
|
||
|
combine each condition with a logical *AND*: all conditions must be
|
||
|
satisfied for an experiment to run. If one condition fails, the experiment
|
||
|
is not applicable.
|
||
|
|
||
|
2. Active experiment disappears from manifest
|
||
|
---------------------------------------------
|
||
|
|
||
|
If a specific experiment disappears from the manifest, the client should
|
||
|
continue conducting an already-active experiment. Furthermore, the
|
||
|
client should remember what the expiration events were for an experiment
|
||
|
and honor them.
|
||
|
|
||
|
The rationale here is that we want to prevent an accidental deletion
|
||
|
or temporary failure on the server to inadvertantly deactivate
|
||
|
supposed-to-be-active experiments. We also don't want premature deletion
|
||
|
of an experiment from the manifest to result in indefinite activation
|
||
|
periods.
|
||
|
|
||
|
3. Inactive experiment disappears from manifest
|
||
|
-----------------------------------------------
|
||
|
|
||
|
If an inactive but scheduled-to-be-active experiment disappears from the
|
||
|
manifest, the client should not activate the experiment.
|
||
|
|
||
|
If that experiment reappears in the manifest, the client should not
|
||
|
treat that experiment any differently than any other new experiment. Put
|
||
|
another way, the fact an inactive experiment disappears and then
|
||
|
reappears should not be significant.
|
||
|
|
||
|
The rationale here is that server operators should have complete
|
||
|
control of an inactive experiment up to it's go-live date.
|
||
|
|
||
|
4. Re-evaluating applicability on manifest refresh
|
||
|
--------------------------------------------------
|
||
|
|
||
|
When an experiment manifest is refreshed or updated, the client should
|
||
|
re-evaluate the applicability of each experiment therein.
|
||
|
|
||
|
The rationale here is that the server may change the parameters of an
|
||
|
experiment and want clients to pick those up.
|
||
|
|
||
|
5. Activating a previously non-applicable experiment
|
||
|
----------------------------------------------------
|
||
|
|
||
|
If the conditions of an experiment change or the state of the client
|
||
|
changes to allow an experiment to transition from previously
|
||
|
non-applicable to applicable, the experiment should be activated.
|
||
|
|
||
|
For example, if a client is running version 28 and the experiment
|
||
|
initially requires version 29 or above, the client will not mark the
|
||
|
experiment as applicable. But if the client upgrades to version 29 or if
|
||
|
the manifest is updated to require 28 or above, the experiment will
|
||
|
become applicable.
|
||
|
|
||
|
6. Deactivating a previously active experiment
|
||
|
----------------------------------------------
|
||
|
|
||
|
If the conditions of an experiment change or the state of the client
|
||
|
changes and an active experiment is no longer applicable, that
|
||
|
experiment should be deactivated.
|
||
|
|
||
|
7. Calculation of sampling-based applicability
|
||
|
----------------------------------------------
|
||
|
|
||
|
For calculating sampling-based applicability, the client will associate
|
||
|
a random value between ``0.0`` and ``1.0`` for each observed experiment
|
||
|
ID. This random value will be generated the first time sampling
|
||
|
applicability is evaluated. This random value will be persisted and used
|
||
|
in future applicability evaluations for this experiment.
|
||
|
|
||
|
By saving and re-using the value, the client is able to reliably and
|
||
|
consistently evaluate applicability, even if the sampling threshold
|
||
|
in the manifest changes.
|
||
|
|
||
|
Clients should retain the randomly-generated sampling value for
|
||
|
experiments that no longer appear in a manifest for a period of at least
|
||
|
30 days. The rationale is that if an experiment disappears and reappears
|
||
|
from a manifest, the client will not have multiple opportunities to
|
||
|
generate a random value that satisfies the sampling criteria.
|
||
|
|
||
|
8. Incompatible version numbers
|
||
|
-------------------------------
|
||
|
|
||
|
If a client receives a manifest with a version number that it doesn't
|
||
|
recognize, it should ignore the manifest.
|
||
|
|
||
|
9. Usage of old manifests
|
||
|
-------------------------
|
||
|
|
||
|
If a client experiences an error fetching a manifest (server not
|
||
|
available) or if the manifest is corrupt, not readable, or compatible,
|
||
|
the client may use a previously-fetched (cached) manifest.
|
||
|
|
||
|
10. Updating XPIs
|
||
|
-----------------
|
||
|
|
||
|
If the URL or hash of an active experiment's XPI changes, the client
|
||
|
should fetch the new XPI, uninstall the old XPI, and install the new
|
||
|
XPI.
|
||
|
|
||
|
Examples
|
||
|
========
|
||
|
|
||
|
Here is an example manifest::
|
||
|
|
||
|
{
|
||
|
"version": 1,
|
||
|
"experiments": [
|
||
|
{
|
||
|
"id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",
|
||
|
"xpiURL": "https://experiments.mozilla.org/foo.xpi",
|
||
|
"xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",
|
||
|
"startTime": 1393000000,
|
||
|
"endTime": 1394000000,
|
||
|
"appName": ["Firefox", "Fennec"],
|
||
|
"minVersion": "28",
|
||
|
"maxVersion": "30",
|
||
|
"os": ["windows", "linux", "osx"],
|
||
|
"jsfilter": "function filter(context) { return context.healthReportEnabled; }"
|
||
|
}
|
||
|
]
|
||
|
}
|