mirror of
https://github.com/classilla/tenfourfox.git
synced 2024-05-28 20:41:33 +00:00
430 lines
15 KiB
ReStructuredText
430 lines
15 KiB
ReStructuredText
.. _experiments_manifests:
|
|
|
|
=====================
|
|
Experiments Manifests
|
|
=====================
|
|
|
|
*Experiments Manifests* are documents that describe the set of active
|
|
experiments a client may run.
|
|
|
|
*Experiments Manifests* are fetched periodically by clients. When
|
|
fetched, clients look at the experiments within the manifest and
|
|
determine which experiments are applicable. If an experiment is
|
|
applicable, the client may download and start the experiment.
|
|
|
|
Manifest Format
|
|
===============
|
|
|
|
Manifests are JSON documents where the main element is an object.
|
|
|
|
The *schema* of the object is versioned and defined by the presence
|
|
of a top-level ``version`` property, whose integer value is the
|
|
schema version used by that manifest. Each version is documented
|
|
in the sections below.
|
|
|
|
Version 1
|
|
---------
|
|
|
|
Version 1 is the original manifest format.
|
|
|
|
The following properties may exist in the root object:
|
|
|
|
experiments
|
|
An array of objects describing candidate experiments. The format of
|
|
these objects is documented below.
|
|
|
|
An array is used to create an explicit priority of experiments.
|
|
Experiments listed at the beginning of the array take priority over
|
|
experiments that follow.
|
|
|
|
Experiments Objects
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
Each object in the ``experiments`` array may contain the following
|
|
properties:
|
|
|
|
id
|
|
(required) String identifier of this experiment. The identifier should
|
|
be treated as opaque by clients. It is used to uniquely identify an
|
|
experiment for all of time.
|
|
|
|
xpiURL
|
|
(required) String URL of the XPI that implements this experiment.
|
|
|
|
If the experiment is activated, the client will download and install this
|
|
XPI.
|
|
|
|
xpiHash
|
|
(required) String hash of the XPI that implements this experiment.
|
|
|
|
The value is composed of a hash identifier followed by a colon
|
|
followed by the hash value. e.g.
|
|
`sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`
|
|
are the two supported hashing mechanisms. The hash value is the hex
|
|
encoding of the binary hash.
|
|
|
|
When the client downloads the XPI for the experiment, it should compare
|
|
the hash of that XPI against this value. If the hashes don't match,
|
|
the client should not install the XPI.
|
|
|
|
Clients may also use this hash as a means of determining when an
|
|
experiment's XPI has changed and should be refreshed.
|
|
|
|
startTime
|
|
Integer seconds since UNIX epoch that this experiment should
|
|
start. Clients should not start an experiment if *now()* is less than
|
|
this value.
|
|
|
|
maxStartTime
|
|
(optional) Integer seconds since UNIX epoch after which this experiment
|
|
should no longer start.
|
|
|
|
Some experiments may wish to impose hard deadlines after which no new
|
|
clients should activate the experiment. This property may be used to
|
|
facilitate that.
|
|
|
|
endTime
|
|
Integer seconds since UNIX epoch after which this experiment
|
|
should no longer run. Clients should cease an experiment when the current
|
|
time is beyond this value.
|
|
|
|
maxActiveSeconds
|
|
Integer seconds defining the max wall time this experiment should be
|
|
active for.
|
|
|
|
The client should deactivate the experiment this many seconds after
|
|
initial activation.
|
|
|
|
This value only involves wall time, not browser activity or session time.
|
|
|
|
appName
|
|
Array of application names this experiment should run on.
|
|
|
|
An application name comes from ``nsIXULAppInfo.name``. It is a value
|
|
like ``Firefox``, ``Fennec``, or `B2G`.
|
|
|
|
The client should compare its application name against the members of
|
|
this array. If a match is found, the experiment is applicable.
|
|
|
|
minVersion
|
|
(optional) String version number of the minimum application version this
|
|
experiment should run on.
|
|
|
|
A version number is something like ``27.0.0`` or ``28``.
|
|
|
|
The client should compare its version number to this value. If the client's
|
|
version is greater or equal to this version (using a version-aware comparison
|
|
function), the experiment is applicable.
|
|
|
|
If this is not specified, there is no lower bound to versions this
|
|
experiment should run on.
|
|
|
|
maxVersion
|
|
(optional) String version number of the maximum application version this
|
|
experiment should run on.
|
|
|
|
This is similar to ``minVersion`` except it sets the upper bound for
|
|
application versions.
|
|
|
|
If the client's version is less than or equal to this version, the
|
|
experiment is applicable.
|
|
|
|
If this is not specified, there is no upper bound to versions this
|
|
experiment should run on.
|
|
|
|
version
|
|
(optional) Array of application versions this experiment should run on.
|
|
|
|
This is similar to ``minVersion`` and ``maxVersion`` except only a
|
|
whitelisted set of specific versions are allowed.
|
|
|
|
The client should compare its version to members of this array. If a match
|
|
is found, the experiment is applicable.
|
|
|
|
minBuildID
|
|
(optional) String minimum Build ID this experiment should run on.
|
|
|
|
Build IDs are values like ``201402261424``.
|
|
|
|
The client should perform a string comparison of its Build ID against this
|
|
value. If its value is greater than or equal to this value, the experiment
|
|
is applicable.
|
|
|
|
maxBuildID
|
|
(optional) String maximum Build ID this experiment should run on.
|
|
|
|
This is similar to ``minBuildID`` except it sets the upper bound
|
|
for Build IDs.
|
|
|
|
The client should perform a string comparison of its Build ID against
|
|
this value. If its value is less than or equal to this value, the
|
|
experiment is applicable.
|
|
|
|
buildIDs
|
|
(optional) Array of Build IDs this experiment should run on.
|
|
|
|
This is similar to ``minBuildID`` and ``maxBuildID`` except only a
|
|
whitelisted set of Build IDs are considered.
|
|
|
|
The client should compare its Build ID to members of this array. If a
|
|
match is found, the experiment is applicable.
|
|
|
|
os
|
|
(optional) Array of operating system identifiers this experiment should
|
|
run on.
|
|
|
|
Values for this array come from ``nsIXULRuntime.OS``.
|
|
|
|
The client will compare its operating system identifier to members
|
|
of this array. If a match is found, the experiment is applicable to the
|
|
client.
|
|
|
|
channel
|
|
(optional) Array of release channel identifiers this experiment should run
|
|
on.
|
|
|
|
The client will compare its channel to members of this array. If a match
|
|
is found, the experiment is applicable.
|
|
|
|
If this property is not defined, the client should assume the experiment
|
|
is to run on all channels.
|
|
|
|
locale
|
|
(optional) Array of locale identifiers this experiment should run on.
|
|
|
|
A locale identifier is a string like ``en-US`` or ``zh-CN`` and is
|
|
obtained by looking at
|
|
``nsIXULChromeRegistry.getSelectedLocale("global")``.
|
|
|
|
The client should compare its locale identifier to members of this array.
|
|
If a match is found, the experiment is applicable.
|
|
|
|
If this property is not defined, the client should assume the experiment
|
|
is to run on all locales.
|
|
|
|
sample
|
|
(optional) Decimal number indicating the sampling rate for this experiment.
|
|
|
|
This will contain a value between ``0.0`` and ``1.0``. The client should
|
|
generate a random decimal between ``0.0`` and ``1.0``. If the randomly
|
|
generated number is less than or equal to the value of this field, the
|
|
experiment is applicable.
|
|
|
|
disabled
|
|
(optional) Boolean value indicating whether an experiment is disabled.
|
|
|
|
Normally, experiments are deactivated after a certain time has passed or
|
|
after the experiment itself determines it no longer needs to run (perhaps
|
|
it collected sufficient data already).
|
|
|
|
This property serves as a backup mechanism to remotely disable an
|
|
experiment before it was scheduled to be disabled. It can be used to
|
|
kill experiments that are found to be doing wrong or bad things or that
|
|
aren't useful.
|
|
|
|
If this property is not defined or is false, the client should assume
|
|
the experiment is active and a candidate for activation.
|
|
|
|
frozen
|
|
(optional) Boolean value indicating this experiment is frozen and no
|
|
longer accepting new enrollments.
|
|
|
|
If a client sees a true value in this field, it should not attempt to
|
|
activate an experiment.
|
|
|
|
jsfilter
|
|
(optional) JavaScript code that will be evaluated to determine experiment
|
|
applicability.
|
|
|
|
This property contains the string representation of JavaScript code that
|
|
will be evaluated in a sandboxed environment using JavaScript's
|
|
``eval()``.
|
|
|
|
The string is expected to contain the definition of a JavaScript function
|
|
``filter(context)``. This function receives as its argument an object
|
|
holding application state. See the section below for the definition of
|
|
this object.
|
|
|
|
The purpose of this property is to allow experiments to define complex
|
|
rules and logic for evaluating experiment applicability in a manner
|
|
that is privacy conscious and doesn't require the transmission of
|
|
excessive data.
|
|
|
|
The return value of this filter indicates whether the experiment is
|
|
applicable. Functions should return true if the experiment is
|
|
applicable.
|
|
|
|
If an experiment is not applicable, they should throw an Error whose
|
|
message contains the reason the experiment is not applicable. This
|
|
message may be logged and sent to remote servers, so it should not
|
|
contain private or otherwise sensitive data that wouldn't normally
|
|
be submitted.
|
|
|
|
If a falsey (or undefined) value is returned, the client should
|
|
assume the experiment is not applicable.
|
|
|
|
If this property is not defined, the client does not consider a custom
|
|
JavaScript filter function when determining whether an experiment is
|
|
applicable.
|
|
|
|
JavaScript Filter Context Objects
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
The object passed to a ``jsfilter`` ``filter()`` function contains the
|
|
following properties:
|
|
|
|
healthReportSubmissionEnabled
|
|
This property contains a boolean indicating whether Firefox Health
|
|
Report has its data submission flag enabled (whether Firefox Health
|
|
Report is sending data to remote servers).
|
|
|
|
healthReportPayload
|
|
This property contains the current Firefox Health Report payload.
|
|
|
|
The payload format is documented at :ref:`healthreport_dataformat`.
|
|
|
|
telemetryPayload
|
|
This property contains the current Telemetry payload.
|
|
|
|
The evaluation sandbox for the JavaScript filters may be destroyed
|
|
immediately after ``filter()`` returns. This function should not assume
|
|
async code will finish.
|
|
|
|
Experiment Applicability and Client Behavior
|
|
============================================
|
|
|
|
The point of an experiment manifest is to define which experiments are
|
|
available and where and how to run them. This section explains those
|
|
rules in more detail.
|
|
|
|
Many of the properties in *Experiment Objects* are related to determining
|
|
whether an experiment should run on a given client. This evaluation is
|
|
performed client side.
|
|
|
|
1. Multiple conditions in an experiment
|
|
---------------------------------------
|
|
|
|
If multiple conditions are defined for an experiment, the client should
|
|
combine each condition with a logical *AND*: all conditions must be
|
|
satisfied for an experiment to run. If one condition fails, the experiment
|
|
is not applicable.
|
|
|
|
2. Active experiment disappears from manifest
|
|
---------------------------------------------
|
|
|
|
If a specific experiment disappears from the manifest, the client should
|
|
continue conducting an already-active experiment. Furthermore, the
|
|
client should remember what the expiration events were for an experiment
|
|
and honor them.
|
|
|
|
The rationale here is that we want to prevent an accidental deletion
|
|
or temporary failure on the server to inadvertantly deactivate
|
|
supposed-to-be-active experiments. We also don't want premature deletion
|
|
of an experiment from the manifest to result in indefinite activation
|
|
periods.
|
|
|
|
3. Inactive experiment disappears from manifest
|
|
-----------------------------------------------
|
|
|
|
If an inactive but scheduled-to-be-active experiment disappears from the
|
|
manifest, the client should not activate the experiment.
|
|
|
|
If that experiment reappears in the manifest, the client should not
|
|
treat that experiment any differently than any other new experiment. Put
|
|
another way, the fact an inactive experiment disappears and then
|
|
reappears should not be significant.
|
|
|
|
The rationale here is that server operators should have complete
|
|
control of an inactive experiment up to it's go-live date.
|
|
|
|
4. Re-evaluating applicability on manifest refresh
|
|
--------------------------------------------------
|
|
|
|
When an experiment manifest is refreshed or updated, the client should
|
|
re-evaluate the applicability of each experiment therein.
|
|
|
|
The rationale here is that the server may change the parameters of an
|
|
experiment and want clients to pick those up.
|
|
|
|
5. Activating a previously non-applicable experiment
|
|
----------------------------------------------------
|
|
|
|
If the conditions of an experiment change or the state of the client
|
|
changes to allow an experiment to transition from previously
|
|
non-applicable to applicable, the experiment should be activated.
|
|
|
|
For example, if a client is running version 28 and the experiment
|
|
initially requires version 29 or above, the client will not mark the
|
|
experiment as applicable. But if the client upgrades to version 29 or if
|
|
the manifest is updated to require 28 or above, the experiment will
|
|
become applicable.
|
|
|
|
6. Deactivating a previously active experiment
|
|
----------------------------------------------
|
|
|
|
If the conditions of an experiment change or the state of the client
|
|
changes and an active experiment is no longer applicable, that
|
|
experiment should be deactivated.
|
|
|
|
7. Calculation of sampling-based applicability
|
|
----------------------------------------------
|
|
|
|
For calculating sampling-based applicability, the client will associate
|
|
a random value between ``0.0`` and ``1.0`` for each observed experiment
|
|
ID. This random value will be generated the first time sampling
|
|
applicability is evaluated. This random value will be persisted and used
|
|
in future applicability evaluations for this experiment.
|
|
|
|
By saving and re-using the value, the client is able to reliably and
|
|
consistently evaluate applicability, even if the sampling threshold
|
|
in the manifest changes.
|
|
|
|
Clients should retain the randomly-generated sampling value for
|
|
experiments that no longer appear in a manifest for a period of at least
|
|
30 days. The rationale is that if an experiment disappears and reappears
|
|
from a manifest, the client will not have multiple opportunities to
|
|
generate a random value that satisfies the sampling criteria.
|
|
|
|
8. Incompatible version numbers
|
|
-------------------------------
|
|
|
|
If a client receives a manifest with a version number that it doesn't
|
|
recognize, it should ignore the manifest.
|
|
|
|
9. Usage of old manifests
|
|
-------------------------
|
|
|
|
If a client experiences an error fetching a manifest (server not
|
|
available) or if the manifest is corrupt, not readable, or compatible,
|
|
the client may use a previously-fetched (cached) manifest.
|
|
|
|
10. Updating XPIs
|
|
-----------------
|
|
|
|
If the URL or hash of an active experiment's XPI changes, the client
|
|
should fetch the new XPI, uninstall the old XPI, and install the new
|
|
XPI.
|
|
|
|
Examples
|
|
========
|
|
|
|
Here is an example manifest::
|
|
|
|
{
|
|
"version": 1,
|
|
"experiments": [
|
|
{
|
|
"id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",
|
|
"xpiURL": "https://experiments.mozilla.org/foo.xpi",
|
|
"xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",
|
|
"startTime": 1393000000,
|
|
"endTime": 1394000000,
|
|
"appName": ["Firefox", "Fennec"],
|
|
"minVersion": "28",
|
|
"maxVersion": "30",
|
|
"os": ["windows", "linux", "osx"],
|
|
"jsfilter": "function filter(context) { return context.healthReportEnabled; }"
|
|
}
|
|
]
|
|
}
|