Action Space#

An action is specified by an action type (e.g., CLICK_COORDS) and the necessary fields for that action type (e.g., coords=[30, 60]).

Supported Action Types#

MiniWoB++ environments support the following action types:

Name	Description
`NONE`	Do nothing for the current step.
`MOVE_COORDS`	Move the cursor to the specified coordinates.
`CLICK_COORDS`	Click on the specified coordinates.
`DBLCLICK_COORDS`	Double-click on the specified coordinates.
`MOUSEDOWN_COORDS`	Start dragging on the specified coordinates.
`MOUSEUP_COORDS`	Stop dragging on the specified coordinates.
`SCROLL_UP_COORDS`	Scroll up on the mouse wheel at the specified coordinates.
`SCROLL_DOWN_COORDS`	Scroll down on the mouse wheel at the specified coordinates.
`CLICK_ELEMENT`	Click on the specified element using JavaScript.
`PRESS_KEY`	Press the specified key or key combination.
`TYPE_TEXT`	Type the specified string.
`TYPE_FIELD`	Type the value of the specified task field.
`FOCUS_ELEMENT_AND_TYPE_TEXT`	Click on the specified element using JavaScript, and then type the specified string.
`FOCUS_ELEMENT_AND_TYPE_FIELD`	Click on the specified element using JavaScript, and then type the value of the specified task field.

There are action types that perform similar actions (e.g., CLICK_COORDS and CLICK_ELEMENT). A common practice is to specify a subset of action types that the agent can use in the config, as described below.

Action Configs#

The list of selected action types, along with other configurations, can be customized by passing a miniwob.action.ActionSpaceConfig object to the action_space_config argument during environment construction.

An ActionSpaceConfig object has the following fields:

Key	Type	Description
`action_types`	`Sequence[ActionTypes]`	An ordered sequence of action types to include.
`screen_width`	`float`	Screen width. Will be overridden by the environment constructor.
`screen_height`	`float`	Screen height. Will be overridden by the environment constructor.
`coord_bins`	`tuple[int, int]`	If specified, bin the x and y coordinates to these numbers of bins. Mouse actions will be executed at the middle of the specified partition.
`scroll_amount`	`int`	The amount to scroll for scroll actions.
`scroll_time`	`int`	Time in milliseconds to wait for scroll action animation.
`allowed_keys`	`Sequence[str]`	An ordered sequence of allowed keys and key combinations for the `PRESS_KEY` action.
`text_max_len`	`int`	Maximum text length for the `TYPE_TEXT` action.
`text_charset`	`str` or `set[str]`	Character set for the `TYPE_TEXT` action.

Presets#

The following preset names can be specified in place of the ActionSpaceConfig object:

"all_supported": Select all supported actions, including redundant ones.
"shi17": The action space from (Shi et al., 2017) World of Bits: An Open-Domain Platform for Web-Based Agents.
"liu18": The action space from (Liu et al., 2018) Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration.
"humphreys22": The action space from (Humphreys et al., 2022) A data-driven approach for learning to control computers.

Adding "_mac_os" to the preset name will change the key modifiers in allowed_keys from Control to Meta.

Key combinations#

The PRESS_KEY action type issues a key combination via Selenium. Each key combination in the allowed_keys config follow the rules:

Modifiers are specified using prefixes “C-” (Control), “S-” (Shift), “A-” (Alternate), or “M-” (Meta).
Printable character keys (a, 1, etc.) are specified directly. Shifted characters (A, !, etc.) are equivalent to “S-” + non-shifted counterpart.
Special keys are inclosed in “<…>”. The list of valid names is specified in miniwob.constants.WEBDRIVER_SPECIAL_KEYS.

Example valid key combinations:"7", "<Enter>", "C-S-<ArrowLeft>".

Action Object#

The action passed to the step method should be a dict whose field inclusion depends on the selected action types in the config:

Key	Type	Description	Inclusion
`action_type`	`int`	Action type index from the `action_types` list in the config.	Always.
`coords`	`np.ndarray` of shape `(2,)`	Left and top coordinates. Depending on the `coord_bins` config, the values can be of type `int8` (binned) or `float32` (unbinned).	When any `*COORDS` action type is selected.
`ref`	`int`	Element `ref` ID. If no element has the specified `ref`, the action becomes a no-op.	When any `_ELEMENT` action type is selected.
`key`	`int`	Key index from the `allowed_keys` list in the config.	When the `PRESS_KEY` action type is selected.
`text`	`str`	Text to type.	When any `*_TYPE_TEXT` action type is selected.
`field`	`int`	Index from the task field list `obs["fields"]`. If the index is out of bound, no text will be typed.	When any `*_TYPE_FIELD` action type is selected.

For instance, if the config only contains action types CLICK_COORDS and PRESS_KEY, the action object can be

action = {
  "action_type": 0,     # CLICK_COORDS
  "coords": np.array([100, 50]),
  "key": 0,             # Ignored by the action CLICK_COORDS
}