Observations
Observation data passed to get_action.
Fields
Each step, the harness passes an Observation to your agent:
| Field | Type | Description |
|---|---|---|
goal | string | Task objective from the Trayn run. |
url | string | Current sandbox URL. |
axtree_txt | string? | Accessibility tree with BIDs. |
axtree_object | AXTreeObject? | Structured accessibility tree. |
extra_element_properties | Record<string, ExtraElementProperties>? | Visibility, bounding box, clickability, and markability by BID. |
screenshot | Uint8Array? | Screenshot when the task needs verifier evidence. |
last_action | string? | Previous action. |
last_action_error | string? | Execution error or verifier feedback from the previous action. |
action_history | string[]? | Semantic summaries of prior server-accepted steps. |
elapsed_time | number? | Seconds since task start. |
open_pages_urls | string[]? | Open page URLs. |
active_page_index | number? | Active page index. |
focused_element_bid | string? | Focused element BID. |
dom_txt | string? | Text DOM snapshot when available. |
pruned_html | string? | Pruned HTML snapshot when available. |
BIDs
BIDs identify elements across frames:
| BID | Meaning |
|---|---|
42 | Element 42 in the main frame. |
a5 | Element 5 inside iframe a. |
ab3 | Element 3 inside iframe b, nested inside iframe a. |
The SDK resolves the frame path before executing element actions.
Accessibility Tree
[42] button "Add to Cart", visible, clickable
[43] textbox "Search" value="laptop", visible, clickable
[a0] link "Help", visible, clickableExtra Element Properties
{
"42": {
visibility: 0.85,
bbox: [100, 200, 80, 32],
clickable: true,
setOfMarks: true,
},
}