browsergym-core

What's Changed

• Miniwob zoom by @recursix in https://github.com/ServiceNow/BrowserGym/pull/352
• prevent lxml to go to version 6 by @recursix in https://github.com/ServiceNow/BrowserGym/pull/356
• Fix_bbox_scale by @recursix in https://github.com/ServiceNow/BrowserGym/pull/357
• Make miniwob installation automatic. by @amanjaiswal73892 in https://github.com/ServiceNow/BrowserGym/pull/359
• Fill->type to trigger drop down by @recursix in https://github.com/ServiceNow/BrowserGym/pull/358

New Contributors

• @amanjaiswal73892 made their first contribution in https://github.com/ServiceNow/BrowserGym/pull/359

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.14.1...v0.14.2

What's Changed

• Some changes in preparation for agentlab's new ToolUseAgent and the new APIs by @TLSDC in https://github.com/ServiceNow/BrowserGym/pull/340
• Miniwob zoom by @recursix in https://github.com/ServiceNow/BrowserGym/pull/351

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.13.4...v0.14.1

What's Changed

browsergym-core

• Optional method AbstractBrowserTask.get_task_id() #281
• Fixed BrowserEnv parameter resizeable_window, now working as expected #281

browsergym-experiments

• Metadata column fix for visualwebarena #278

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.13.2...v0.13.3

What's Changed

browsergym-experiments

• Experiment traces can now be exported into the TapeAgents format #238
• Installs weblinx_browsergym as a dependency #261
• WA/VWA full instance reset will only issue a warning instead of crashing if not properly set-up #272
• New debug benchmark visualwebarena_tiny #271

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.13.1...v0.13.2

What's Changed

browsergym-experiments

• webarena / visualwebarena instance massage after reset #248 #250 #254 #259

browsergym-core

• Fixed gym warnings "obs not within observation space" #251
• Trace downgrades from INFO to DEBUG#252
• More robust env.close(), can now be used in a finally block even after reset failure #253
• Optional AbstractBrowserTask.teardown() method #255
• Browsergym's register_task() now supports both frozen, non-overrideable task_kwargs as well as overrideable default_task_kwargs arguments #255
• More robust frame marking #256 #258

browsergym-assistantbench

• Refactored AssistantBench mechanism for saving test predictions to JSON files #242

browsergym-webarena

• Relaxed playwright<1.40 restriction #257

browsergym-visualwebarena

• Relaxed playwright<1.40 restriction #257

Full Changelog

https://github.com/ServiceNow/BrowserGym/compare/v0.13.0...v0.13.1

What's changed

browsergym-core

• More robust frame marking with lenient last try #245
• Tasks can now choose their own locale and timezone_id #244

browsergym-experiments

• Pre-download WebLINX data in prepare_backend() #226
• Increase AssistantBench max_steps to 30 #244
• Add select_option to webarena / visualwebarena default action set #247

browsergym-visualwebarena

• Hide huggingface progress bar when downloading the visual evaluation model #241

browsergym-assistantbench

• Set locale="en-US" and timezone_id="America/New_York"

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.12.0...v0.13.0

Bugfixes

browsergym-experiments

• Fixes WebLINX task list #235
• Refactors experiment ID generation #236
• Adds VisualWebArena task dependencies #237 #239

browsergym-visualwebarena

• Fixes VisualWebArena tasks with visual validation (missing captioning_fn in evaluator) #240
• Adds a torch dependency (to run the captioning model) #240

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.11.3...v0.12.0

Bugfixes

• Fix duplicate depends_on in webarena metadata #228

Improvements

• Easier webarena / visualwebarena setup with (running nltk.download() at import time) #227
• More robust full_reset() for webarena / visualwebarena #230
• Removed ARIA extraction warnings #233
• New benchmark configuration webarena_tiny #232

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.11.2...v0.11.3

Bugfixes

• Add incomplete ExpResult.status #225

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.11.1...v0.11.2

New features

• Set max steps to 30 in webarena / visualwenarena benchmarks #214
• Benchmark dependency graph utilities #220
• Include nltk.download() in prepare_backend() for webarena / visualwebarena benchmarks #224

Bugfixes

• Rename benchmark after subset_from_split() #221
• ExpArgs.exp_dir sanitization #222
• get_step_info() bugfix #223

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.11.0...v0.11.1

New features

browsergym-experiments

• New weblinx benchmark 🎉 #208 (thanks @xhluca)
• New ExpResults.status() #219 (thanks @recursix)

browsergym-core

• New hide_all_bids option in flatten_dom_to_str() and flatten_axtree_to_str() #212 (thanks @imenelydiaker)
• Leaner Unicode() gym space #218

Bugfixes

• Benchmark.prepare_backends() fixes #209

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.10.2...v0.11.0

New features

• New Benchmark.prepare_backend() method #204

Bugfixes

• save_step_info() bugfix when obs==None (truncated episode due to None action) #207

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.10.1...v0.10.2

Minor changes

• train / test splits for WorkArena L2 and L3 tasks #203
• More fine-grained per-benchmark action sets #202

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.10.0...v0.10.1

New features

• New BrowserGym benchmark AssistantBench, packaged as browsergym-assistantbench. Thanks @oriyor ! #186

  import browsergym.assistantbench
  
  env = gym.make("browsergym/assistantbench.validation.12")
  env = gym.make("browsergym/assistantbench.test.42")
  

• Default train/test splits for all benchmarks

  miniwob = DEFAULT_BENCHMARKS["miniwob"]  # 125 tasks x 5 seeds
  miniwob_train = miniwob.subset_from_split("train")  # 62 tasks x 5 seeds
  miniwob_test = miniwob.subset_from_split("test")  # 63 tasks x 5 seeds
  

Breaking Changes

• Various updates and refactors to the new Benchmark class #197 #198 #199

Fixes

• Improved experiment logging #182

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.9.0...v0.10.0

New features

• Benchmarks with default config (tasks x seeds) and metadata #173 #191

  from browsergym.experiments import BENCHMARKS, Benchmark
  
  # make a custom benchmark
  benchmark = Benchmark(
    name="miniwob_click_test",
    high_level_action_set_args=HighLevelActionSetArgs(
      subsets=["bid"],
      multiaction=False,
      strict=False,
      retry_with_force=False,
      demo_mode="off",
    ),
    env_args_list=[
      EnvArgs(
        task_name="miniwob.click-test",
        task_seed=42,
        max_steps=5,
     )
    ],
  )
  
  # use a pre-existing benchmark
  miniwob = BENCHMARKS["miniwob_all"]()
  
  # use only a task subset
  miniwob_original = miniwob.subset_from_glob(
   column="miniwob_category", glob="original"
  )
  

• New playwright key modifier "ControlOrMeta" #187
• Global demo_mode flag #177

  import browsergym.core.action
  
  browsergym.core.action.set_global_demo_mode(True)  # boolean
  

Fixes

• Multi-tab actions fix #188

Full Changelog: https://github.com/ServiceNow/BrowserGym/compare/v0.8.1...v0.9.0

Fixes

browsergym-core

• fixed a bug with set-of-marks line drawing #184 #185

browsergym-core

• Breaking changes
  • goal refactor #110 obs["goal_object"] now replaces the old obs["goal_image_urls"] obs["goal"] is now deprecated the new goal_object now contains a list of openai-style messages, which can include an arbitrary mix of text and / or images.

browsergym-visualwebarena

• Breaking changes

  • goal refactor #110, the goal is now a list of openai-style messages with goal images as base64 image_url messages.

• Fixes

  • goal images are now self-hosted as part of the homepage #171 #165

browsergym-experiments

• Improvements
  • leaner trace files #169

other

• the legacy demo agent has been removed
• the basic demo agent has been leaned out and upgraded to support the new goal_object format #110
• other minor changes #166 #164

browsergym-core

• Depencency bump
  • playwright dependency bumped from playwright>=1.32,<1.40 to playwright>=1.39,==1.* (fixes problems with 1.32.1) #159

browsergym-experiments

• Bugfixes
  • #155 introduced a bug when agents returns action==None. This was fixed by #163

browsergym-experiments

• Breaking change

  • save package versions to a separate file package_versions.txt instead of the logs #152

• Bugfixes

  • minor fixes #154 #155

browsergym-core

• Bugfix
  • fixed bug in Frame has been detached catching #153

browsergym-core

• Improvements
  • ignoring Frame has been detached errors during unmarking #148 #147
  • more robust handling of missing frames in extract_merged_axtree #148 #146

browsergym-experiments

• New features
  • new field html_page: str in AgentInfo to enable agents to send an HTML page for visualization #13

browsergym-core

• Improvements
  • the 26 hard limit on max number of iFrames per page is removed #143
  • handling more Frame was detached errors during marking / unmarking #143

browsergym-experiments

• Breaking change
  • AbstractAgentArgs.agent_name becomes a field (instead of a property) #138

browsergym-experiments

• Bugfix
  • Fixed a bug due to an incompatibility with joblib when running pip list or pip freeze. Python distributions are now extracted using importlib and logged in the regular logfile #135

browsergym-core

• New features

  • new option tags_to_mark: Literal["all", "default_html"] for all BrowserGym environments #129 When set to "all", all page elements get marked and receive a BID. When set to "default_html", only standard HTML tags get marked and receive a BID (default behaviour).

• Improvements

  • the bid of div and span elements will now be available in the AXTree as well as the DOM #119 The aria-description field is now used in addition to the aria-roledescription as a fallback. This allows us to recover the bid of elements with generic role (div, span) for which aria-roledescription is not supported in the AXTree. Thanks @MurtyShikhar for noticing #110

browsergym-experiments

• New features
  • The output of pip list and pip freeze is now logged when an experiment is ran #130

browsergym-core

• New features
  • HighLevelActionSet has a new option retry_with_force #118 When activated, all Playwright actions will be retried with argument force=True if they fail. For example:

    def check(bid: str):
        ...
        if retry_with_force:
            try:
                elem.check(timeout=500)
            except Exception as e:
                elem.check(force=True, timeout=500)
        else:
            elem.check(timeout=500)
    

browsergym-experiments

• Improvements

  • More robust experiments loop #120

• New Features

  • Introducing experiment IDs (preparing for dependency graph between experiments) #121

browsergym-experiments

• New convenience namespace bgym, available whenever the browsergym-experiments package is installed

  import bgym
  
  bgym.Agent
  bgym.AgentInfo
  bgym.EnvArgs
  bgym.ExpArgs
  bgym.AbstractAgentArgs
  bgym.ExpResult
  bgym.StepInfo
  bgym.StepTimestamps
  bgym.AbstractActionSet
  bgym.HighLevelActionSet
  bgym.PythonActionSet
  

browsergym-visualwebarena

• New benchmark available!! #100 Bigs thanks to @ljang0

  import browsergym.visualwebarena
  
  env = gym.make("browsergym/visualwebarena.423")
  ...
  

  <img width="800" alt="VisualWebArena task" src="https://github.com/user-attachments/assets/0b78d4d1-7dc6-4b9f-ac2f-433fa30a1690">

browsergym-core

• New features
  • Tasks can now return a goal with both a message and images (in the form of URLs)

    class MyTask(AbstractBrowserTask):
      def setup(self, page: playwright.sync_api.Page):
        goal = {
          "message": "Where can I buy this car?",
          "image_urls": ["https://en.wikipedia.org/wiki/Electric_car#/media/File:Nissan_Leaf_2_(45992539055).jpg"]
        }
        info = {}
        return goal, info
    

  • Observations returned by BrowserEnv (reset(), step()) now have a new field for goal images

    obs, info = env.reset()
    # textual goal as before
    print(obs["goal"])
    # list of goal images (as urls, possibly using data:image/png;base64,... encoding)
    print(obs["goal_image_urls"])
    

  • Goal images are also sent to the chat, with a new user_image role

browsergym-webarena

• Breaking changes
  • Setup environment variables have been renamed to avoid clash with VisualWebArena #100

    export WA_SHOPPING=...
    export WA_SHOPPING_ADMIN=...
    export WA_REDDIT=...
    export WA_GITLAB=...
    export WA_WIKIPEDIA=...
    export WA_MAP=...
    export WA_HOMEPAGE=...
    

browsergym-experiments

• New features
  • new class AgentInfo #102
  • new AbstractAgentArgs methods prepare() and close() #106

browsergym-core

• Breaking changes

  • "infeas" action set now included by default #70 (report_infeasible(reason) action)
  • minimum Python requirement 3.9 #74

• New features

  • new option hide_all_children for the flatten_axtree_to_str() method #88
  • GPT-4o support for the demo agent #84

• Improvements

  • more robust _wait_dom_loaded() #77 : now catches all Playwright Errors (was the reason of some crashes with IFrames #76)
  • faster element marking #91 : thanks @manuel-delverme !
  • more robust Miniwob environment #75 : episode ends in failure if the agent navigates to a different page
  • better handling of infeasible actions for webarena #70

browsergym-experiments

• Minor bugfixes
  • #69

browsergym-core

• Refactors

  • minor changes in high-level actions description and examples

browsergym-experiments

• Bugfixes

  • action=None now recorded as truncated=True
  • webarena environments can be used again in the experiments loop (#61)

browsergym-core

• Refactors

  • minor error message refactors
  • module-based logging traces

• Bugfixes

  • overlay_som() bugfix divide by 0

browsergym-experiments

• New features

  • new field StepInfo.task_info

• Bugfixes

  • loop should be robust to crashes and interrupts

other

• missing requirements for demo_agent
• Makefile fixes

browsergym-core

• New features

  • Makefile :tada: (thanks @imenelydiaker )

    make install  # local pip install of all packages
    make install-demo
    make demo
    

  • option hide_bid_if_invisible: boolean = False for flatten_axtree_to_str() and flatten_dom_to_str()

• bugfixes

  • missing timeout=500 for high-level click action (was hanging for very long if unsuccessful)

browsergym-core

• New features

  • report_infeasible(reason) action Adds a message {"role": "infeasible", "message": reason} in the chat, and terminates the episode. Validations functions which expect the agent to report a task as infeasible can look for a message with the "infeasible" role, and check for its content if a specific reason is required.
    • new high-level action report_infeasible(reason) and action set "infeas"
    • new python primitive report_infeasible_instructions(reason)
    • new message role "infeasible" in the chat
    • new BrowserEnv argument terminate_on_infeasible: boolean = True

• Breaking changes

  • methods flatten_axtree_to_str() and flatten_dom_to_str() return more compact representations
    • properties som, clickable, visible instead of som="1", clickable="1", visible="1"
    • visible="0" is no more printed

• Bugfixes

  • minor fixes in browsergym.experiment.loop

browsergym-experiment

• Bugfixes

  • save the last observation in browsergym.experiments.loop

browsergym-experiments

• Breaking changes

  • Agent API refactor

    # former API
    Agent.action_mapping(action: str) -> str
    Agent.observation_mapping(obs: dict) -> Any
    
    # new API
    Agent.action_set: AbstractActionSet
    Agent.obs_preprocessor(obs: dict) -> Any
    

browsergym-core

• New features

  • a bunch of tools to run and records experiments in browsergym.experiments
  • chat messages now have a "timestamp" info
  • a new simple, lean demo_agent (previous demo agent will be moved somewhere else at some point, expect it to disappear soon)

• Bugfixes

  • duplicate bid fix (not perfect but should solve 99% of the cases)

browsergym-core

• minor fix: high-level action parser now properly handles keyword arguments in Python function calls (were converted to non-keyword arguments before)

browsergym-webarena

• minor fix: synced with latest webarena version (libwebarena=0.0.3), mostly typo fixes in task intents

browsergym-core

• New features

  • :tada: Set-of-Marks :tada: a new method is available to easily overlay element boxes and bid attributes on top of the screenshot, following ideas from WebVoyager and OSWorld

    from browsergym.utils.obs import overlay_som
    
    ...
    obs, info = env.reset()
    screenshot_with_som = overlay_som(obs["screenshot"], obs["extra_element_properties"], fontsize = 12, linewidth = 2, tag_margin = 2)
    

  • new high-level actions upload_file and mouse_upload_file

  • new field "extra_element_properties" in each observation. Contains a dict with bid keys, which gives the extra properties computed by browsergym for every element with a bid on the current page. Example:

    {
      "23": {
        "visibility": 0.6,  # float between 0 and 1
        "bbox": [56, 345, 12, 20],  # [x, y, width, height]
        "clickable": True,  # boolean
        "set_of_marks": False,  # boolean
    }
    

  • new set_of_marks property (computed with JS tag browsergym_set_of_marks), following WebVoyager implementation (boolean 0 or 1, whether element should be part of the set-of-marks overlay)

  • new clickable property, extracted from Chrome's DOMSnapshot's isClickable

  • new info fields "action_exec_start", "action_exec_timeout" and "action_exec_stop" after each env.step() call, useful for video editing

  • new resizeable_window parameter in BrowserEnv to switch between setting the viewport size via Chrome (previous behavior, resizeable window and viewport) or via Playwright (new default behaviour, viewport is not resizeable)

• Breaking changes

  • changed visibility tag in JS from browsergym_is_in_viewport (boolean 0 or 1) to browsergym_visibility_ratio (value between 0.0 and 1.0), extracted as the visibility extra property (see new features)
  • BrowserEnv parameters viewport (viewport size), slow_mo (pause between playwright calls) and timeout (default playwright timeout) are now provided by the task. They can still be set in the environment's constructor to override the value provided by the task, which will display a warning.
  • each task inheriting AbstractBrowserTask must now take a seed at instantiation (in constructor), instead of via the task.setup() method. This is also where each task should decide its desired browser setting by setting its attributes task.viewport, task.slow_mo and task.timeout (see point above)

• Refactors

  • bid-based high-level actions fail faster (500 ms)
  • shorter nested bids with alphabetical bids for iframes (21-53 -> a53)
  • fix mouse display position in demo mode (absolute -> fixed)
  • modern chat theme
  • refactored coordinate computation using Chrome's DOMSnapshot instead of JS, should be more robust to edge cases
  • refactored visibility computation using the IntersectionObserver API, should be more robust to edge cases
  • more robust frame marking, supports edge cases such as sandboxed iframes, and pdf viewers in <embed> tags

browsergym-miniwob

• fixed goal conversion to text in task browsergym/miniwob/click-menu-2