A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis

Introduction

Large language models (LLM) (Brown et al., 2020; Chowdhery et al., 2022; OpenAI, 2023) can solve variety of natural language tasks, such as arithmetic, commonsense, logical reasoning, question answering, text generation (Brown et al., 2020; Kojima et al., 2022; Wei et al., 2022), and even interactive decision making tasks (Ahn et al., 2022; Yao et al., 2022b). Recently, LLMs have also demonstrated success in autonomous web navigation, where the agents control computers or browse the internet to satisfy the given natural language instructions through the sequence of computer actions, by leveraging the capability of HTML comprehension and multi-step reasoning (Furuta et al., 2023; Gur et al., 2022; Kim et al., 2023).

However, web automation on real-world websites has still suffered from (1) the lack of pre-defined action space, (2) much longer HTML observations than simulators, and (3) the absence of domain knowledge for HTML in LLMs (Figure 1). Considering the open-ended real-world websites and the complexity of instructions, defining appropriate action space in advance is challenging. In addition, although several works have argued that recent LLMs with instruction-finetuning or reinforcement learning from human feedback improve HTML understanding and web automation accuracy (Furuta et al., 2023; Kim et al., 2023), their architectures are not always suitable to process real-world HTML documents; as presented in Figure 2, HTML tokens of real websites are much longer than those of simulators, and most LLMs have shorter context lengths than the average HTML tokens in real websites. It is prohibitively costly to treat such long documents as inputs directly, and even to adopt prior techniques for structured documents, such as text-XPath alignment (Li et al., 2021b) or text-HTML token separation (Wang et al., 2022a). To prioritize broad task generalization and model-size scaling, such domain knowledge for HTML codes is not applied in recent LLMs.

In this work, we introduce WebAgent, an LLM-driven autonomous agent that learns from self-experience to complete user instructions on real websites by combining canonical web actions in a program space (Figure 3). WebAgent (i) plans sub-instructions per step by decomposing natural language instructions, (ii) summarizes long HTML pages into task-relevant snippets based on sub-instructions, and (iii) acts via programming on real websites by grounding sub-instruction and HTML snippet into executable Python codes. We combine two LLMs to form WebAgent: Flan-U-PaLM (Chowdhery et al., 2022; Chung et al., 2022) for grounded code generation, and newly introduced HTML-T5, a domain-expert pre-trained language model, for task planning and conditional HTML summarization. HTML-T5 has an encoder-decoder architecture and is specialized to capture the structure – syntax and semantics – of long HTML pages better by adopting local and global attention encoder (Guo et al., 2022). It is self-supervisedly pre-trained with a mixture of long-span denoising objectives (Tay et al., 2022) on a large-scale HTML corpus from CommonCrawl. To ground language model agents into real websites, we introduce self-experience supervision, where the domain-expert language models are finetuned with self-generated demonstrations.

Existing LLM-driven agents often solve decision making tasks with a single LLM conditioned on different prompts per role (Kim et al., 2023; Sun et al., 2023; Zheng et al., 2023), which is, however, not enough for real-world tasks whose complexity is higher than that of simulators. The empirical evaluations reveal that our method incorporating self-bootstrapped specialist language models improves HTML understanding and grounding, and achieves better generalization than single LLM agent. In real-world web automation, WebAgent significantly increases the success rate by 50%, and error analysis emphasizes that coupling task planning with HTML summarization in specialized language models is essential for task success. Moreover, HTML-T5 not only works as a core module for WebAgent but also achieves strong results by itself on the web-based tasks. On MiniWoB++ (Liu et al., 2018; Shi et al., 2017), HTML-T5 achieves 18.7% higher success than previous language model agent (Gur et al., 2022) while also outperforming competitive baselines, such as naive local-global attention models (Guo et al., 2022) and its instruction-finetuned ones (Chung et al., 2022). On Mind2Web (Deng et al., 2023), an offline task planning dataset, HTML-T5 achieves SoTA performances among MindAct with FLan-T5-XL and GPT-4 (OpenAI, 2023). In summary, our key contributions are:

We propose WebAgent, integration of two modular LLMs under self-supervision for real-world web automation. The domain-expert language model handles planning and HTML summarization, and a generalist language model generates executable programs.

We newly introduce HTML-T5, pre-trained language models with local-global attentions and a mixture of long-span denoising on large-scale HTML corpus, which capture the syntax and semantics of HTML better.

WebAgent notably improves the success rate by over 50% in real websites. HTML-T5 itself outperforms prior language model agent by 18.7% in MiniWoB++, and realizes SoTA performance in Mind2Web while surpassing GPT-4.

Related Works

Web Automation Web automation is a sequential decision making task where agents manipulate browsers following given instructions (Shi et al., 2017), such as form filling (Diaz et al., 2013) or information retrieval (Adolphs et al., 2022) through the sequence of computer actions (Li et al., 2020; Mazumder & Riva, 2020; Shvo et al., 2021). Prior works have realized the web automation via reinforcement learning (Gur et al., 2019; Humphreys et al., 2022; Jia et al., 2019; Shaw et al., 2023), finetuned (Furuta et al., 2023; Gur et al., 2022) or prompted LLMs (Kim et al., 2023; Sun et al., 2023; Yao et al., 2022b; Zheng et al., 2023) on the simulated websites (Shi et al., 2017; Toyama et al., 2021; Yao et al., 2022a). However, there are still huge gaps between simplified simulators and real web environments; for instance, the average tokens for HTML pages are about 15 times larger (Figure 2), and pre-defined action space for specific websites is a strong assumption that may harm the generalization to out-of-distribution web pages or instructions.

MindAct (Deng et al., 2023) could be the most relevant work, where finetuned language model summarizes the raw HTML document into task-relevant snippets, and another model predicts the web actions in a multi-choice QA format. While MindAct also combines several language models, it has just adopted DeBERTa (He et al., 2021) and Flan-T5 (Chung et al., 2022) for summarization and actor modules, and evaluated it on the offline dataset. In contrast, we design HTML-T5, specialized for web-based tasks, to handle long HTML documents. WebAgent leverages HTML-T5 finetuned with self-experience for summarization and planning, and Flan-U-PaLM as a capable programmer, which enables it to generate open-ended web actions and to act on online real-world websites.

Program Synthesis In addition to common LLMs (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023), several works have proposed programming-focused language models (Chen et al., 2021a; Feng et al., 2020; Li et al., 2022; Wang et al., 2021) and their benchmarks (Austin et al., 2021; Hendrycks et al., 2021a; Lu et al., 2021). Another line of work has investigated the tool augmentation of LLMs (Parisi et al., 2022) by decoding API calls (Schick et al., 2023) or Python snippets to be parsed with the interpreter (Gao et al., 2023). Most works deal with the program synthesis on the static dataset, except for the attempts in robotics (Liang et al., 2023) and game (Trivedi et al., 2022; Wang et al., 2023a), where LLMs output Python or JavaScript snippets to command the agents. Similarly, we leverage the ability of code generation as an open-ended action space for web-based agents to manipulate the real website, and demonstrate LLMs can sequentially decode Python selenium codes considering the given sub-instructions and HTML in the prompts.

See extended related works on document understanding and LLM for task planning in Appendix B.

WebAgent

WebAgent is composed of interactions between HTML-T5, a domain-expert language model, which predicts the sub-instruction for the next-step program and conditionally summarizes long HTML documents, and Flan-U-PaLM (Chowdhery et al., 2022; Chung et al., 2022), an instruction-finetuned LLM for grounded program synthesis (Figure 3). In contrast to a single LLM conditioned on different prompts per role, such a modular approach can deal with real-world tasks better. Moreover, to align WebAgent with real websites, we introduce self-experience supervision to ground the agent into real-world tasks. We describe the details of each component in the following sections, and provide the example workflow in Appendix D.

Previous works demonstrate that generalist LLMs, such as T5 (Raffel et al., 2020), Flan-T5 (Chung et al., 2022), and InstructGPT (Ouyang et al., 2022), have a capability of manipulating the web environments (Shi et al., 2017) with great HTML comprehension (Furuta et al., 2023; Gur et al., 2022; Kim et al., 2023). However, they have not fully leveraged the HTML-specific inductive bias on syntax and semantics considered in the prior specialist transformer models (Li et al., 2021b; Wang et al., 2022a; Zhao et al., 2022). We here introduce HTML-T5, a pre-trained encoder-decoder language model, by interpolating the generalist and specialist nature of language models to solve downstream HTML-based web automation tasks efficiently. HTML-T5 processes HTML documents in a text-to-text manner, and leverages local and global attentions (Ainslie et al., 2020; Guo et al., 2022) in the encoder to handle the hierarchical structure of long HTML inputs. We pre-train it with large-scale HTML corpus curated from CommonCrawl on a mixture of long-span denoising objectives (Raffel et al., 2020; Tay et al., 2022), and finetune it for each downstream task. Especially, for WebAgent, we employ self-experience supervision to align the model with real websites.

Model Architecture In contrast to natural language texts, HTML documents have an explicit hierarchy from the tree structure; the relation of each element (e.g. , , ) and its attributes (e.g. class, label, id) are often defined locally, and those are iteratively integrated globally (e.g. ,

,

). To capture such a hierarchical structure of HTML, we adopt local and global attention mechanisms (Guo et al., 2022), instead of common dense attention (Raffel et al., 2020; Vaswani et al., 2017). Local attention restricts each token to only attend to neighboring tokens to the left and right. Transient global attention allows each input token to attend to beyond nearby tokens, by dividing the input sequence into blocks of tokens and computing global tokens with summation and normalization of the embeddings of every token in the block. Figure 4 describes the concepts of HTML-T5; leaf elements in HTML (green) could be processed by local attention, and internal elements (purple) could be compressed into transient global attention, which naturally fit the hierarchical syntax of HTML documents. We leverage the implementation of LongT5 (Guo et al., 2022) as base architectures using dense attention in the decoder.

Pre-Training with Mixture of Long-Span Denoising The performance of language models in downstream tasks highly depends on the knowledge learned in pre-training. To incorporate further inductive bias on HTML into scalable language models, we perform self-supervised pre-training with large-scale HTML corpus. We here employ span denoising objective, where we mask the input texts with random spans of tokens (following normal distributions with mean span length $\mu$), and the models take all other tokens from the documents as inputs to predict corrupted spans (Ainslie et al., 2023; Raffel et al., 2020; Tay et al., 2022). To deal with the sparsity of contents tokens in HTML documents, we introduce a mixture of long-span denoising objective, by masking input tokens with longer mean span lengths than popular value for natural language (e.g. $\mu=3$). Such a shorter mean span length only masks less meaningful chunks, such as (Figure 4), which might not be helpful for LLMs to capture the syntax and semantics of HTML. In contrast, longer span can contain more semantically meaningful chunks, such as . We empirically find $\mu\in\{8,64\}$ is the optimal mixture (Section 4.2).

We adopt 4096 input sequence length and 910 output sequence length during the denoising pre-training. In total, 15% of input tokens are randomly masked. For the dataset, we prepare 100 WARC files (April 2019) from CommonCrawl, and pre-process the raw HTML by removing non-Unicode and alphanumeric documents and extracting subtrees around elements that have for attribute, to reduce the noise in training corpus, which results in about 3.41M examples. We train the models with 100K iterations following other pre-training strategies for T5 families (Chung et al., 2022; Lester et al., 2021). See Appendix C for further details.

2 Alignment with Self-Experience Supervision

Another bottleneck for building real-world web automation agents is collecting demonstrations to align LLM with real websites. Humans could perform instruction following on real websites easily, but it is infeasible to manually annotate all the instruction decomposition, snippet extractions, and executable programs. To reduce such a burden, we introduce a self-experience supervision, where the language model agents learn from the experience that they themselves face on real websites with minimal human intervention. We first prepare the templates of instructions. The scripted agents procedurally parse instructions into the sequence of sub-instructions, regular-expression-based retrieval specifies the elements to be summarized, and conditioned on those, Flan-U-PaLM executes web actions via program synthesis. The generated demonstrations following the steps above may result in success and failure, but the success criteria for real-world tasks is hard to automate. Instead, to filter the experience, we leverage the environmental feedback that can remove critical failures; for instance, the program execution errors, retriever errors, and clearly wrong prefix of URL (Ni et al., 2023). Our WebAgent aligns domain-expert language models, HTML-T5, with those self-collected real-world experiences via finetuning (Wang et al., 2022b). This self-supervision process realizes the generalization and alignment of language model agents to challenging real-world tasks.

Finetuning for Planning and Summarization We align language models to perform closed-loop planning with a sequence of sub-instructions and to summarize long HTML documents into concise snippets relevant to the current plan. As a core module of WebAgent, HTML-T5 finetuned with self-generated demonstrations takes task instructions (e.g. please search 2 bedroom and 2+ bathroom houses in new york, ny with a max price of $7500 on real estate website), sub-instruction histories (e.g. go to real estate website, type in new york, ny into search, click on search, click on price, click on max rent), and raw HTML as inputs. Then, it predicts the next sub-instruction (e.g. type in 7500 into max rent) and the corresponding data-ref attributes to extract the snippet with XPath instead of naively decoding the raw snippet. In the later experiments in Section 4.1, we will demonstrate that linking HTML summarization into sub-instruction prediction is important for real-world web automation performance.

3 Grounded Program Synthesis

Web automation on real-world websites suffers from the open-ended action space, compared to the simplified simulators (Shi et al., 2017; Yao et al., 2022a). Unlike previous works (Gur et al., 2019; Humphreys et al., 2022; Jia et al., 2019; Liu et al., 2018), real-world web agents could not pre-define a categorical action space to specify which elements on the websites they should interact. To overcome such an open-domainness, we introduce act via programming paradigm in web automation by leveraging the capability of LLMs on conditional code generation (Chen et al., 2021a; Liang et al., 2023). Given a few canonical examples for program generation, next sub-instruction, and extracted HTML snippet from HTML-T5, Flan-U-PaLM (Chowdhery et al., 2022; Chung et al., 2022) with 540B parameters decodes an executable Python program (Figure 3) using Selenium WebDriver, a library for browser automation. Such a conditional program synthesis demands that LLMs are capable enough to not only generate the code following natural language instructions, but also understand the semantics and functionality of HTML elements. We provide several Python snippet examples generated by Flan-U-PaLM as follows (we treat sub-instructions as comments in the script):

Experimental Results

To study how a modular combination of LLMs under self-supervision enables real-world web automation by overcoming open-endedness and long context documents, we execute instruction-following tasks on real websites (Section 4.1). In Appendix E, we also test WebAgent on WebSRC (Chen et al., 2021b), a static HTML comprehension benchmark, compared to prior transformer models specialized for structured documents (Li et al., 2021b; Zhao et al., 2022). In addition, we quantify the performance of HTML-T5 itself on simulated web benchmark, MiniWoB++, and offline task planning benchmark, Mind2Web (Section 4.2).

Evaluation Methodology We first evaluate WebAgent with the real-world navigation performance under human supervision, at real estate website (a platform for housing), social media website (a network of communities), and map website. These three websites have different properties. real-estate requires long-horizon planning (about 20 steps per episode) for complex form-filling with a few page transitions (at least 2 pages), and social-media needs shorter plans (about 10 steps per episode) with many page transitions (at least 4 pages) by selecting appropriate hyperlinks on the page. map is the easiest domain with shorter plans and a few page transitions. WebAgent receives natural language instructions (e.g. Can you search for a studio bedroom, 1+ bathroom apartments in oroville, ca for corporate housing on real estate website?, or Could you present the most new thread of Python community filtered by Tutorial tag on social media website?), and acts via planning, summarizing by HTML-T5, and then programming by Flan-U-PaLM (Figure 5). Through the self-experience supervision process, we curate 260 episodes on real estate website, 230 episodes on social media website, and 410 episodes on map website to finetune HTML-T5.

We prepare 20 different natural language instructions (see Appendix F for the full list), and measure the success rate and score for the evaluation. The score represents the percentage of required attributes covered during the episode (Yao et al., 2022a); for instance, (1) apartments for (2) corporate housing with (3) studio bedroom and (4) 1+ bathroom located in (5) oroville, ca, can be specified in the instruction. When the agents could search the housing satisfying (1), (2), (5) and not (3), (4), the score is 60 ($=100\times 3/5$). If the agents achieve 100 score, that episode will mark as success.

Results For comparison, we prepare three baselines, consisting of language model modules and a single LLM conditioned on different prompts per role, such as Flan-U-PaLM (Chung et al., 2022), that with a planning language model (Flan-U-PaLM+P), and that with a summarization language model (Flan-U-PaLM+S). If they do not use language model modules, prompted Flan-U-PaLM plans in an open-loop manner (Plan: ✗), and regular-expression-based retrieval summarizes given raw HTML (Sum: ✗). Table 1 shows that by leveraging planning and summarization language model modules, WebAgent achieves best 65% success and 87.6% score on real-estate, 70% success and 85.8% score on social-media, and 80% success and 93.8% score on map, significantly outperforming single Flan-U-PaLM, or with partial language model modules (most of those achieve about 10 - 30% success). This result suggests that self-experience supervision notably improves the performance, and closed-loop planning grounded on HTML observations via finetuned domain language models is more suitable for open-ended web automation than open-loop planning with few-shot LLMs. This trend is remarkable in real-estate (even Flan-U-PaLM+P achieves 50% success), where the longer planning horizon is needed to fulfill instructions. We also observe that coupling sub-instruction prediction with HTML summarization in language model modules plays a critical role in task success. The development of more capable planning modules to decompose the given instructions adaptively and accurately could help WebAgent improve the performance further.

Error Analysis We also analyze the reason of failures by categorizing them into programming, planning, and summarization errors (Table 1). Programming error does not satisfy the given sub-instructions or HTML snippet. Planning error predicts sub-instructions conflicting with user instructions, and summarization error fails to extract the relevant HTML snippets for given sub-instructions. From the website perspective, the failures on real-estate concentrate in planning because of its long-horizon nature. map also fails in planning when confusing starting point and destination. In contrast, social-media tends to fail in programming due to the ambiguous sub-instructions or summarization including redundant hyperlinks, which results in transiting wrong pages or clicking unexecutable elements. From the method perspective, WebAgent often fails in planning by predicting incorrect sub-instructions (for instance, in real-estate, WebAgent generates incorrect plans in 70% of failure episodes), while other baselines more fail in programming or summarization steps. This observation indicates that, through the self-experience supervision, the ratio of programming and summarization errors has decreased while the fundamental difficulty of planning, which requires consistent and accurate prediction over long horizon without error accumulation, still remains.

2 Ablation of HTML-T5

In addition to the evaluation as WebAgent system, we extensively examine HTML-T5 about (i) the generalization to other websites with Mind2Web dataset (Deng et al., 2023), (ii) the performance on MiniWoB++, a standard web automation benchmark (Liu et al., 2018; Shi et al., 2017), and (iii) its architecture and pre-training objective. We adopt 16K tokens for the context window unless otherwise mentioned. We also evaluate HTML-T5 on the pre-training dataset and model initialization, offline task planning with self-generated real-estate traces, and description generation benchmark (Gur et al., 2022) to test HTML understanding on static dataset in Appendix H.

Mind2Web (Deng et al., 2023) is an action-annotated real-world dataset with over 2K instructions collected from 137 websites. It provides action prediction tasks that measure the generalization of LLMs across the tasks, websites, and their domains (e.g. travel, shopping). Conditioned on the top-50 HTML snippet candidates, task instruction, and action history, LLMs should predict the next step action by choosing a target element to interact with in a multi-choice QA format and generating the operation such as click, type, or select option. We finetune HTML-T5-XL with the training dataset. The performance is evaluated with element accuracy, operation F1, and step success rate that cares for both element and operation correctness. Table 3 reveals that HTML-T5 significantly outperforms baselines with Flan-T5-XL or GPT-4 (OpenAI, 2023) across task/website/domain generalization, which increases element accuracy by 20-30%, operation F1 by 5-10%, and step success rate by 20-30%. This highlights that HTML-T5 can handle real-world web automation tasks better and shows generalization beyond our real-world evaluation with 3 websites.

MiniWoB++ We here evaluate HTML-T5 on simulated web environments, MiniWoB++ with 56 tasks by running 100 evaluation episodes per task. We finetune HTML-T5 with 12K human demonstrations (Liu et al., 2018), and compare the average success rate to prior supervised-learned agents (Gur et al., 2022; Humphreys et al., 2022), LongT5, and its instruction-finetuned variants (Chung et al., 2022; Furuta et al., 2023) we prepared We finetune LongT5 models with Flan dataset released by Chung et al. (2022). As a sanity check, we test them on representative reasoning and summarization tasks (see Appendix I). . Table 4 shows that HTML-T5-XL significantly outperforms WebN-T5, the prior best model, by 18.7%. Notably, we demonstrate HTML-denoising consistently improves the performance on top of LongT5 in all the model sizes, better than instruction-finetuning introduced in prior work (Furuta et al., 2023). Furthermore, we finetune HTML-T5-XL with 347K demonstrations from Furuta et al. (2023), which performs better than 11B-parameter Flan-T5-XXL even with 3B parameters, achieving 85.6% success. These prove we successfully incorporate domain knowledge on HTML comprehension for web automation into pre-trained language models.

Architecture and Objective We hypothesize that local and global attention mechanisms can capture the hierarchical structures of HTML documents better than dense attention. We compare the web automation performance among 56 MiniWoB++ tasks (Gur et al., 2022), by finetuning HTML-T5 with public 12K-episode dataset (Liu et al., 2018). We adopt 2048 and 4096 tokens as input length and prepare Base-size architectures. Table 2 (left) reveals that the combination of local and global attentions achieves the superior success rate by over 18% compared to the instruction-finetuned dense attentions (Chung et al., 2022; Raffel et al., 2020) and local attention only. Surprisingly, local attention only still surpasses the dense attention by about 9%, which suggests local relation between elements and attributes in HTML are essential for web tasks.

As for pre-training objective in Table 2 (right), HTML-denoising generally improves the performance on offline task planning on real estate website and MiniWoB. Especially, using only longer span lengths ($\mu\in\{8,64\}$) outperforms other choices, including the popular configuration in natural language domain ($\mu\in\{3,8,64\}$ + Prefix LM objective), which can reduce the less meaningful prediction from shorter spans (e.g. $\mu=3$), and inject the structural bias of HTML into language models better. See Appendix H.2 for further results with model scaling.

Discussion and Limitation

Modular Approach with Specialist Language Models We demonstrate it is beneficial to divide web automation into planning, HTML summarization, and code generation, and to combine domain-expert language models aligned with self-experience data. Such modular approaches have also been adopted to support the inference of LLMs (Xu et al., 2023), multimodal tasks (Zeng et al., 2022), and robotics (Ahn et al., 2022), which, however, might cause additional computational costs and latency.

Broad Generalization across the Internet Because open-loop planning with prompted Flan-U-PaLM achieves at most 10 - 30% success, we have demonstrated that self-experience supervision on real websites is essential for planning modules. As we demonstrated in Mind2Web, our method could generalize across the internet if we have enough data. It would be expected to collect demonstrations at scale and align larger domain-expert models with them in future works.

Feedback for Program Synthesis We leverage Flan-U-PaLM with 540B parameters, as a capable program synthesis module via few-shot prompting. Such a large model, however, makes it challenging to reflect the feedback about the errors in generated code, compared to smaller models. We leave it as future direction to incorporate the feedback for program synthesis into larger language models.

Evaluation for Real-world Web Automation Beyond the simulated web environments (Shi et al., 2017; Yao et al., 2022a), we have exhibited WebAgent can follow given complex and sometimes ambiguous instructions on real estate, social media and map websites. On the other hand, it is costly to evaluate the performance of autonomous agents in the real world. Automated evaluation with minimal human intervention would be helpful for the scalable development of real-world web agents.

Conclusion

We build a system for real-world web automation, combining HTML-T5 for planning and HTML summarization and Flan-U-PaLM for grounded program synthesis. Our proposed WebAgent achieves around 70-80% success on real websites via self-experience supervision, outperforming single LLM approach by over 50%, which suggests dividing the sequence of sub-problems with multiple language models can increase the entire task success. We also propose a scalable recipe for HTML-specialized language models where we train local and global attention mechanisms with a mixture of long-span denoising objectives to capture the hierarchical structures of HTML documents. HTML-T5 not only plays an essential role in WebAgent but also can achieve the best results on a variety of HTML-based benchmarks such as Mind2Web and MiniWoB++. We hope our work contributes to getting us one-step closer to the practical deployment of autonomous web agent systems.

We thank Heiga Zen, Yingjie Miao, Yusuke Iwasawa, Joshua Ainslie, Santiago Ontanon, Quoc V. Le, Zoubin Ghahramani, Jeff Dean, Tris Warkentin for the supports and advises on this work. HF was supported by JSPS KAKENHI Grant Number JP22J21582.

References

Appendix

Appendix A Broader Impacts

The development of autonomous agents should consider the security and safety aspects. In the real website evaluation, we have carefully conducted the experiments under human supervision in case undesired behaviors happen. We use Selenium WebDriver https://www.selenium.dev/, a popular library for browser automation, and limit the access per second not to stress the server. We have anonymized the real websites we tested on for safety and privacy concerns.

Appendix B Extended Related Works

Document Understanding Understanding structural documents has been a practical challenge for transformer-based language models. Prior works employ layout-informed tokens (Xu et al., 2019) or even multimodal tokens from visual inputs (Appalaraju et al., 2021; Li et al., 2021a; c). Especially, for the documents written in markup languages, text-XPath alignment (Li et al., 2021b), token separation between text and HTML (Wang et al., 2022a), or extra topological information of HTML (Zhao et al., 2022) are proposed to leverage their syntax better. On the other hand, such a domain knowledge conflicts with recent generalist and scaling trends around LLMs (Anil et al., 2023; OpenAI, 2023). Because web agents require the instruction-conditioned HTML understanding, it also would be desirable to reconcile specialist aspects for HTML documents with generalist capabilities for natural language tasks. In this work, we design HTML-T5 to incorporate the structural bias of HTML by combining local-global attention for the encoder and a mixture of long-span denoising, while it can solve instruction-following better in downstream web-based tasks.

LLM for Task Planning The prior knowledge of commonsense in LLMs has allowed us to leverage them for a variety of task planning. For instance, Huang et al. (2022) propose LLM agent that generates natural language plans in an open-loop manner. Nottingham et al. (2023) and Wang et al. (2023b) perform sequential closed-loop planning on MineCraft. Singh et al. (2022) decode robotic plans with pythonic text, and several works incorporate planning definition and domain language into the outputs (Liu et al., 2023; Silver et al., 2023; Valmeekam et al., 2023). On the other hand, our WebAgent leverages finetuned specialist language models and performs closed-loop planning coupled with HTML summarization by decomposing given instructions. We empirically prove that our system is superior to open-loop planning with a single generalist LLM with prompting.

Appendix C Implementation Details of HTML-T5

We use the implementation of local and global attentions released by Guo et al. (2022) https://github.com/google-research/longt5. Following Guo et al. (2022), we set the local radius to $r=127$, and block size for transient global attention to $k=16$. For the pre-training objective, similar to Tay et al. (2022), we construct the mixtures and then use long mean span lengths: $\mu\in\{8,64\}$, and all the denoising ratio (percentage of masked tokens in the input sequence) is set to 0.15. We adopt 4096 input sequence length and 910 output sequence length during the pre-training. The batch size for training is set to 128. We train the models with 100K iterations following other pre-training strategies for T5 families (Chung et al., 2022; Lester et al., 2021). We leverage SeqIO (Roberts et al., 2022) and T5X (Roberts et al., 2022) library to manage the training pipeline. We also use SentencePiece (Kudo & Richardson, 2018) with 32K tokens from C4 dataset (Raffel et al., 2020) as a tokenizer. During the downstream finetuning, we adopt 16K tokens for the context window unless otherwise mentioned. We have used cloud TPU-v3, which has a 32 GiB HBM memory space, with 128 cores for the experiments.

For the dataset, we prepare 100 WARC files (April 2019) from CommonCrawlhttps://commoncrawl.org/, and pre-process the raw HTML by removing non-Unicode and alphanumeric documents and extracting subtrees around elements that have for attribute, to reduce the noise in training corpus, which results in about 3.41M examples (Table 5).

Appendix D WebAgent Example Flow in real-estate website

Appendix E WebSRC: Static HTML Comprehension

To emphasize the advantage of our modular approach, we test WebAgent on a static website comprehension benchmark, WebSRC (Chen et al., 2021b), which is a contextual QA dataset with HTML documents. The questions require an understanding of the spatial and logical structure of websites, and the answers are either text span on HTML or yes/no. For the comprehensive evaluation, WebSRC has three different types of websites, KV, Comparison, and Table. KV task is a value extraction from the attribute key. Comparison task has several entities with the same attributes. Table task requires a structural understanding with header columns and values in the row. We finetune HTML-T5 for snippet extraction to predict data-ref corresponding to the answer and use dev set for the evaluation.

As did in real-world web automation, HTML-T5 first predicts data-ref attribute of task-relevant snippet from the input HTML document. To make sure there is enough context, we extract the snippet from the predicted element to the two-level-up via XPath. If it exceeds the context length of Flan-U-PaLM, we limit it into parent elements. If it still does not work, we truncate the end of extracted snippet to fit within the token budget. Because snippet extraction in table structure often loses the context to solve question-answering, we just truncate HTML documents for Table tasks. Flan-U-PaLM predicts the answers seeing 5-shot examples.

As shown in Table 6, single LLM, such as Flan-U-PaLM or HTML-T5, has struggled to the limited context length or model capacity. In contrast, WebAgent, our LLM-collaborative approach, enhances the performance from both single generalist and specialist LLMs, and shows competitive results with strong baselines. This demonstrates that modular LLMs work complementarily to each other. Figure 7 presents the performance comparison on different types of websites (KV, Comparison, Table) among MarkupLM (Li et al., 2021b), TIE (Zhao et al., 2022), and WebAgent. WebAgent is better at Comparison tasks, but inferior to structural understanding for KV and Table tasks, compared to other baselines, which suggest that generalist LLMs are still not suitable for recognizing structural data such as table.

Appendix F List of Language Instructions for Real-world Web Automation

can you search for a studio bedroom, 1+ bathroom houses in escondido, ca for corporate housing and price less than 12100 on real estate website.

can you find me a studio bedroom, 1+ bathroom townhomes in hollywood, ca and price less than 14600 on real estate website.

can you search for a studio bedroom, 1+ bathroom condos in inglewood, ca for senior housing and price less than 8700 on real estate website.

I would like to search for a studio bedroom, 1+ bathroom houses in compton, ca and price more than 1200 for corporate housing on real estate website.

can you search for a studio bedroom, 1+ bathroom apartments in oroville, ca for corporate housing on real estate website.

find me a studio bedroom, 1+ bathroom houses in modesto, ca on real estate website.

can you search for a studio bedroom, 1+ bathroom condos in redwood city, ca for student and price more than 1900 on real estate website.

find me a 1 bedroom condos in santa clara, ca and price between 1600 and 7400 on real estate website.

find me a 1 bedroom, 3+ bathroom apartments in martinez, ca with min price 1800 on real estate website.

can you find me a 2 bedroom, 2+ bathroom townhomes in concord, ca and price more than 600 on real estate website.

can you find me a studio bedroom, 2+ bathroom apartments in san diego, ca and price less than 9300 on real estate website.

find me a studio bedroom houses in novato, ca and price between 1500 and 6700 on real estate website.

can you find me a studio bedroom, any bathroom townhomes in petaluma, ca and price more than 1000 on real estate website.

search for a 1 bedroom apartments in modesto, ca and price more than 1000 on real estate website.

find me a 1 bedroom, 2+ bathroom apartments in watts, ca for senior housing less than 6300 on real estate website.

can you find me a 1 bedroom houses in victorville, ca that have dog friendly, furnished and price more than 700 on real estate website.

I need a 2 bedroom, any bathroom condos in inglewood, ca and price more than 1000 on real estate website.

find me a 2 bedroom, 2+ bathroom apartments in livermore, ca on real estate website.

can you find me a 2 bedroom apartments in santa clara, ca that has parking and price less than 10300 on real estate website.

can you search for a 2 bedroom condos in oakland, ca on real estate website.

Show me the most hot thread in r/google at social media website.

Can you point out the most hot thread in r/learnpython at social media website.

Could you check the 1st hot thread in r/artificial at social media website.

Can I check the most hot thread in Taiwan on social media website.

Show me the first new thread in r/facebook at social media website.

Present the most new thread of r/Python filtered by Tutorial flair on social media website.

Could you check the 1st new thread in r/facebook at social media website.

I want to read the 1st hot thread from r/Python tagged as Daily Thread at social media website.

Present the most hot thread of r/google filtered by Info | Mod Post flair on social media website.

Show me the most new thread in r/learnmachinelearning filtered by Help flair at social media website.

Can you point out the first hot thread in r/deeplearning at social media website.

Could you check the 1st hot thread in r/machinelearningnews at social media website.

Present the most hot thread of r/artificial filtered by News flair on social media website.

Please find me the first hot thread in r/facebook at social media website.

Present the most new thread of r/machinelearningnews filtered by Startup News flair on social media website.

Show me the most hot thread in r/artificial filtered by AI Art flair at social media website.

Could you check the first new thread in r/facebook at social media website.

I want to read the most top thread from r/google tagged as Info | Mod Post at social media website.

Show me the most new thread in r/startups filtered by Share Your Startup flair at social media website.

Could you check the 2nd new thread in r/facebook at social media website.

Show me the way from San Jose to Mountain View by 2nd Cycling at map website.

Please show me the way from The Painted Ladies to San Francisco Zoo with 3rd Best option at map website.

Could you tell me the path from California Academy of Sciences to de Young Museum by 1st Transit at map website.

Could you tell me the way from Union Square to The Painted Ladies with 2nd Cycling option at map website.

Please present the way from Chappell Hayes Observation Tower to San Jose with 2nd Walking option at map website.

Please present the path from Jack London Square to Emeryville by 2nd Cycling at map website.

I’d like to move The Midway from Children’s Fairyland by 1st Cycling at map website.

I’d like to move Chase Center from San Francisco - Oakland Bay Bridge with 2nd Transit option at map website.

I want to move Pier 39 from Berkeley by 3rd Cycling at map website.

I want to go to Emeryville from Mountain View with 2nd Cycling option at map website.

Can you point out the way from San Mateo to Stanford University by 2nd Cycling at map website.

Could you point out the way from Palace of Fine Arts to UC Berkeley by 1st Cycling at map website.

Point out the way from The Painted Ladies to San Francisco Museum of Modern Art by 2nd Driving at map website.

Could you find the path from Union Square to Palo Alto by 1st Cycling at map website.

Please check the way from San Jose to San José Mineta International Airport with 1st Walking at map website.

Check the path from San Francisco Zoo to Berkeley with 1st Cycling at map website.

I’d like to check Parking Lots along the way from Stanford University to The Painted Ladies with Best option at map website.

Check Gas stations along the way from de Young Museum to Oakland with Driving option at map website.

Please show me Hotels along the way from Palace of Fine Arts to Berkeley by Transit at map website.

Check Gas stations along the way from Bay Area Discovery Museum to Santa Cruz with Best option at map website.

Appendix G Example Episode in Real-World Web Automation

Appendix H Extensive Ablation of HTML-T5

To test our recipe described in Section 2.1, we compare the different dataset and model initialization for pre-training on downstream task performances; offline task planning on real-estate and average success rate on MiniWoB with 12K dataset. We use Base-size models for the experiments. For HTML-denoising, we prepare the corpus from CommonCrawl with (Extracted) or without (Raw) subtree extraction around label elements on the documents. We also compare the initialization of base architectures before HTML-denoising; from scratch or with pre-trained models on PEGASUS objective (Zhang et al., 2020) that is a masked important sentence prediction from long-context paragraph. Table 7 reveals that snippet extraction on HTML corpus improves downstream performances since such a pre-processing can reduce the noise in raw HTML. Moreover, initialization with PEGASUS pre-trained weights is essential for HTML-T5, because of the long-context and instruction-following nature of HTML-based tasks.

H.2 Offline Evaluation on Task Planning with Model Scaling

We compere the offline task planning performance between HTML-T5 and LongT5 (without HTML-denosing) with different model sizes; with Base (220M parameters), Large (770M parameters), and XL (3B parameters). As described in Section 3.1, the models predict the next sub-instructions in a closed-loop manner considering the current HTML observations, user instructions, and previous sub-instruction histories as inputs. For offline task planning evaluation, we use the demonstrations on real-estate website; preparing 130 demonstrations and splitting them into train (90%) and test splits (10%). We report the best per-step exact match accuracy in test set.

Table 8 shows that HTML-T5 outperforms LongT5 on the accuracy of sub-instruction prediction, which demonstrates that HTML-denoising pre-training captures the structural bias of HTML better without sacrificing the ability to understand natural language instructions. This also implies that our proposed HTML-denoising can scale to larger-size models consistently.

H.3 Description Generation

We also investigate the capability of HTML-T5 on static HTML comprehension tasks, as well as interactive decision making tasks. We use Description Generation benchmark (Gur et al., 2022), where the models generate the textual description of elements, typically used for accessibility purposes and annotated with a special attribute in the HTML schema known as for. We evaluate the understanding the structure of HTML as it would appear to a user, despite not having access to the rendered website directly.

We compare LaMDA (Thoppilan et al., 2022), T5, LongT5, and HTML-T5 with respect to accuracy, BLEU (Papineni et al., 2002), and ROUGE-1 (Lin, 2004) score. As shown in Table 9, local and global attention mechanisms, underlying between LongT5 and HTML-T5, could almost solve the benchmark by improving the previous best performance by over 10%, with still improved performance as model size increases. Compared to the effect of local-global attention, HTML-T5 marginally improves against LongT5, which emphasizes that local and global attentions are critical to capture the hierarchical structure of HTML documents.

Appendix I Flan-LongT5

In the web automation literature (Furuta et al., 2023; Kim et al., 2023), instruction-finetuned LLMs have great success in HTML comprehension and improve the task success. For the comparison to HTML-denosing, we prepare the instruction-finetuned LongT5 (i.e. Flan-LongT5) by leveraging Flan dataset released by Chung et al. (2022). We finetuned the pre-trained LongT5 with 100K iterations and picked up the best checkpoints.

As a sanity check of instruction-tuning, we evaluate Flan-LongT5 with few-shot/zero-shot settings on CoT benchmark (GSM8K (Cobbe et al., 2021), StrategyQA (Geva et al., 2021), SVAMP (Patel et al., 2021), Asdiv (Miao et al., 2021), CommonsenseQA (Talmor et al., 2019)), BigBench-Hard (BBH) (Suzgun et al., 2022), and MMLU (Hendrycks et al., 2021b) as tested in Longpre et al. (2023). We reevaluate the performance of Flan-T5, using official checkpoints https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints. We also check the performance of Flan-LongT5 on downstream summarization tasks, originally evaluated on LongT5 (Guo et al., 2022). We use arXiv (Cohan et al., 2018), PubMed (Cohan et al., 2018), BigPatent (Sharma et al., 2019), Multi-News (Fabbri et al., 2019), MediaSum (Zhu et al., 2021), CNN / Daily Mail (Nallapati et al., 2016) dataset for the evaluation, measuring the performance with ROUGE-1/2/L metrics.

Table 10 shows that we have successfully replicated the LongT5 version of instruction-finetuned language models. Flan-LongT5 achieves competitive results to original Flan-T5; for instance, Flan-LongT5-Large (36.64) outperforms Flan-T5-Large (35.25), but Flan-LongT5-XL (39.05) is still behind Flan-T5-XL (43.03) on average. This might be caused by the training instability of XL-size models (Guo et al., 2022). Because, unlike HTML-T5 on HTML-based tasks, reasoning tasks do not have long-context or hierarchical syntax, it is not surprising for Flan-LongT5 not to outperform Flan-T5. Table 11 also demonstrates that we have successfully conducted instruction-tuning without losing the capability of long text summarization.

Appendix J Per-Task Performance on MiniWoB++