WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Introduction

Recent advances in natural language processing (NLP) and reinforcement learning (RL) have brought about several exciting developments in agents that can perform sequential decision making while making use of linguistic context . On the other hand, large-scale language models like GPT-3 and BERT are excelling at traditional NLP benchmarks such as text classification, information extraction and question answering. While the former set of tasks are limited in their set of linguistic concepts and prove difficult to scale up, the latter tasks usually contain static, non-interactive datasets that lack adequate grounding to extra-linguistic concepts . In order to make further progress in building grounded language models, we believe there is a need for scalable interactive environments that contain: (1) language elements that reflect rich, real-world usage and are collectible at scale, and (2) task feedback that is well-defined and automatically computable to facilitate interactive learning, without the constant need for expensive feedback from humans.

The world wide web (WWW) is a massive open-domain interactive environment that inherently satisfies the first aforementioned requirement through its interconnected set of pages with natural text, images and interactive elements. By being simultaneously scalable, semantic, interactive, dynamic and realistic, the web is uniquely different from existing environments for autonomous agents like games or 3D navigation. Moreover, the web also provides a practical environment to deploy trained agents, with great potential for alleviating human efforts in tedious tasks (e.g. buying products, booking appointments). While there has been prior work on building web-based tasks, they either lack depth in the transition and action spaces, or prove difficult to scale up. Some benchmarks only contain either a single classification task or interactions containing only a handful of different pages in each episode . Others propose tasks with longer horizons but are either limited to following hyperlinks for web navigation or require human-in-the-loop feedback due to the lack of an automated reward function .

In this paper, we introduce WebShop (Figure 1) – a large-scale interactive web-based environment for language understanding and decision making – and train autonomous agents to complete tasks on this benchmark. With the goals of being scalable and containing realistic language and visual elements, WebShop emulates the task of online shopping on an e-commerce website, where the agent’s goal is to understand a human-provided text instruction and purchase a product to match the specifications. To do so, the agent needs to query the website’s search engine, choose items to explore from search results, open and read their description and details, and select the necessary options (e.g. 32 oz., red color) before clicking the ‘Buy’ button. In order to pick the optimal product that matches user requirements, the agent may need to view and compare various products (including backtracking between pages), and potentially perform multiple searches. WebShop contains over one million products scraped from amazon.com, over $12$ thousand crowdsourced instructions, and a diverse semantic action space of searching text queries and choosing text buttons. It is packaged into a convenient OpenAI Gym environment and can be rendered in two modes (HTML or simple) with parallel observation spaces that are easy for human and model respectively. Rewards are automatically computed using a combination of programmatic matching functions that consider the attributes, type, options and price of the chosen product, alleviating the need for human evaluation and providing a path to scaling up interactive learning.

We develop several agents to perform this task, using both reinforcement learning (RL) and imitation learning (IL). We also leverage the latest pre-trained language models for representing and generating text. Our modular architecture includes a factorized processing of state observations and action choices using ResNets (visual) and Transformers (text), followed by an attention fusion layer that helps the agent contextually score each action. Our best agent achieves an average score of $62.4$ (out of $100$) and successfully completes the task $28.7\%$ of the time, significantly higher than a heuristic baseline that achieves $45.6$ and $9.6\%$, respectively. While this demonstrates the potential for IL and RL, the agents are still much lower than human experts, who can achieve $82.1$ and $59.6\%$ on this task. In our analysis (§5.3), we observe that the task requires patience and consistency, which is lacking in some crowdsource workers, leading to lower scores. Even with this caveat, the gap between human performance and the model remains significant. We perform several analyses and ablation studies to identify the cause of this gap and find several avenues for agent improvement in the future including more robust search generation, explicit memory modules, and better handling of noisy web text. Finally, we also demonstrate an instance of sim-to-real transfer by deploying agents trained with WebShop to operate on amazon.com and ebay.com, and find that they can achieve similar performances despite search engine and product differences, and consistently outperform the rule baseline of using the first result returned by the commercial search engines when directly searching the instruction texts. This demonstrates the practical potential of our work towards developing agents that can operate autonomously on the world wide web (WWW).

Related Work

Reinforcement learning on the web. Nogueira and Cho introduced WikiNav as a benchmark for RL agents navigating pages, but the task is purely navigational with the actions restricted to either choosing a hyperlink to follow or deciding to stop. The World of Bits (WoB) benchmark enables training of RL agents to complete tasks on webpages using pixel and Document Object Model (DOM) observations. Several follow-up papers have tackled MiniWoB using techniques like workflow-guided exploration , curriculum and meta-learning , DOM tree representation , adversarial environment generation and large-scale behavioral cloning . However, MiniWoB lacks long-range decision making across multiple different pages and does not scale easily in terms of difficulty or size due to its use of low-level mouse clicks and keystrokes as actions. In contrast, WebShop requires navigating longer paths with context-based action selection and backtracking, and it uses high-level $search$ and $choose$ actions that are more scalable and transferable to real settings. While not directly operating on web pages, AndroidEnv and MoTIF provide environments to train agents for interacting with apps and services on mobile platforms.

Non-interactive web-based tasks. Various supervised classification tasks on webpages have been proposed, including predicting web elements , generating API calls and semantic parsing into concept-level navigation actions . Perhaps most similar content-wise to our work is the Klarna product page dataset which contains over $50,000$ product pages labeled with different element categories for supervised classification. All these works only consider supervised settings with a single decision, and may require the definition of web APIs or command templates for each domain. Our benchmark, WebShop, combines webpages with realistic text and image content with a rich and diverse interaction space for long-range sequential decision making.

Leveraging the web for traditional NLP tasks. Several papers have explored the use of the web for information extraction and retrieval , question answering , dialog , and training language models on webtext . These approaches primarily use web search engines as a knowledge retriever for gathering additional evidence for the task at hand. Perhaps most similar to our work is WebGPT , which uses a web interface integrated with a search engine to train RL agents to navigate the web and answer questions. However, our environment has a more diverse action and observation space (including images) and does not require human-in-the-loop evaluation.

The WebShop Environment

We create WebShop as a large-scale web-based interactive environment with over $1.1$ million real-world products scraped from amazon.com. In this environment, an agent needs to find and purchase a product according to specifications provided in a natural language instruction. WebShop is designed in a modular fashion which disentangles the website transitions from the task-specific aspects like instructions and reward, allowing for easy extension to new tasks and domains.

WebShop can be formulated as a partially observable Markov decision process (POMDP) $(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{U},\mathcal{O})$ with state space $\mathcal{S}$, action space $\mathcal{A}$, deterministic transition function $\mathcal{T}:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$, reward function $\mathcal{R}:\mathcal{S}\times\mathcal{A}\to$, instruction space $\mathcal{U}$, and a state observation space $\mathcal{O}$.

State and action. A state $s\in\mathcal{S}$ represents a web page, which falls into one of the four types – the search page that contains a search bar, the results page that lists a set of products returned by a search engine, the item page that describes a product, or the item-detail page that shows further information about the product (Figure 1A(1-4) respectively). We define the following notations for a product $y$. We denote $\bar{y}$ to be the aggregation of the various text fields including product title, description, and overview. We denote $y_{\text{price}}$ to be the price, $Y_{\text{opt}}$ to be a set of buying options, and $I$ to be a set of images, each corresponding to a specific option. Finally, each product is associated with $Y_{\text{att}}$, a set of attributes hidden from the agent which is extracted from the title and the item-detail pages (§3.2). The attributes are used for the automatic reward calculation.

An action $a\in\mathcal{A}(s)$ can either be searching a text query (e.g. search[Red shoes]) or choosing a text button (e.g. choose[Size 9]) as shown in Table 1. These two action types are not available simultaneously – search is only allowed when the agent is at the search page; on all other pages, click is the only action choice. The chosen action argument (button) will be clicked as a web link as opposed to the low-level mouse-click actions in previous environments such as World of Bits . The transitions initiated by clicks deterministically redirect the web page to one of the four page types (Table 1). The transition initiated by search is based on a deterministic search engine (§3.2).

Observation. Using Flask and OpenAI Gym , we provide two parallel observation modes to render the state and instruction $\mathcal{S}\times\mathcal{I}\to\mathcal{O}$: (1) HTML mode that contains the HTML of the web page, allowing for interaction in a web browser(Figure 1A), and (2) simple mode which strips away extraneous meta-data from raw HTML into a simpler format (Figure 1B). The human performance scores in §4.2 are collected in the HTML mode, while all models are trained and evaluated in the simple mode. Note that while the environment allows for training reinforcement learning agents on raw pixels in HTML mode (like in Shi et al. ), we believe that it provides a very low-level non-semantic action space. Moreover, it is straightforward to write a translator that converts any new HTML page into simple format for use with trained agents, which enables sim-to-real transfer.

Instruction and reward. Each natural language instruction $u\in\mathcal{U}$ contains the following information: a non-empty set of attributes $U_{\text{att}}$, a set of options $U_{\text{opt}}$, and a price $u_{\text{price}}$. The instruction is generated based on a target product $y^{*}$ by human annotators. The instruction collection process is lightweight and scalable (§3.2). Concretely, $U_{\text{att}}\subseteq Y_{\text{att}}^{*}$ is a subset of the product attributes, $U_{\text{opt}}\subseteq Y_{\text{opt}}^{*}$ is a subset of the product option field-value pairs, $u_{\text{price}}>y_{\text{price}}^{*}$ is a price set to be higher than the target product price. For example, the instruction “Can you find me a pair of black-and-blue sneaker that is good in rain weather? I want it to have puffy soles, and price less than $90$ dollars.” contains the aforementioned attributes $U_{\text{att}}=\{\text{``waterproof''},\text{``soft sole''}\}$ and option $U_{\text{opt}}=\{\text{``color'': ``black and blue''}\}$. In each episode, the agent receives a reward $r=\mathcal{R}(s_{T},a)$ in the end at timestep $T$, where $a=\texttt{choose[}{\text{buy}}\texttt{]}$, $y$ is the product chosen by the agent in the final state $s_{T}$, and $Y_{\text{att}}$ and $Y_{\text{opt}}$ are its corresponding attributes and options. The reward is defined as:

where the type reward $r_{\text{type}}=\texttt{TextMatch}(\bar{y},\bar{y}^{*})$ is based on text matching heuristics to assign low reward when $y$ and $y^{*}$ have similar attributes and options but are obviously different types of products. For example, “butter” and “plant-based meat” differ in types but may both contain attributes “cruelty-free”, “non-GMO”, and an option “size: pack of 2”. The exact formula for $\texttt{TextMatch}(\cdot)$ is in the Appendix §A.5.

Evaluation metrics. We use two evaluation metrics: (1) Task Score: defined as $(100\times\text{avg. reward})$, which captures the average reward obtained across episodes; and (2) Success Rate (SR) defined as the portion of instructions where $r=1$. Note that it is possible to obtain $r=1$ for an episode even if the final product is not $y^{*}$ — for example, there could be many items that satisfy the goal “I want a red shirt”, even if the goal is generated from a specific red shirt item.

2 Environment Implementation

Data scraping. We use ScraperAPI to scrape $1,181,436$ products from amazon.com across $5$ categories (fashion, makeup, electronics, furniture, and food) using $113$ sub-category names as queries. The product texts (title and item details) have an average length of $262.9$ and a vocabulary size $224,041$ (word frequency higher than $10$). In addition, the products have a total of $842,849$ unique options, reflecting the scale and complexity of the data. More details about product scraping is in the Appendix §A.1.

Search engine. We use Pyserini for the search engine, where indices are built offline using a BM25 sparse retriever with text for each product concatenated from the title, description, overview, and customization options. The search engine is deterministic, which eases imitation learning and result reproducibility. More details in A.3.

Attribute mining and annotation. Each product is annotated with a set of hidden attributes, which are used to represent its latent characteristics as well as to calculate the reward as detailed in §3. An attribute is a short natural language phrase that describes the property of the product (see examples in Figure 1). We mine the attributes by calculating TF-IDF scores for all bi-grams in the concatenated titles and descriptions based on each product category. We review the top $200$ bi-grams for each category, remove the noisy ones by inspection (decide based on whether the bi-gram is human understandable), and assign them to the products. We consolidate a pool of $670$ attributes. See more details in the Appendix §A.2.

Natural language instructions. We use Amazon Mechanical Turk (AMT) to collect natural language instructions that specify goal products with appropriate options. Specifically, an AMT worker is presented with a sampled goal product, including the product title, category, attributes, and the buying options, and asked to write a command to instruct an automatic shopping agent to find the target. Workers are instructed to avoid being too specific such as including the entire title in the instruction, but stay faithful to describing the target product. We collect a total of $12,087$ linguistically diverse instructions with an overall vocabulary size of $9,036$ words and an average length of $15.9$ words. We provide the detailed annotation process and interface in the Appendix §A.4.

Human demonstrations. We collect trajectories from humans performing the task in the HTML mode of WebShop to understand the task difficulty for humans and to analyze how humans would solve the task. We use qualification tests to train and select motivated workers to perform the task. We recruit and train a total of $13$ workers for data collection, and among them we select the top $7$ performing workers to be “experts” (see Appendix §A.6 for examples). We also leverage this data to perform imitation learning (described in §4.2).

3 Research Challenges

WebShop brings together several research challenges for autonomous systems from various subfields in NLP and RL into a single benchmark. These include: 1) generation of good search queries and reformulation , 2) strategic exploration for navigating through the website , 3) robust language understanding for textual state and action spaces , and 4) long-term memory for comparing items or backtracking (Figure 1). While we believe individual advances in each of these will improve agent performance, WebShop also provides an ideal testbed for the development of interdisciplinary techniques that tackle more than one of the above mentioned challenges simultaneously. For example, external memory modules may be very effective if combined with strategic exploration, or exploration could be helpful in information query reformulation. Further analysis based on human and model trajectories is in §5.3.

Methods

We propose various models that combine language and image pre-training with imitation learning (IL) and reinforcement learning (RL). More details are provided in the Appendix §B.

A simple rule baseline is to search the exact instruction text, then choose and buy the first item in the results page without choosing any options. The heavy lifting of the lexical search engine makes it also a simple non-learnable information retrieval (IR) baseline, and would lead to a non-trivial attribute reward. However, simple heuristic rules cannot resolve noisy natural language options, strategically explore, or learn to generate what to search, so the total reward and task success rate should be low.

2 Imitation Learning (IL)

For the text generation and choice problems presented in WebShop, we propose using two pre-trained language models to separately learn how to search and choose from human demonstrations.

Imitating human search generation. We frame searching as a sequence-to-sequence text-generation problem: the agent generates a search action $a=\texttt{search[}{\dots}\texttt{]}$ given an instruction $u$ without considering any other context (e.g. past searches, visited items). We use $M=1,421$ instruction-search pairs from $1,012$ training human trajectories to construct a dataset $\mathcal{D}=\{(u,a)\}_{i=1}^{M}$ and fine-tune a BART model parameterized by $\phi$ to perform conditional language modeling:

Imitating human choice. The choice-based imitation model (Figure 3) predicts a probability distribution over all the available click actions $\mathcal{A}(o)$ in observation $o$ and maximizes the likelihood of the human clicked button $a^{*}\in\mathcal{A}(o)$. We construct a dataset $\mathcal{D}^{\prime}=\{(o,\mathcal{A}(o),a^{*})\}_{i=1}^{M^{\prime}}$ of $M^{\prime}=9,558$ samples from the training human trajectories. We use a $12$-layer pre-trained BERT model parameterized by $\theta$ to encode the $o$ into an observation representation of contextualized token embeddings, and we similarly encode each action. Each action representation is passed into a cross-attention layer with the observation representation, then mean pooled into a single vector and multiplied with a matrix $W$ to obtain a scalar score $S(o,a)$. The policy $\pi_{\theta}\left(a\mid o,\mathcal{A}(o)\right)$ is the softmax distribution over action scores $S(o,a)$:

Handling Images. We use a pre-trained ResNet-50 to pre-process images across different products and options into a $512$ dimensional feature vector, which is then transformed into $768$ dimensions with a learned linear layer and concatenated to $\text{BERT}(o)$ as the observation representation.

Full pipeline. Combining the above during environment interaction, we use the BART model in the search page to generate the top-$5$ search queries via beam search and choose a random one. For other pages, we sample one action from $\pi_{\theta}\left(a\mid o,\mathcal{A}(o)\right)$ using the BERT model. We find these methods useful to encourage diverse actions. In contrast, an ineffective strategy that uses only the top generated search query or the button with the highest probability might lead to limited product candidates or being stuck (e.g. bouncing back and forth between pages).

3 Reinforcement Learning (RL)

The value $V(o)$ is learned with an L2 loss $\mathcal{L}_{\text{value}}=(R_{t}-V(o_{t}))^{2}$. We also add an entropy loss \mathcal{L}_{\text{entropy}}=\sum_{a\in\mathcal{A}(o_{t})}\pi_{\theta}\big{(}a_{t}\mid o_{t},\mathcal{A}(o_{t})\big{)}\log\pi_{\theta}\big{(}a_{t}\mid o_{t},\mathcal{A}(o_{t})\big{)} to prevent premature convergence. Our full RL model minimizes the total loss $\mathcal{L}_{\text{RL}}=\mathcal{L}_{\text{PG}}+\mathcal{L}_{\text{value}}+\mathcal{L}_{\text{entropy}}$.

Experiments

We split a total of $12,087$ instructions into an i.i.d. distributed train / development / test split of $10,587$ / $1,000$ / $500$ instances for all models. While future work can investigate splits with more generalization gaps (e.g. split by product category), we will show the i.i.d. split is already challenging for current models. We randomly sample a subset of the $10,587$ training instructions, then collect $1,012$ human demonstrations for task verification and imitation learning (IL) and a further $54$ demonstrations from instances in the development set for IL hyperparameter tuning and checkpoint selection. We also collect human trajectories for all $500$ test instructions and report human and model performances averaged across these $500$ instructions. More setup details are in the Appendix §C.

2 Results

Task performance. From Figure 4, we observe that the rule baseline obtains a low score of $45.6$ and a very low success rate of $10\%$ since it cannot resolve options specified in language or explore more products, empirically demonstrating the non-trivial nature of the task. The IL model significantly outperforms the rule baseline on both metrics, achieving a score of $59.9$. Further RL finetuning improves the score to $62.4$ while slightly hurting the success rate ($29.1\%\rightarrow 28.7\%$) (analyzed further in §5.3). We also observe a significant gap between models and humans – our best model’s success rate ($29.1\%$) is less than half of expert humans ($59.6\%$) and only $60\%$ of the average human ($50\%$). This indicates a great room for model improvement by tackling reseach challenges in WebShop.

IL ablations. Figure 4 also contains several ablations that confirm important design choices for models. When the choice action model for the IL agent is randomly initialized (IL (w/o LP Choice); LP = language-pretraining), the success rate drops by nearly two-thirds, indicating the importance of language pre-training for our task. When the search query generator in the IL agent is replaced by a simple rule, which always uses the instruction text (IL (w/o LP Search)), both reward and success rate drop by around $3$ points. This suggests the importance to explore by expanding the search space for exploration, but it is not as critical as learning to choose the right options. We experiment with incorporating history of one past observation and the last five actions into the model and find a slight degradation in the score from $59.9$ to $57.3$, suggesting more advanced techniques are needed to leverage past information. More ablations in §C.

RL ablations. When we directly train an RL agent (RL) from pre-trained BERT parameters, the performance is even worse than the rule baseline. This suggests that IL warm-starting is critical, possibly because of the significant domain shift from traditional language tasks. We also consider a simple RL model with RNN text encoders instead of the Transformer (RL (RNN)), which has a success rate more than $10\%$ worse than the IL + RL model with a much larger variance. We hypothesize that RL with a more powerful architecture could help boost and stabilize the performance if the model is initialized with better language and task priors.

3 Analysis

To better understand the differences between the agents and human experts, we perform several fine-grained analyses. We first break down the overall score into its four sub-parts according to Eq. (1): 1) attribute score ($|U_{\text{att}}\cap Y_{\text{att}}|/|U_{\text{att}}|$), 2) option score ($|U_{\text{opt}}\cap Y_{\text{opt}}|/|U_{\text{opt}}|$), 3) price score ($\mathbf{1}[y_{\text{price}}\leq u_{\text{price}}]$), and 4) type score ($r_{\text{type}}$). We report trajectory statistics such as the average number of states, unique items visited, and number of searches per episode in Table 2 and provide qualitative examples of the trajectories in Table 3.

Human expert vs. agents. Human experts outperform the agents on all score sub-parts (Table 2), but the most significant boost comes from the option score (a $28\%$ gap), revealing that agents have trouble selecting the correct product options. Humans also have longer trajectories, explore more items and perform more searches than the agents, with a higher variance, demonstrating their flexibility. Table 3 provides some samples trajectories. In the first example, the human decides to search again after removing ‘inches’, ‘width’, ‘height’, and ‘white’ from the query since product texts often contain abbreviated symbols for these terms like ‘"’, ‘w’, and ‘h’. Thus, search generation is challenging for models since it involves reasoning and adapting to grounded environments, and ideas from query reformulation could help alleviate this. Agents also struggle to perform robust semantic matching, which is important in choosing options that contain noisy paraphrases of instruction spans. In the second example, the human explores several products first, and decides to return to the first explored product, demonstrating long-term memory that is lacking in the IL+RL model.

Effect of RL fine-tuning after IL. Table 2 also shows that RL fine-tuning adapts the IL model to become more ‘greedy’ and less ‘exploratory’, as the average trajectory length drops from $9.4$ to $4.8$, and the model explores fewer items and search queries. As a result, the attribute, type, and price scores all increase, but option score drops from $45.2$ to $38.9$. This points to the need for a better balance exploration with exploitation during RL, e.g. by using intrinsic bonuses.

To disentangle the effects of learning to search from choosing the right actions, we construct a Choice oracle that has access to the hidden reward function as well as hidden attributes and options underlying each product and instruction.A similar search oracle is also possible but harder to design since the search space is infinite. One possible oracle is to search for the underlying product name for each instruction, but that makes choice trivial as the underlying product is then almost always the first search result. Given a search query, the Choice oracle will perform an exhaustive search over every result item, try out all combinations of options and finally choose the best item with options that maximize the reward — meaning each episode will take hundreds or thousands of steps, as opposed to $4.5$ and $11.3$ steps on average for the IL+RL model and human experts (Table 2). We use 500 test instructions and consider four types of search queries: the instruction text (used by rule baseline), top IL BART generated query (used by all learning models), and the first and last queries from human experts in each test trajectory.$74.8\%$ of the time there is only one query in the trajectory. Choice oracle improves the success rate of rule heuristics from $9.6\%$ to $85.4\%$, and even the human expert success rate from $59.6\%$ to $87.8\%$ (Table 4), confirming that choosing the right actions is indeed a major bottleneck for current models with great room for improvement. However, using a better search query is still important even with such a strong Choice oracle, as the last human search query still outperforms other search queries. This also suggests human experts improve search query qualities over reformulations.

4 Zero-shot Sim-to-real Transfer

Finally, we conduct a ‘sim-to-real’ transfer experiment where our models trained on WebShop are tested on the real-world Amazon (amazon.com) and eBay (ebay.com) shopping websites without any fine-tuning. We sample $100$ test instructions and deploy $3$ WebShop models (rule, IL, IL+RL) to interact with Amazon and eBay, and manually score each episode based on Eq. (1). As shown in Table 5, model performances on the two website are similar to WebShop performances in Figure 4, except for the rule baseline, likely due to the better search engine of Amazon than WebShop.

On amazon.com, IL+RL achieves a Score of $65.9$ and SR of $25\%$, outperforming the Rule baseline’s Score of $45.8$ and SR of $19\%$ by large margin. Similarly, on ebay.com, IL+RL achieves a Score of $62.3$ and SR of $21\%$, widely outperforming the Rule baseline’s Score of $31.7$ and SR of $7\%$. These results confirm positive sim-to-real values of trained agents for real-world web tasks despite domain shifts in data (products) and dynamics (search engine). We also obtain a human average score of $88.0$ / $79.7$ and success rate of $65\%$ / $40\%$ by asking turkers (§3.2) to find the instructed product on the Amazon and eBay websites respectively. While humans perform much better than agents, their web interactions are much slower — taking on average $815$ seconds per episode as opposed to $<8$ seconds per episode for our IL and IL+RL models on Amazon. This sim-to-real transfer only requires two minor coding additions, suggesting that environments like WebShop are suitable for developing practical grounded agents to reduce human effort on real-world web tasks. We provide additional performance and in-depth analysis in Appendix §D.

Discussion

We have developed WebShop, a new web-based benchmark for sequential decision making and language grounding, modeled on interaction with an e-commerce website. We performed an empirical evaluation of autonomous agents trained using imitation and reinforcement learning, and demonstrated promising results on sim-to-real transfer to real-world shopping websites. Our qualitative and quantitative analysis of model and human trajectories (§5.3) identified several research challenges in WebShop and provided insights for future model development by incorporating multidisciplinary techniques. For example, pre-training with multi-modal data , web hypertext , or web instruction-action mapping could help agents better understand and leverage rich semantics of webpage content, actions, and instructions. Ideas from query (re)formulation may help agents expand the range of search exploration, and improved action exploration and memory mechanisms could help agents make better decisions over the long horizon and large action space. The modular design of WebShop also allows for new web tasks and domains to be easily incorporated, which we hope will help shape future research into grounded language agents with stronger capabilities for real-world web interaction.

Acknowledgements

We thank Alexander Wettig, Ameet Deshpande, Austin Wang, Jens Tuyls, Jimmy Yang, Mengzhou Xia, Tianyu Gao, and Vishvak Murahari from the Princeton NLP Group for proofreading and providing comments. This material is based upon work supported by the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References

Checklist

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

Did you describe the limitations of your work? [Yes] See Section 6 Discussion and Appendix.

Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6 Discussion and Appendix.

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [N/A]

Did you include complete proofs of all theoretical results? [N/A]

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See supplementary materials.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Data splits are described in the Section 5 first paragraph. Hyperparameters and training details are in the Appendix.

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] Figure 3 includes error bars, Table 2 includes min/max statistics along with averages.

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In appendix training details.

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes] Citations include ScraperAPI, Flask, OpenAI Gym, BERT, BART, A2C.

Did you mention the license of the assets? [Yes] Discussed in appendix.

Did you include any new assets either in the supplemental material or as a URL? [Yes] In the supplementary materials.

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [Yes] Discussed in appendix, we only scrape publicly available data from the Internet.

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Discussed in Appendix.

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] In appendix.

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] Discussed in Appendix.

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] Discussed in Appendix.

Appendix A Environment Details

We use ScraperAPI to extract publicly available product information from amazon.com. We use five categories (beauty, food, fashion, furniture, electronics) and 313 associated sub-category names appeared in amazon.com (e.g. “Women’s Loafers & Slip-Ons” in fashion, “Pendants and Chandeliers” in furniture) to scrape $1,181,436$ products. We filter products with duplicate titles or product IDs, but do not perform extra filtering in order to avoid selection bias. Specifically, as amazon.com has its own content screening process, we did not find any personally identifiable information or offensive content during random sampling checks.

A.2 Product Attribute Mining

We use TfidfVectorizer from scikit-learn to extract probable bi-grams as attributes from product title and descriptions for further annotation. We manually inspect these attributes to keep only the specific and human-readable ones and filter out the rest. An attribute should be suitable in at least one of the following use: 1) IsGoodFor, 2) HasA (contains), 3) WhichIs, and 4) IsA. For example, attributes such as “oz ml” and “men women” will be filtered out since it’s unparsable. On the other hand, “hair color” will also be filtered since it is not specific enough to fit in the above $4$ categories. Attributes such as “dry skin” can fit the IsGoodFor in the context of a make-up product being good for dry skin.

A.3 Search Engine

Each time the agent performs a search, the top 50 items are retrieved and displayed across five search result pages, where each page contains 10 items and the agent can use actions choose[Prev/Next page] to navigate across result pages. Figure 1 shows that when searching directly with the instruction text, the corresponding item appears in the first search page (rank 1-10) nearly 1/3 of the time, but it cannot be found in any search pages (rank 50+) more than half of the time. This indicates that while the search engine can decently retrieve items based on lexical matching, directly searching the instruction is not enough for solving the task, and good query (re)formulation based on the instruction is important.

A.4 Instruction Collection

We collect human written instructions by providing the workers a product including the title, product category, and its set of attributes and options (Figure 5, 6). We conduct qualification task by having each participating workers to work on $2-5$ examples. We inspect and assign qualification to $213$ workers to perform the instruction writing task. We pay for each example $0.15$ dollars. We do not anticipate any potential participant risk.

A.5 Reward Calculation

The type reward $r_{\text{type}}$ consists of $3$ elements: 1) course-grain product category match ($c=1$ if matched), 2) fine-grain category match ($f=1$ if matched), and 3) product title match. Course-grain product category refers to the $5$ categories described in §3.2. Fine-grain category is the chain of categories that the product is under on the Amazon website. For example, and eye mask sheet would be under the Beauty & Personal Care > Skin Care > Eyes > Wrinkle Pads & Patches fine-grain category. The product title refers to $\bar{y}$ described in §3.

Here, TextMatch($\bar{y}$, $\bar{y}^{*}$) is a simple string match between the selected product title text and the goal product title text. We use only the words tagged with PNOUN, NOUN, and PROPN tags parsed by the SpaCy parser in the title text.

A.6 Human Trajectory Collection

We use the HTML environment in Figure 1 to collect human trajectories. We select a pool of $13$ workers using qualification tasks where each workers complete $5$ examples. The workers that achieve an average reward more than $0.75$ are qualified. The task instruction is shown at the end of Appendix. We observe a pronounced performance gap between the very high performing workers and average workers. We use the top $50\%$ of these qualified workers as experts ($7$ workers in total). We pay for each completed trajectory $0.7$ dollars. In human evaluation, $8$ out of the $13$ workers participated and $5$ among them are in the aformentioned expert pool. The $8$ participants achieve an overall score of $75.5$ and a success rate of $50.0\%$ We observe non-negligible variance even within the experts—the best performer achieves a score of $87.4$ and success rate of $69.5\%$, while the lowest performing worker achieves a score of $45.8$ and success rate of $10\%$. The best performing worker also shows better consistency—drawing at a standard deviation of $2.3$ in score, contrasting the lowest performing counterpart at $3.1$. We provide examples of common human failure cases such as not matching the option/attribute due to impatience (Table 7), cautioning some caveats of the task with human workers.

A.7 Reward Verification

We randomly select $100$ samples each from the pools of trajectories generated by average and expert MTurk workers. Each trajectory is then manually re-scored against a human criteria; the purpose of this is to determine how representative the reward function is of a human’s judgment towards whether the chosen product satisfies the given instructions. The human score calculation procedure exactly follows the formula laid out in Section A.5 – the attribute, option, price, and type scores are individually determined, then aggregated to calculate the overall score – except for one main modification. Instead of the exact matching approach, points are awarded if (1) the picked product’s attributes, options, or type are lexically similar or synonymous with the goal’s product information and (2) the desired value is not found verbatim anywhere in the picked product’s descriptions. For instance, if the value lightweight is specified as a desired attribute for an instruction, but the value easy carry is found instead in the picked product’s description, then the attribute score for the picked product is increased to reflect that the lightweight value was found. On the other hand, if cyan is desired as an option for a goal product, but the user picks blue even though cyan is available as a choice, then no points are awarded. To ensure the score is calculated without bias, the original rewards for each trajectory were not compared with the human evaluation scores until the human evaluation scoring was completed.

For the average trajectories, the automatic task score was $74.9$ and our manual score was $76.3$ with a Pearson correlation of $0.856$. For expert trajectories, the respective scores were $81.5$ and $89.9$ with a Pearson correlation of $0.773$. Therefore, the automatic reward seems to provide a reasonably close lower bound to the actual task performance. We find that for average workers, $87.0\%$ of automatic scores are within a $10\%$ of the manual score, with the main source of error being synonyms or lexically similar words that don’t get matched correctly in the automatic reward function.

Table 8 reflects our observation that our reward function is similar to a human’s score, with a consistent tendency to over-penalize the picked product. For every trajectory’s product, the human score across all categories (e.g. attributes, options) is always greater than or equal to the original score. This under-scoring is a result of our reward function’s exact matching criterion. In future work, we hope to improve our matching functionality such that, within the context of a single product with respect to the goal instructions, it can identify synonyms and decide whether to award additional points.

Appendix B Model Details

Our cross attention layer follows Seo et al. . Denote the $i$-th contextualized token embedding from the observation and action to be $\mathbf{o}_{i}$ and $\mathbf{a}_{i}$ respectively. The attention between $\mathbf{o}_{i}$ and $\mathbf{a}_{j}$ is defined as

where $\otimes$ denotes element-wise product and $\mathbf{w}_{1},\mathbf{w}_{2},\mathbf{w}_{3}$ are learnable vectors. The observation-contextualized vector for $j$-th action token is then

We then average pool all $\mathbf{ca}_{j}$ to derive the action score $S(o,a)$:

where $n_{a}$ is the number of tokens for action $a$.

B.2 RNN Baseline

Our RNN baseline is inspired by Guo et al. , where we use the same attention layer as described above, but replace the Transformer text encoder with one-layer bi-directional Gated Recurrent Units (GRU) of hidden dimension 512. Another difference is that we also add an cross attention between the instruction and action input word embeddings, as we hypothesize it might help option text matching.

Appendix C WebShop Experiment Details

The training code for our IL models is adapted from Huggingface glue training example, whose repository is licensed under Apache License 2.0. We use a training batch size of 1 with 32 gradient accumulation steps, a learning rate of $2\times 10^{-5}$, and 10 training epochs. The training takes around 2 hours on one RTX 2080 GPU with a GPU memory of around 10GB.

C.2 RL Training Details

We train the RL models using 4 parallel environments for $100,000$ training steps. The backprogation through time (BPTT) is taken at every 8 steps. We use an Adam optimizer with a learning rate of $10^{-5}$ (for Transformer models) or $5\times 10^{-4}$ (for RNN models).

For RL models with the Transformer (BERT) architecture, it takes around 27 hours on one RTX 3090 GPU with a GPU memory of around 20GB. For RL models with the GRU architecture, it takes around 20 hours on one RTX 2080 GPU with a GPU memory of around 10GB.

To disentangle the effects of learning to search from choosing the right actions, we construct a Choice oracle that has access to the hidden reward function as well as hidden attributes and options underlying each product and instruction.A similar search oracle is also possible but harder to design since the search space is infinite. One possible oracle is to search for the underlying product name for each instruction, but that makes choice trivial as the underlying product is then almost always the first search result. Given a search query, the

C.3 Sampling vs. Top-1

We show comparisons between using beam search vs. top-1 for both the search model and the choice model in Table 9. During testing, the search model uses beam search to generate top-5 search queries. We randomly and uniformly sample from the top-5 queries to increase search diversity in case of multiple searches. We conduct experiments to instead always use the top-1 search, which shows slight performance improvement (see table below), and we will include the result in the paper. The choice model has a fixed set of action candidates at each step (e.g. all available buttons), and we sample from the choice policy what action to take, as always taking the top action will lead to significantly detorior performances.

C.4 Image Ablation

We train 3 trials with different random seeds for both the IL model and the ablated IL model without images, with performances over 500 test cases (10). Removing image only slightly hurts the overall performance, but significantly reduces the variance. This is reasonable as our current instruction and reward setups only use textual information, and we believe future efforts to incorporate visual information into the task setup will better challenge models’ visual understanding, and make pre-trained vision-language models such as CLIP more useful.

Appendix D Sim-to-real Details

To test how well our IL agent trained in WebShop performs on amazon.com (ebay.com similarly), we wrote a series of scripts that generally achieve two steps - translate a real Amazon URL into our IL model’s input (text observation, set of valid actions) and map the model’s output back to a real Amazon URL. The following procedure is repeated until the IL model generates a "buy now" action:

Amazon URL $\rightarrow$ Amazon HTML $\rightarrow$ Amazon Page Information: Using ScraperAPI , we first get the HTML source code for a given Amazon page, then extract information relevant to rendering the equivalent page in the WebShop environment (e.g. title, price, options).

Amazon Page Information $\rightarrow$ WebShop HTML $\rightarrow$ Text Observation: Given the scraped information, we generate the corresponding WebShop page in HTML mode, then transform it into a simple mode text observation.

Amazon Page Information $\rightarrow$ Valid Action Set: From the scraped information, we determine what valid actions the model can take (i.e. search[Red shoes], choose[Size 9]). This logic is captured as a mapping of page type to permissible actions.

Text Observation, Valid Action Set $\rightarrow$ IL Model $\rightarrow$ Amazon URL: Given the text observation and allowed of valid actions, the IL model produces an action. This action is then used to construct a corresponding Amazon URL via a set of mapping rules, and the loop is repeated. This continues until the model generates a "buy now" action, terminating the loop.

D.2 Sim-to-real Transfer Results

The resulting numbers in Table 5 closely cohere to the reported numbers of WebShop found in Figure 4, suggesting that the WebShop has promise for developing grounded agents that can operate on real web environments. Between the two websites, transfer to Amazon is better than eBay as we note that (i) eBay has a larger product gap from WebShop, e.g. some item categories like food are disallowed in eBay. (ii) the eBay search engine seems weaker, and would sometimes display no results for lengthy instructions. The following Table 11 is an example of a trajectory generated by the IL agent searching on the real Amazon website.

It is evident that the exploratory behavior and patterns learned and exhibited by the agent within the WebShop environment is not lost in this transfer. These results point to the opportunity for sim-to-real trained agents to transfer to other real-world web tasks despite the domain shift in both data (products) and dynamics (search engine) With that said, the gap between human and model performance also encourage us to look into expanding on the current limitations in our work regarding both the model and the WebShop environment.

Appendix E Potential Societal Impacts and Limitations

WebShop is designed to minimize human efforts in data collection and processing, but there are still potential concerns regarding diversity, fairness, and representation. Developing RL agents that interact with the web also bear safety concerns, especially when transferring from simulation to real-world websites. We also discuss other limitations regarding the semantics of current task (instruction/reward).

Diversity and representation in data collection. We chose five common categories from amazon.com and scrape all products using all subcategories to minimize bias. However, our data is still biased toward the website country (USA) and website language (English), and may only represent a subset of all possible products that users potentially want to buy. Having this limitation in mind, the design of WebShop allows the product data to be easily updated for different representations of real-world usage.

Bias in data processing. Currently our attribute labeling is manually done and may be biased by the labeller’s own experience (e.g. more knowledge toward product attributes like sports rather than makeup). An more automatic alternative would be to employ trained NLP models (e.g. relation extraction) to extract product attributes, which might be less biased than one labeller. Our reward design is general and could be updated to weight more toward attributes, options, price, etc.

Safety for developing web agents. Unlike recent work that directly employs agents on the World Wide Web (WWW), WebShop aims to provide a realistic simulation environment to train agents in a controllable and safe manner. In our preliminary sim-to-real experiments, the agent could only update the current webpage’s url in two fixed and safe ways (i.e. search for results, open an item), and any form sending action (e.g. click options or buy) is held within the sim-to-real interface for later reward calculation. As a result, only navigation is done on the real-world website. For future deployment to real-world websites with more advanced functions, we believe a good specification of possible model behaviors is key to avoid harmful actions.

Limitations in the current task. Our current instructions are still limited by the attributes and options used. While attributes are simple and sometimes too generic (e.g. “easy to use”), the options might get too specific (e.g. “d17(dedicated right, back)”). Therefore, an agent might sometimes use a special option as cues to find the product, while ignoring other parts of the instruction. To better leverage images and texts (including reviews written by human users, which are not used in current work) of products for more semantic and challenging instructions is an important future direction from WebShop.

Instruction for Human Trajectory Collection

The following pages display the human trajectory collection document mentioned in §A.6.

See pages 1- of figure/human-traj-inst.pdf