SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation

Introduction

Today, millions of users engage with spreadsheets across various fields, including management, marketing, and finance . Assisting users with spreadsheet manipulation can significantly reduce their workload, thereby enhancing productivity and efficiency . Initially, program synthesis aided spreadsheet manipulation , followed by deep learning methods for automated manipulation . More recently, leveraging the powerful capabilities of LLMs , spreadsheet agents such as SheetCopilot and SheetAgent have been developed to assist people in manipulating spreadsheets via natural language instructions. These agents employ LLMs to transform instructions into solutions, typically presented as executable code for spreadsheet operations.

However, current spreadsheet-related agents are still far from being truly helpful assistants for automating spreadsheet tasks. Users of Copilot and similar agents find these tools useful for reducing the effort of searching online, but they do not necessarily improve task completion time or success rates . A critical reason for this shortcoming is that the benchmarks used to evaluate these agents do not truly reflect real and challenging user demands. The primary limitations could be summarized as follows.

First, these benchmarks either use the self-instruct technique to synthesize queries based on a limited set of real-world spreadsheet manipulation demands , or they rely on crowdworkers to manually create instructions for given spreadsheets based on a few manipulation examples . As illustrated in the left part of Figure 1, real user questions collected from online forums (e.g., excelforum.com) differ significantly from synthetic queries, presenting complex demands that often require additional context, such as the users’ previous attempts and encountered issues.

Second, the spreadsheets used in these benchmarks are overly simplified , typically containing only one regular relational table, similar to the tables used in TableQA tasks and the tables in databases of Text2SQL tasks . While both spreadsheets and relational tables store tabular data in a two-dimensional structure, there are significant differences that make spreadsheets more challenging to handle. As shown in the right part of Figure 1, spreadsheets offer a flexible way of organizing data without distinct table boundaries or structures, permitting the inclusion of multiple tables in non-standard formats—such as nested, incomplete, or missing headers—and the presence of free-form text within a single cell. Additionally, spreadsheets support various non-textual elements (e.g., color, bold) compared to the relational tables in databases or the CSV/JSON format tables in TableQA tasks . These distinct features are not fully utilized in current benchmarks.

Third, these benchmarks for LLMs typically involve a single test for each instruction, where the predicted answer, derived from the solution generated by the LLMs applied to the given spreadsheet, is compared to the ground truth answer . This evaluation, however, may result in false positive solutions that are tailored to the specific spreadsheet and do not generalize to other spreadsheets with varying values. A robust solution from LLMs should be able to manage multiple spreadsheets, even corner cases, as inputs, apply the solution, and consistently yield correct answers.

To address the identified need, we introduce a challenging benchmark called SpreadsheetBench. Our benchmark features complex user instructions alongside spreadsheets that are flexibly organized by real-world users. These resources are gathered exclusively from four online forums for addressing Excel problems, which are highly ranked by several reputable websites including general Google Search, Feedspot.com—known for its forum search functionality—and Deskbright.com, a site focused on online education, particularly in Microsoft Excel. The instructions often require detailed contextual information for clarification, including descriptions of attempted solutions, incorrect answers received, and the issues encountered. The spreadsheets exhibit diverse formats, potentially featuring multiple tables within a single spreadsheet, non-standard tables that extend beyond conventional relational structures, cells with only textual content, and other non-textual elements.

We also propose an Online Judge (OJ)-style metric for more reliable benchmarking, adapted from coding skill assessments . This metric is well-suited for spreadsheet manipulation tasks where LLMs are tasked with generating code solutions. Within our benchmark, each instruction is associated with multiple input-output spreadsheets serving as test cases, sharing a similar structure but differing in data. A code solution must pass all these test cases to ensure evaluation reliability and accuracy.

Conclusively, we have developed SpreadsheetBench comprising 912 instructions and 2,729 test cases, with an average of three test cases per instruction. These instructions cover 10 primary manipulation categories and involve spreadsheets with tables extending beyond 100 columns and 20,000 rows. Moreover, 35.7% of the spreadsheets contain multiple tables, and 42.7% feature non-standard relational tables. This benchmark is notable for its real-world instructions, diverse spreadsheet formats, and a comprehensive testing strategy.

We conduct a thorough evaluation of various types of models, including commonly studied TableQA models, open-source LLMs for general tasks and coding tasks, the most advanced close-source models, and spreadsheet-specific LLMs. The performance of these models vary widely, with scores ranging from 0.05% to 23.65% according to our proposed OJ-style evaluation metric. Notably, some methods score as low as 0%, highlighting the benchmark’s difficulty. The results also suggest the importance of improving coding abilities within LLMs for spreadsheet manipulation, and indicate that multi-round prompting, allowing LLMs to read spreadsheet data and error feedback from a code compiler, can increase the probability of correct responses.

SpreadsheetBench

In this section, we begin by outlining the task formulation, followed by an introduction to the benchmark construction pipeline and the presentation of resulting benchmark statistics. Finally, we introduce the proposed OJ-style evaluation metric.

We denote the dataset of a benchmark as $\mathcal{D}=\{(q_{i},c_{i},a_{i})\}$, where $q_{i}$ represents an instruction, $c_{i}$ denotes a spreadsheet, and $a_{i}$ is the corresponding answer of $q_{i}$ on $c_{i}$. The objective of spreadsheet manipulation is to enable an LLM to generate a code-based solution $s_{i}$, apply it to $q_{i}$ to obtain a predicted answer $\hat{a}_{i}$, which can then be compared with the ground truth answer $a_{i}$.

In this benchmark, an instruction $q_{i}$ involves the modification of either several cells or the entire sheet in a spreadsheet file, which we refer to as cell-level and sheet-level manipulation, respectively.

The spreadsheet $c_{i}$ stores tabular data in a two-dimensional format, offering a flexible way for data organization without a distinct table boundary or structure. A single sheet in a spreadsheet file can contain multiple tables in diverse formats, free-form textual information, and non-textual elements.

An answer $a_{i}$ refers to the modified spreadsheet resulting from the application of a solution, which is expected to be a piece of code generated by LLMs. We generate code as solutions, as the goal is to manipulate spreadsheets rather than query information from them. Note that $\mathcal{D}$ does not include solutions because we allow for various solutions by LLMs, with our focus solely on the correctness of final answers after applying the solutions.

2 Benchmark Construction

The construction of the benchmark follows a four-stage pipeline: 1) data sourcing, 2) data filtering, 3) data formatting, and 4) test case construction, as illustrated in Figure 2. Subsequently, we detail the perturbation techniques used to prevent data leakage.

1. Data Sourcing. Our objective is to collect high-quality and representative spreadsheet manipulation questions from real-world sources. As depicted in the initial section of Figure 2, we utilize general Google Search, the specialized forum search engine Feedspot.com, and the Microsoft Excel-focused online learning platform Deskbright.com to identify four popular and regularly updated forums and blog sites—ExcelForum, Chandoo, MrExcel, and ExcelGuru—as our data sources (detailed information available in Appendix B.1). We specifically target posts that fall under categories such as General Questions, Formula, VBA & Macros, Formatting, Pivot Table, and Power Query to ensure relevance to spreadsheet manipulation.

2. Data Filtering. Since the original posts in forums are diverse with various questions, as shown in the second part of Figure 2, we select the question by four main criteria: 1) is solved, 2) solely pertains to spreadsheet manipulation, 3) is feasible and testable, and 4) is representative.

Initially, we directly identify questions tagged as “solved” from all posts. For posts without explicit tags, we employ GPT-4 to determine their solved status, as patterns such as thank-you notes are often observed at the end of a post indicating the question is solved.

Next, we focus on extracting questions strictly related to spreadsheet manipulation. Most popular spreadsheet forums primarily discuss Excel-related issues involving software functions like input boxes, buttons, and forms. Unlike InstructExcel , our goal is to establish a software-independent benchmark centered solely on spreadsheet manipulation. Consequently, we discard all irrelevant software-specific questions using keywords such as “input boxes”, “buttons”, and “forms”, and engage annotators to check the remaining posts.

Finally, we select questions that are both feasible and representative. To ensure feasibility, aided by annotators, we exclude posts without spreadsheet attachments, those with excessive replies, or ambiguous presentations. To identify representative posts, we filter those with high view counts and ask the annotators to retain both simple and difficult questions.

3. Data Formatting. According to the task definition outlined in Section 2.1, it is necessary to formulate the instruction, the spreadsheet, and the answer in a manner that facilitates convenient and accurate automatic evaluation.

Instruction Generation. The original posts are often cluttered, with the description of the user question dispersed across multiple replies. Therefore, we need to extract and condense these into a self-contained instruction from all replies. Initially, we utilize GPT-4 to recreate a coherent instruction from the original post. Annotators then verify this instruction to resolve any issues arising from possible incomplete context extraction from the post.

Answer Position Annotation. Regarding solution derivation, forum users typically discuss solutions in formulas or VBA code. Annotators are responsible for identifying the solution from the replies and applying it to the corresponding spreadsheet to derive the answers. However, discrepancies in the placement of answers by annotators compared to those predicted by LLMs can lead to mistaken evaluations. To mitigate this issue, we allow annotators to incorporate specific constraints in the instructions that annotate the answer position. As described in Section 2.1, answer positioning should consider both cell-level and sheet-level manipulation. For cell-level manipulations, annotators should specify exact cell positions, such as “D2:D6”. For sheet-level manipulations, the entire range of the spreadsheet, such as “A2:D6”, needs to be included in the instruction.

4. Test Case Construction. According to observations on online forums, it is evident that some solutions provided by forum users may be applicable only to the specific example spreadsheet provided, potentially failing when applied to other scenarios with slight variations in the spreadsheet data. To enhance the reliability of user-provided solutions, it is essential to create multiple spreadsheets and develop multiple test cases for each instruction. This preparation is essential for the subsequent OJ-style evaluation metric.

Specifically, for instructions that solely offer a solution without derived answers from forum posts, we derive the answer for the original provided spreadsheet by manually manipulating the input spreadsheet using the given solution presented by formulas or VBA code. The original spreadsheet, along with the derived answer, forms an input-output test case for the current instruction.

Subsequently, annotators are requested to modify the data in the original spreadsheet twice to generate two distinct spreadsheets. By applying the same solution to each of these modified spreadsheets, we can obtain two corresponding answers. It’s important to note that annotators do not make casual alterations to the spreadsheet data; instead, they are instructed to concentrate on real-world corner cases (e.g., cells that should contain zeros but remain empty). Annotators are also instructed to make valid modifications consistent with the question’s requirements.

Ultimately, for each instruction suitable for multiple test cases, we derive three test cases from the original single spreadsheet-answer data, as illustrated in the fourth part of Figure 2. Then $\mathcal{D}=\{(q_{i},c_{i},a_{i})\}$ is changed to $\mathcal{D}=\{(q_{i},\{(c_{ij},a_{ij})\})\}$.

Against Data Leakage. Datasets initially obtained from online forums may be susceptible to data leakage issues, given that many LLMs are pre-trained using a vast corpus of web text .

The above mentioned strategies, along with some new approaches, can introduce perturbations to the collected data to mitigate the data leakage problem:

Instruction Generation: The aforementioned instruction generation strategy, carried out by GPT-4 and human annotators, involves revising the original questions in the posts, thereby preventing LLMs from memorizing the original questions.

Spreadsheet Modification: The test case construction strategy outlined above involves modifying the original provided spreadsheets, preventing LLMs from memorizing the original spreadsheets.

Answer Position Changing: We also alter the position of the tabular data in the original spreadsheets and the corresponding answer in the resulting spreadsheets. By doing so, the originally provided solution from the posts cannot be directly used to derive answers with changed positions, thus preventing LLMs from memorizing the provided solution from the posts.

3 Benchmark Statistics

Data Statistics. We have developed SpreadsheetBench comprising 912 instructions and 2,729 test cases, with an average of three test cases per instruction. We analyze it and derive several observations, as depicted in Figure 3. In Figure 3(a), the instructions in our benchmark cover a broad spectrum of spreadsheet manipulation types, including find, extract, sum, highlight, remove, modify, count, delete, calculate, and display. The outer circle illustrates the objects being manipulated, showcasing the diversity of the benchmark.

Figures 3(b) and (c) display the distributions of row and column sizes in our spreadsheet files, revealing long-tailed distributions. Specifically, 80% of spreadsheets have a row size spanning from 1 to 49, and a column size spanning from 1 to 13. It is noteworthy that the row or column size in the long tail part is substantial, indicating the challenging nature of our benchmark.

Figure 3(d) demonstrates that more than one-third of spreadsheets contain multiple tables in one sheet. Figure 3(e) reveals that nearly half of the spreadsheets contain non-standard relational tables with nested, incomplete or missing headers. These observations indicate that the SpreadsheetBench significantly increases the complexity of understanding and manipulating the spreadsheets.

Comparison with Previous Benchmarks. Table 1 compares SpreadsheetBench with previous spreadsheet manipulation benchmarks. In addition, numerous benchmarks exist for table-related tasks. These include table question answering (TableQA), Table-to-text, and table data analysis . We have not included benchmarks for these table-related tasks, as their instructions are specifically tailored for QA and use CSV/JSON files as input, which significantly diverges from tasks that involve spreadsheet manipulation.

SpreadsheetBench is sourced exclusively from real-world data and exhibits a higher average word count per instruction. We have collected a larger number of spreadsheet files, including both single-sheet and multiple-sheet formats in each file. Our spreadsheet files contain multiple sheets with non-standard relational tables and multiple tables within a single sheet. Real-world questions often involve additional explanations within the spreadsheet, a characteristic not present in previous benchmarks. Furthermore, we employ OJ-style evaluation metrics with three test cases per instruction.

The benchmarks SheetCoplitBench and SheetRM generate instructions using LLMs’ self-instruct techniques, which are quite different from users’ actual queries. Furthermore, the manipulated spreadsheets in these benchmarks are relatively simple, seldom featuring multiple sheets and tables, and non-standard relational tables. On the other hand, InstructExcel offers a more comprehensive benchmark, particularly with complex spreadsheet files that cover various features. Even though the instructions in InstructExcel are created by annotators, they remain relatively simple. The average word count per instruction is only 9.8, compared to 85.7 in our benchmark. Additionally, InstructExcel does not include additional explanations or multiple test cases for the instructions, making it less complex and potentially less reliable than our benchmark.

4 Evaluation Metrics

Exact match is a commonly employed metric for spreadsheet manipulation tasks , but it may only work for the current spreadsheets and may not be robust to variations in the data within the spreadsheet. To address this, we adopt an approach similar to the online judge system commonly used for assessing coding capability . We prepare multiple spreadsheet files as test cases for each instruction, as mentioned in Section 2.2, and employ a similar OJ-style metric to ensure the reliability of the evaluation process. Figure 2 illustrates this process, where an LLM is expected to generate a solution given an instruction and a spreadsheet test case. Subsequently, the solution is applied to multiple spreadsheet test cases for OJ-style evaluation. In contrast to some benchmarks that perturb data in tables to create new data samples and perform one model inference for each data sample , we only require one model inference to generate a solution for all test cases of each instruction.

We establish two distinct scoring criteria to calculate the final score. The soft restriction adheres to the scoring principles of the OJ system from the IOI, granting partial credit when a solution only passes some test cases . The calculation is as follows:

The hard restriction follows the ICPC scoring rules of the OJ system, where no partial credit is awarded . The score is determined as follows:

In the above equations, $T_{i}$ represents the test cases (i.e., $\{(c_{ij},a_{ij})\}$) of the $i$-th instruction, and $r_{ij}$ indicates the result of the model’s solution applied on the $j$-th test case, respectively. Specifically, $r_{ij}$ is marked as Accept (ACC) if the applied solution’s answer on the $j$-th test case (i.e., the resulting spreadsheet) match the ground truth answer. The soft restriction metric does not penalize models for failing to provide a flawless solution that addresses every common and corner cases, whereas the hard restriction metric encourages models to strive for the most perfect solution.

Experiments

Baselines. We evaluate LLMs across five categories: (1) TableQA models, including fine-tuned TaPEx (based on BERT) , TaPas (based on BART) , and prompt-based Binder (GPT-3.5) ; (2) Open-source code models, including CodeQwen (7B) and DeepseekCoder (33B) ; (3) Open-source general models, including Mixtral 8x7B and Llama 3 (70B) ; (4) Close-source models, including GPT-3.5https://platform.openai.com/overview and GPT-4o ; (5) Spreadsheet-specific methods or products, including SheetCopilot and Copilot in Excelhttps://www.microsoft.com/microsoft-copilot/. We only sample a five percent data samples from our benchmark to evaluate the last category, as we can only manually evaluate these methods within their products, which is too costly and time-consuming to evaluate the entire benchmark.

Inference Setting. We evaluate LLMs under two distinct settings: 1. Single Round: In this mode, we present the model with the initial few rows of spreadsheet files within the prompt, allowing for only one inference. Since TableQA models can only produce textual answers, we conduct separate inferences for each test case and compare the resulting textual answers to the ground truth. For non-TableQA models, we instruct them to generate a code-based solution for all test cases of a given instruction. 2. Multi-Round: Building on the single-round prompt setting, we incorporate additional prompt that utilizes the ReAct technique and code execution feedback to enhance the accuracy of code solutions produced by LLMs over multi-round conversation. Specifically, we offer LLMs two choices: to generate code for retrieving spreadsheet file contents or to provide the ultimate solution. For the first choice, we supply code execution outcome to the subsequent round. In the second, the conversation concludes once the desired spreadsheet file is produced. In both instances, we furnish error feedback if the code fails to execute, enabling the model to refine its code in subsequent iterations. We impose a limit of five rounds for the multi-round setting.

Overall performance. The results shown in Table 2 indicate that current LLMs and spreadsheet agents are inadequate in managing complex spreadsheet manipulation tasks as required by real-world scenarios. Even the most advanced spreadsheet agent, Copilot in Excel, only achieves an accuracy of roughly 20%. GPT-4o, the SOTA LLM, scores around 17% in accuracy, aligning with Copilot in Excel’s performance. Open-source LLMs significantly underperform compared to the SOTA model, likely due to their limited comprehension and coding proficiency. Binder shows the poorest results, while TaPEx and TaPas, designed specifically for TableQA, score zero across all metrics (omitted from the table). This underscores the distinction in difficulty between TableQA and spreadsheet manipulation. Overall, there is a substantial gap between existing LLMs or products and human performance produced by Excel experts, emphasizing the critical need for advancement in LLMs tailored for spreadsheet manipulation.

Alongside the performance evaluation of each model, our findings include the following insights:

(1) Code Model vs. General Model. Given that spreadsheet manipulation solutions primarily rely on coding, LLMs tailored for coding, such as DeepseekCoder, exhibit superior performance compared to open-source general models like Llama-3 (70B). This highlights the need to enhance coding capacities within LLMs for effective spreadsheet manipulation.

(2) Single Round vs. Multiple Round. The majority of LLMs experience significant improvement in their capabilities through multi-round conversation, with some models achieving nearly a tenfold increase. However, GPT-4o’s performance slightly declines in a multi-round setting. This is due to GPT-4o’s adherence to instructions, which leads to duplicated content when retrieving the spreadsheet, as it includes the initial spreadsheet rows already provided in the prompt. Other LLMs, unable to follow the retrieval instruction, rely on the given initial rows. We provide the performance of GPT-4o without the initial spreadsheet rows in Appendix C.

(3) Soft Restriction vs. Hard Restriction. The shift from soft to hard restrictions leads to a modest decrease in LLM performance, suggesting that the solutions produced by these models may not remain effective when the spreadsheet content is altered. Given that real-world spreadsheet applications often involve frequent content changes, it is crucial to develop solutions that are robust to such modifications. The introduced OJ-style evaluation metrics are well-suited to assess such robustness.

Analysis. We also conduct analysis on the factors that influence the performance of LLMs on the spreadsheet manipulation task.

(1) Great spreadsheet complexity leads to diminished performance. Our benchmark is organized into four categories based on row and column size, the presence of multiple tables, and the inclusion of non-standard tables, resulting in eight distinct subsets, as detailed in Table 3. Our analysis reveals that model performance notably declines when dealing with spreadsheets that have extensive rows and columns, multiple tables, and non-standard structures.

(2) Increase row size, no significant performance gain. We assess the effect of row size in the prompt under a single round setting. Figure 4(a) shows that expanding input rows from 5 to 10 doesn’t notably enhance performance. This could be due to the excessive context length when processing additional rows.

(3) GPT4o does not benefit from the multi-round setting. As shown in Figure 4(b), the performance of GPT-3.5 improves as the round number increases from 2 to 4. However, GPT-4o, the current most advanced LLM, does not benefit from the multi-round setting. Our analysis shows that GPT-4o’s initial solutions have a substantially higher rate of executability and accuracy than other models. Consequently, the additional step of repeatedly retrieving spreadsheet content and executing code does not markedly improve GPT-4o’s performance.

Conclusion

We create a rigorous benchmark, SpreadsheetBench, for evaluating LLMs’ capacity in spreadsheet manipulating. Compared with existing benchmarks, SpreadsheetBench incorporates 912 authentic instructions sourced from online forums, more diverse spreadsheet files with rich formats, multiple tables, irregular tables, and more comprehensive testing with multiple test suites, ensuring better coverage of corner cases.

References

Appendix A Broader Discussion

The major limitation of SpreadsheetBench lies in data selection and test case construction process. (1) Data Selection: As part of our data selection process, we eliminate posts that lack an acknowledged response or are difficult to formalize. Although the proportion of these posts that contain valuable spreadsheet manipulation questions is low, they may also be appropriate for our benchmark under careful human annotation. (2) Test Case Construction: Our benchmark has a significantly higher number of questions compared to online judge competitions. Consequently, we did not meticulously devise corner cases for each question, taking into account all possible scenarios and edge cases. However, we still request annotators to remain vigilant regarding potential corner cases while annotating each question (See Figure 7 for an example).

A.2 Potential Impact

SpreadsheetBench are constructed based on real user queries and spreadsheet files. Our objective is to improve the proficiency of LLMs in comprehending and manipulating spreadsheets, with the ultimate aim of automating the spreadsheet manipulation process and reduce human workload in the future. Besides, our benchmark consists of tabular data and corresponding questions, making it suitable for evaluating the anbility of LLMs in understanding and manipulating structured data. Our benchmark exclude any content that could violate personal privacy, contain explicit material, depict violence, or involve other sensitive subjects during our human annotation process. Thus, we believe that the probability of our benchmark causing adverse effects on safety, security, discrimination, surveillance, deception, harassment, human rights, bias, and fairness is extremely minimal.

A.3 Ethical Consideration

In this section, we discuss the ethical considerations regarding our data construction process. (1) Data Risk Control: During our annotation process, we filter out the content in questions or spreadsheet files that is inappropriate for presentation to a general audience, such as contents that depict violence, sensitive subjects, etc. In addition, two leaders from the annotation team and two authors conduct an additional verification to ensure that our benchmark does not contain any instances of personally identifiable information. (2) Annotator Treatment and Consent: We recruit crowdsourced annotators for the construction of our benchmark (See B.2 for details). We have formalized employment agreements with all the annotators and remunerated them according to the predetermined wage criteria and working schedule. All employment agreements adhere to local regulations. (3) Copyright: Our benchmark is indirectly derived from posts on public Excel forums and blogs. The majority of the raw data is acquired from the ExcelForum website using Python crawler scripts, while the data from the remaining three websites is gathered through web browsers. To prevent copyright infringement, we initially consult the Robots Exclusion Protocolhttps://en.wikipedia.org/wiki/Robots.txt of the ExcelForum website. The Robots Exclusion Protocol implemented by the ExcelForum websitehttps://www.excelforum.com/robots.txt specifies that Bytespider is prohibited from accessing all directories, Mediapartners-Google is prohibited from crawling the xls files on the website, and all crawlers are prohibited from crawling the search.php directory. Our crawler follows this Robots Exclusion Protocol of the website. The crawler we use is not affiliated with Bytespider or Mediapartners-Google, and it does not crawl the search.php directory. Though we follow the Robots Exclusion Protocol, considering potential licensing and copyright risks, we will refrain from engaging in any secondary distribution of the raw data.

Appendix B Details of Data Collection

In this section, we introduce our data collection process. First, we present a concise overview of the four Excel websites that we use. Subsequently, we introduce the detail of our data annotation process. Finally, we show some data examples to intutively demonstrate the high quality of our data and the difference compared to previous benchmarks.

The data for our benchmark is obtained from four well-known Excel forums and blogs. These websites contain diverse spreadsheet-related questions in the real world and high-quality answers. Below is a brief introduction.

ExcelForumhttps://www.excelforum.com/ is one of the most popular spreadsheet forums in the world, containing 1 million threads, 5 million posts, and 1.3 million members as of May 2024. The highest number of concurrent online users recorded was 7,174. ExcelForum provides a detailed categorization of the spreadsheet-related problem. The main categories are Excel General, Excel Programming / VBA / Macros, Excel Formulas & Functions and Excel Charting & Pivots. The forum enforces strict rules to ensure the quality of threads, such as eliminating duplicate or ambiguous questions. Moreover, the use of AI tools (e.g., ChatGPT, GPT-4) to create forum questions and answers is not permitted. Although verifying AI-generated content can be challenging, the forum strives to minimize the presence of such content, which enhance the authenticity of the questions and the reliability of the answers.

MrExcelhttps://www.mrexcel.com/ is a widely recognized online platform that hosts more than 1 million discussion threads. Unlike ExcelForum, MrExcel does not employ a detailed classification of questions. Instead, most questions are posted in one category called Excel Questions, which contains over 1 million threads. MrExcel adopts a number of rules to ensure the quality of threads and the authenticity of the answers (i.e., the forbidden of AI tools).

ExcelGuruhttps://excelguru.ca/ is an online platform designed to enhance one’s understanding and proficiency in Excel and Power business intelligence (BI), which is an advanced function provided by Excel. While the forum section of ExcelGuru is relatively small, the article section offers a number of excellent blogs and articles focused on Excel pivot tables and power queries. Pivot tables and power queries are two powerful features that are used for data summarization, analysis, and reporting. They extract meaningful insights from one table and create a new table to present them. Each blog or article includes a complex question with a detailed solution, which is suitable for the data construction of our benchmark.

Chandoohttps://chandoo.org/ is famous for its exceptional blogs, which are characterised by their superior quality and frequent updates. On average, Chandoo publishes one blog per week, resulting in a grand total of over two thousand blogs. While the publication of blogs is less frequent than forum posts, they offer well-organized questions, answers in spreadsheet format, and detailed instructional guides, making them more convenient for data formatting and a valuable addition to our benchmark. The blogs on Chandoo covers a wide variety of topics, including VBA Macros, Excel Challenges, Formula Forensics, Excel Howtos, etc, which are all related to realistic user demands and advanced spreadsheet manipulation techniques. Each blog contains a detailed solution and corresponding Excel files.

B.2 Data Annotation

We conduct three key annotation tasks during the construction of SpreadsheetBench, including (1) Data Selection, (2) Data Formatting, and (3) Test Case Generation. We first present the composition of our annotation team, followed by a concise description of the main procedures utilized for these three annotation tasks.

Annotation Team. We employ a team of 20 individuals who specialize in Excel and have extensive experience in annotation. All team members hold a bachelor’s degree and are compensated in line with market rates. For data annotation, we allocate a predetermined quantity of data to annotators on a daily basis, and the allocation is modified in accordance with local regulations regarding work hours. For data validation, we hire two experienced validators with bachelor’s degrees for the first round of quality checks. Moreover, two authors who hold master’s degrees perform a secondary round of quality assessments to guarantee the high quality of data.

Annotation Process. We sequentially perform the three mentioned annotations and meticulously verify the data obtained from the previous annotation before each subsequent annotation to ensure the quality of the data.

For Task (1), we aim to select the data that is feasible and testable from a preliminary processed dataset. The preliminary processed dataset is obtained by using keywords and LLM to exclude data that are unrelated to spreadsheet manipulation, such as software usage questions (See Figure 12) and questions related to components in Excel, such as button, userform, etc (See Figure 13). The data selection pipeline is illustrated in Figure 16. First, we discard posts that do not include the spreadsheet files corresponding to the question, as these types of posts often focus on software usage or other questions that are not related to spreadsheet manipulation. Subsequently, we eliminate posts that lack an acknowledged solution. While certain posts may contain user-provided solutions, it is important to note that the questioner may not acknowledge or confirm whether these solutions adequately address their needs. Finally, we try to obtain the answer spreadsheet files corresponding to the solution. Some answerers provide both the solution and the answer spreadsheet files, and we can directly download these files as the initial answer. Others may only offer the solution (e.g., formula, VBA code), and we need to manually apply these solutions in the attached spreadsheet files of the problem. In this situation, we need to check whether the provided solution can be successfully executed. Otherwise, we will discard the posts that contain solutions that cannot be executed.

For Task (2), we aim to format the post into a self-contained question, an instruction type, an answer position, and a spreadsheet file corresponding to the question. This data formatting process transforms the raw post into testable data and is feasible for our evaluation metrics (See Figure 6 for an example). The annotation team follows the instruction in Figure 17 for the data formatting task. First, we need to do the question verification to check whether the questions summarized by LLM are complete and accurate. Second, we need to check the spreadsheet file corresponding to the question. This involves removing some of the text-based responses and relocating the tabular data to the appropriate location. Subsequently, we need to obtain the spreadsheet files corresponding to the solution, by applying the solution (e.g., formula, VBA code) to a specific position determined by the answerer. Finally, we need to annotate the content of the solution, the answer position, and statistical information related to the spreadsheet content, including whether the spreadsheet file contains multiple tables and non-standard relational tables.

For Task (3), we aim to generate two more test cases based on the original questions and the spreadsheet file. The annotation manual is shown in Figure 18. Annotators are asked to modify the position applied by the solution to change the values of cells in answer position. For solutions in formula format, the answer position change immediately once the position applied by the solution are modified. For other situation (e.g., VBA code solution), annotators need to rerun the solution to obtain the updated answer. Annotators must comprehend both the questions and the corresponding solution for each modification. Additionally, they are instructed to carefully consider the exceptional scenarios in the questions and are encouraged to alter the cell value to simulate this exceptional situation (See Figure 7 for an example).

B.3 Data Examples

As mentioned in Section 2.3, SpreadsheetBench incorporates complex questions based on real-world scenarios and diverse types of tables in spreadsheet files, compared to previous benchmarks. In this section, we present four data examples in SpreadsheetBench to illustrate the attributes of real-world problems and the challenging of our benchmark.

Figure 8 shows a cell-level manipulation data example which aims to extract a specific part of a text string from a column of cells. The instruction contains the demand of the user and an example manipulating action of the question, which rarely occur in synthetic instructions of the previous benchmarks. Furthermore, the table within the spreadsheet file is a non-standard relational table that lacks a complete table header. The final result is required to be filled in the cells from B3 to B14, which ensure the uniqueness of the answer.

Figure 9 shows a sheet-level manipulation data example aimed at creating a grid for a duty assignment schedule. The instruction contains the demand of the user, the current formula solution and the error encountered by the user. Besides, the spreadsheet file contains two relational tables in one sheet. The final result should be filled in a two dimensional position (i.e., F2 to T22).

Figure 10 shows a sheet-level manipulation data example aims to search across sheets based on a sheet name and two dates (i.e., first date and last date). The instruction contains a complex requirements, a detailed explanation, a manipulation example and the error encountered by the user. Moreover, the spreadsheet file contains multiple sheets, including the table to be manipulated (e.g., sheets named BUYING or SALES) and examples of possible answers (e.g., sheets named CASE1 or CASE2). The final result should be inserted into a two-dimensional location on the sheet named SH2.

Figure 11 shows a test case modified from the question in Figure 10. This test case is constructed following the user’s requirements in the instruction. The user expects to obtain a result that will search within a specific sheet name (cell G2). Only the rows with a date falling between the first date (cell C2) and last date (Cell E2) should be returned. Therefore, we modify the value of the sheet name (cell G2) to construct a new test case, as highlighted in yellow. This test case yields search results in the SALES sheet, rather than the original case that searches in the BUYING sheet. The values in the answer position also change, as highlighted in red. By modifying the sheet name, we can alter the sheet that is being searched, enabling a more comprehensive assessment of the solution’s effectiveness.

Appendix C Inference Details and Additional Experiments

In this section, we introduce the inference details and human evaluation settings of our main experiments. Subsequently, we carry out supplementary experiments on our multi-round inference setting and present a detailed analysis of our evaluation metrics.

The baseline models are divided into five types, including TableQA models, Open-source code models, Open-source general models and Spreadsheet-specific methods or products. For open-source models that need to be deployed on our own, we utilize PyTorch, transformers, and vLLM to load the model. We conduct experiments and obtain inference results on an Ubuntu 22.04 server with graphic cards that contain 8 NVIDIA A100 SXM 80GB GPUs. For spreadsheet-specific methods or products, we evaluate within Google Sheetshttps://www.google.com/sheets/about/ and Microsoft Excelhttps://www.microsoft.com/microsoft-365/excel on Windows 10 and macOS. For the detailed information of each baseline model, please refer to the website url in Table 4.

C.2 Details of Experimental Setup

All the baseline LLMs share a unified hyper-parameter (temperature to 1 and top_p to 1) on two inference settings. The prompt on single round setting and multiple round setting are shown in Figure 19 and Figure 20, respectively. For spreadsheet-specific methods or products including SheetCopilot and Copilot in Excel, two of the authors manually upload the spreadsheet files to Google Sheets or the cloud storage of Excel to evaluate the corresponding instructions. As a result of the financial and time costs, we limit our assessment to fifty instructions and one test case per instruction.

We hire four annotators who are experts in Excel to obtain the human performance of our benchmark. To minimize costs, we use a subset of fifty instructions and three test cases for each instruction to conduct our human evaluation. Annotators are asked to provide formula solutions or VBA code to finish each instruction instead of filling in the answer in the target position directly. The process of human evaluation is completed within one day.

C.3 Analysis of Inference Settings

The main result presented in Table 2 reveals an intriguing observation: the performance of GPT-4o in multi-round setting is slightly lower to that in single-round setting, which is contrary to the result of other models. In this section, we perform comprehensive experiments to determine the influence of various components in the multi-round setting on the outcomes. Furthermore, we perform multiple times of experiments for each inference setting to ensure the statistical significance of the results. Below is the introduction of four inference settings:

(1) Single + 5 Rows: This setting is identical to the single-round setting that we employed in the main experiments. We add five rows of the spreadsheet file in pandas dataframe format to the prompt and obtain the direct code solution of LLM in the single round setting.

(2) Multiple + Execution Feedback + 5 Rows: Compared to setting (1), LLMs is able to refine the code solution within five rounds if the solution fails to execute. The executor will provide the error traceback to the LLM to start the conversation of next round.

(3) Multiple + ReAct + Execution Feedback: Compared to setting (2), we introduce the ReAct mechanism to prompt LLMs to autonomously retrieve the content of the spreadsheet file, while excluding the five rows from the prompt.

(4) Multiple + ReAct + Execution Feedback + 5 Rows: This setting is identical to the multi-round setting that we employed in the main experiments. Based on the setting (3), we include the five rows of the spreadsheet file in the prompt.

The experiments are performed on a sample of 200 data points (21.9% of the entire benchmark) and we use GPT-3.5 and GPT-4o for evaluation. We conduct four runs for each setting and calculate the average performance, as illustrate in Table 5. The following findings have been discovered:

1. LLMs benefit from multi-round setting. As shown in Table 5, the performance of GPT-3.5 and GPT-4 improves in three multi-round settings when compared to the single-round setting. This may be attributed to either the ReAct mechanism, the feedback of code execution, or both, which we will determine later. Besides, the performance of GPT-4o on multi-round setting is higher than on single-round setting, which is inconsistent with the main result in 2. This is due to our practice of conducting four times of experiments and calculating the average result, which enhances the stability and reliability of our findings. We also enforce strict alignment between single-round and multi-round prompts by deleting one sentence that ask the LLM to answer carefully in the single-round prompt.

2. LLMs perform worse when adding ReAct mechanism. Comparing settings (2), (3), and (4), we find that using only the feedback from code execution achieves the highest performance. While the ReAct mechanism provide the model with an additional way to obtain spreadsheet information (instead of only 5 rows from the prompt), they are unable to effectively utilize this additional data. While the ReAct mechanism provides the model with an additional way to obtain spreadsheet information (beyond just the 5 rows from the prompt), it is unable to effectively utilize this additional data. Furthermore, existing LLMs lack the ability to make flexible decisions about whether to proceed with reading the content of the spreadsheet or which additional sections should be read. For instance, the model may continue to read the content of the first five rows of the spreadsheet even though this information has already been provided in the prompt. This phenomenon demonstrates that the current state-of-the-art LLM still lacks spreadsheet understanding and reasoning ability.

C.4 Analysis of Impact Factors

Based on the four inference settings introduced in Appendix 5, we conduct an analysis of two key factors in these settings, including the the number of rows provided in the prompt and the round number in multiple round setting.

The experiments are performed on a sample of 50 data points. For setting (1), (2) and (4), we modify the number of rows specified in the prompt to 1, 2, 5, and 10 rows, and analyze the impact on the performance of GPT-4o. For setting (2), (3), and (4), we modify the number of rounds of the interactive inference environment. The following findings have been discovered:

1. The performance of LLMs benefits from a limited number of rows. As illustrated in Figure 5(a), the performance of GPT-4o improves across all three settings as the row size increases from 1 row to 5 rows. However, in two of the settings, the performance declines as the row size increases to 10 rows. This phenomenon is consistent with our observation in the result of Appendix 5. LLMs fail to process and comprehend a relatively large number of rows, thus requiring improvements in LLMs for understanding tabular data.

2. The performance of LLMs increases with the number of rounds. As illustrated in Figure 5(b), GPT-4o shows an overall upward trend in performance as the number of rounds increases in all three settings. This result is inconsistent with the result in Figure 4(b), as we conduct the experiments multiple times and obtain an average result, which is much more stable. We also find that setting (3) is comparable to setting (4) when the number of rounds is four and five. This phenomenon illustrates that the current LLMs are incapable of effectively retrieving information beyond the first five rows of the spreadsheet in the ReAct mechanism.

C.5 Analysis of Evaluation Metrics

As our evaluation metrics are based on exact match of the cells in the answer position, we conduct a human evaluation to verify the reliability of the metrics. Specifically, we sample 50 instructions from the entire benchmark with three test cases of each and sample the corresponding inference result of GPT-4o. We follow DS-1000 and report the following four indexes:

(1) Test Case Level False Discovery Rate: Among all the test cases that successfully pass our automated evaluation, none of them are deemed incorrect by our annotator.

(2) Test Case Level False Omission Rate: Among all the test cases that fail our automatic evaluation, 3.8% of them are deemed correct by our annotator.

(3) Instruction Level False Positive Percentage: Among all the instructions, none of them have any incorrect sample predictions that pass our automatic metric.

(4) Instruction Level False Negative Percentage: Among all instructions, 4% instructions contain at least one correct sample prediction that fails to pass our automatic metric.

Our evaluation metrics do not misclassify accurate results as inaccurate ones, as we achieve a 0% error rate in indices (1) and (3). It may misclassify a small proportion of correct test cases to incorrect ones. This occurs because the code solution generated by LLMs may produce additional content beyond the intended answer (See Figure 14 and Figure 15). In general, our evaluation metric is reliable, as it accurately identifies correct results and only misclassifies a small fraction of correct test cases as incorrect.