AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning

Introduction

Language agents Wang et al. (2023a); Xi et al. (2023); Guo et al. (2024), which leverage the powerful reasoning capabilities Qiao et al. (2023b); Zhang et al. (2023) of Large Language Models (LLMs) to generate executable actions for observing the external world, have emerged as essential components of AI systems designed to address intricate interactive tasks Torantulino (2023); Osika (2023); Nakajima (2023); Tang et al. (2023); Xie et al. (2023). The process of endowing LLMs with such interactive capabilities is referred to as Agent Learning wherein planning Huang et al. (2024) plays a pivotal role, which is responsible for decomposing complex tasks Wei et al. (2022); Yao et al. (2023); Team (2023); Qian et al. (2023), invoking external tools Shen et al. (2023); Lu et al. (2023); Qin et al. (2023), reflecting on past mistakes Shinn et al. (2023); Madaan et al. (2023), and aggregating information from various sources to reach the final targets. There have been a lot of works Li et al. (2023); Shen et al. (2023); Hong et al. (2023); Talebirad and Nadiri (2023); Chen et al. (2023d, b) that directly prompt closed-source off-the-shelf LLMs to plan on particular tasks. Despite their convenience and flexibility, closed-source LLMs inevitably suffer from unresolved issues, as their accessibility often comes at a steep price and their black-box nature makes the result reproduction difficult. In light of this, some recent endeavors have shifted their focus towards imbuing open-source models with planning capabilities through fine-tuning Chen et al. (2023a); Zeng et al. (2023); Yin et al. (2023).

However, despite the achievements of the existing fine-tuning-based method, they are not without limitations. On the one hand, training open-source models necessitates a substantial amount of annotated task data and still relies on closed-source models to synthesize planning trajectories. However, fulfilling these requirements in many real-world scenarios, such as private personal bots or sensitive company business, often proves to be rocky. On the other hand, from the perspective of agent framework, fine-tuning-based methods compel one single language agent to learn all planning abilities, placing even greater pressure on them. These contradict Simon’s principle of bounded rationality (Mintrom, 2015), which states that “precise social division-of-labor and clear individual tasks can compensate for the limited ability of individuals to process and utilize information”.

To this end, we introduce AutoAct, an automatic agent learning framework, which does not rely on large-scale annotated data and synthetic trajectories from closed-source models while incorporating explicit individual tasks with precise division-of-labor (see Fig. 1). Given a limited set of user-provided data examples, AutoAct starts with a Meta-Agent to obtain an augmented database through self-instruct (Wang et al., 2023b). Then, armed with a prepared tool library, the Meta-Agent can automatically synthesize planning trajectories without any assistance from humans or strong closed-source models. Finally, we propose the division-of-labor strategy which resembles cell differentiation based on the self-synthesized trajectories (genes), where the Meta-Agent acts as a stem cell (Colman, 2008) and differentiates into three sub-agents with distinct functions: task decomposition, tool invocation, and self-reflection, respectively. Our differentiation process is essentially a parameter-efficient training process on the self-synthesized trajectories with low-consumption resources. We list the differences between AutoAct and prior works in Tab. 3.

Experiments on complex question-answering tasks with different LLMs demonstrate that AutoAct yields better or parallel performance compared to various strong baselines. Extensive empirical analysis demonstrates the effectiveness of our appropriate division-of-labor strategy.

AutoAct

The Meta-Agent is responsible for all the preparatory work before self-differentiation and serves as the backbone model for all sub-agents. Given limited target task information and a pre-prepared tool library, the Meta-Agent can differentiate into an agent group capable of collaborating to accomplish the target task. In AutoAct, the Meta-Agent can be initialized with any open-source model.

Target Task Information.

In this paper, we mainly focus on agent learning from scratch, which means the task information at hand is quite limited, primarily encompassing three aspects: task name $\mathcal{M}$, task description $\mathcal{P}$, task data examples $\mathcal{C}$. Concretely, $\mathcal{P}$ represents a detailed description of the task’s characteristics. $\mathcal{C}=\{q_{i},a_{i}\}_{i=1}^{|\mathcal{C}|}$ indicates $|\mathcal{C}|$ question-answer example pairs of the task, where $|\mathcal{C}|$ is very small which users can effortlessly provide (e.g., a few demonstrations). For a more in-depth view of task information, please refer to Appx. D. Note that the task information serves as the only user-provided knowledge of the task for AutoAct to conduct automatic agent learning.

Tool Library.

To facilitate our agents in automatic task planning, we provide a comprehensive tool library at their disposal. The tool library can be denoted as $\mathcal{T}=\{m_{i},d_{i},u_{i}\}_{i=1}^{|\mathcal{T}|}$, where $m$ represents the tool name, $d$ defines the tool functionality, $u$ details the tool usage instruction, and $|\mathcal{T}|$ stands for the tools amount of the library. In our automatic procedure, the Meta-Agent has the autonomy to select appropriate tools from the tool library based on the task information. Users also have the option to expand the tool library according to their specific needs, allowing for more flexible utilization. We list part of our tool library in Appx. E.

2 Starting from Scratch via Self-Instruct

To acquire a sufficient amount of task data and provide an ample training resource, it is necessary to augment the data based on the examples at hand. We accomplish this process through self-instruct. Initially, the database $\mathcal{D}$ is set to be equal to the task data examples $\mathcal{C}$, with $\mathcal{C}$ as the seed for data generation. In each round, the Meta-Agent generates new question-answer pairs by few-shot prompting, and the few-shot prompt examples are randomly sampled from $\mathcal{D}$. The generated data will be added to $\mathcal{D}$ followed by filtering, with the exclusion of format erroneous and duplicate data before its inclusion. Eventually, we obtain a database $\mathcal{D}=\{q_{i},a_{i}\}_{i=1}^{|\mathcal{D}|}$, where the number of data $|\mathcal{D}|$ satisfies $|\mathcal{D}|\gg|\mathcal{C}|$. The prompt we use for self-instruct can be seen in Appx. F.1 and we list some cases generated through self-instruct in Appx. G.

3 Automatic Agent Learning via Self-Planning

With the tool library at hand, we ask the Meta-Agent to select applicable tools for each task automatically. Specifically, we put $\mathcal{T}=\{m_{i},d_{i},u_{i}\}_{i=1}^{|\mathcal{T}|}$ in the form of a tool list as part of the prompt. Along with $\mathcal{T}$, the prompt also includes the task’s description $\mathcal{C}$. Finally, we instruct the Meta-Agent to select an appropriate set of tools $\mathcal{T}_{s}$ ($\mathcal{T}_{s}\subset\mathcal{T}$) to wait for synthesizing trajectories. The prompt we use for automatic tool selection can be seen in Appx. F.2.

Trajectories Synthesis.

Without depending on closed-source models, we enable the Meta-Agent to synthesize planning trajectories on its own. Equipped with $\mathcal{T}_{s}$, we instruct the Meta-Agent to synthesize trajectories in a zero-shot manner on the database $\mathcal{D}$ adhering to the format of Thought-Action-Observation as defined in Yao et al. (2023). In order to obtain high-quality synthesized trajectories, we filter out all the trajectories with $\texttt{reward}<1$ and collect trajectories with exactly correct answers ($\texttt{reward}=1$) as the training source for self-differentiation. The prompt for trajectories synthesis can be seen in Appx. F.3.

Self-Differentiation.

In order to establish a clear division-of-labor, we leverage synthesized planning trajectories to differentiate the Meta-Agent into three sub-agents with distinct functionalities:

Plan-Agent $\pi_{\rm plan}$ undertakes task decomposition and determines which tool to invoke in each planning loop (Eq. 2).

Tool-Agent $\pi_{\rm tool}$ is responsible for how to invoke the tool (Eq. 3) by deciding the parameters for the tool invocation.

Reflect-Agent $\pi_{\rm reflect}$ engages in reflection by considering all the historical trajectories and providing a reflection result (Eq. 4).

We assume that the planning loop at time $t$ can be denoted as $(\tau_{t},\alpha_{t},o_{t})$, where $\tau$ denotes Thought, $\alpha$ signifies Action, and $o$ represents Observation. $\alpha$ can be further expressed as $(\alpha^{m},\alpha^{p})$, where $\alpha^{m}$ is the name of the action, and $\alpha^{p}$ is the parameters required to perform the action. Then the historical trajectory at time $t$ can be signaled as:

Eventually, supposing that the prompts of target task information, planning format requirements, and the question are all combined as $\mathcal{S}$, the responsibilities of each sub-agent can be defined as:

where $\tau^{r}$ and $\alpha^{r}$ represent the thought and action of the reflection process respectively, and $\mathcal{H}$ is the planning history after finishing the answer. The trajectories can be reorganized based on the responsibilities above and fed to the Meta-Agent for self-differentiation. Our differentiation is a parameter-efficient fine-tuning process to achieve resource-efficient learning. Particularly, for each sub-agent, we train a specific LoRA Hu et al. (2022).

Group Planning.

At inference time, once the tool name $\alpha_{t}^{m}$ generated by the Plan-Agent is triggered at time $t$, the Tool-Agent is roused to decide the parameters $\alpha_{t}^{p}$ transferred to the specific tool. The return result of the tool is treated as the observation $o_{t}$ and handed to the Plan-Agent. After the collaboration between the Plan-Agent and Tool-Agent reaches a prediction, the Reflect-Agent comes to reflect on the history and provide a reflection result contained in the reflection action $\alpha^{r}$. If the reflection result indicates that the prediction is correct, the whole planning process ends. Otherwise, the Plan-Agent and Tool-Agent will continue the planning based on the reflection information. The specific sequence of the group planning process can be found in the example on the right of Fig. 2.

Experimental Setup

We evaluate AutoAct on HotpotQA Yang et al. (2018) and ScienceQA Lu et al. (2022). HotpotQA is a multi-hop QA task challenging for rich background knowledge, the answer of which is usually a short entity or yes/no. Following Liu et al. (2023), we randomly select 300 dev questions divided into three levels for evaluation, with 100 questions in each level. For HotpotQA, the $\texttt{reward}\in$ is defined as the F1 score grading between the prediction and ground-truth answer. ScienceQA is a multi-modal QA task spanning various scientific topics. We also divide the test set into three levels based on the grade, with 120 randomly sampled data in each level. Since ScienceQA is a multi-choice task, the $\texttt{reward}\in\{0,1\}$ is exactly the accuracy. Note that due to the limitations of LMs in generating images, for ScienceQA, during the self-instruct stage, we directly generate captions for the images instead.

Baselines.

We choose the open-source Llama-2 models Touvron et al. (2023) as the backbones of our Meta-Agent and sub-agents. The compared baselines include CoT Wei et al. (2022), ReAct, Chameleon Lu et al. (2023), Reflexion Shinn et al. (2023), BOLAA Liu et al. (2023), ReWOO Xu et al. (2023), FireAct Chen et al. (2023a). We detail each baseline in Appx. B. To ensure fairness, we maintain an equal training trajectory volume of 200 for FireAct and AutoAct (200 synthesized data). As Reflexion provides answer correctness labels during reflection but other methods including AutoAct do not, we test all the other methods twice and choose the correct one for evaluation. For all the prompt-based baselines, we uniformly provide two examples in the prompt.

Training Setups.

We fine-tune all our models with LoRA Hu et al. (2022) in the format proposed in Alpaca Taori et al. (2023). All the training and inference experiments are conducted on 8 V100 GPUs within 16 hours. We detail the hyper-parameters for training in Appx. B.

Results

As shown in Table 1, the 13b and 70b models consistently outperform various prompt-based baselines. The 70b model even surpasses the agent performance of GPT-3.5-Turbo, achieving a rise of $\uparrow$3.77% on HotpotQA and $\uparrow$6.39% on ScienceQA. The performance of the 7b model is comparable to other methods to some extent. Therefore, whether in a single-agent or multi-agent architecture, prompt-based methods relying on few-shot demonstrations fail to precisely customize the behavior of the agent, which is also supported by the fact that FireAct widely outperforms ReAct and BOLAA in the context of iterative planning. In addition, our investigation reveals a visible disparity in open-source models between the performance of many prompt-based planning baselines (relying on various external tools) and CoT (relying on the models’ intrinsic reasoning abilities). This discrepancy underscores the formidable challenge of unlocking planning capabilities by prompting.

Compare to Fine-tuning-based Agent Learning Baselines.

Further focusing on FireAct in Tab. 1, despite the aid of GPT-4, FireAct’s approach of assigning the entire planning task to a single model proves to be burdensome. As a result, its performance on ScienceQA even falls short compared to the prompt-based global planning method, Chameleon. AutoAct decouples the planning process and reaches a clear division-of-labor among sub-agents for group planning, resulting in an improvement than FireAct, with $\uparrow$5.77% on HotpotQA and $\uparrow$6.67% on ScienceQA with 70b model. Additionally, AutoAct achieves self-planning without relying on closed-source models and large-scale labeled datasets, which paves the way for automatic agent learning with open-source models from scratch. In ablation study (§4) and human evaluation (§5), we will further validate that the quality of trajectories synthesized by AutoAct is not inferior to FireAct trained on trajectories synthesized using GPT-4.

Single-agent Learning vs. Multi-agent Learning.

Under identical settings, multi-agent architectures generally exhibit better performance than single-agent (ReAct vs. BOLAA, FireAct vs. AutoAct), which aligns with Simon’s theory of bounded rationality. Seemingly contrary to expectations, despite being a single-agent architecture, Chameleon outperforms BOLAA (even FireAct on ScienceQA). However, we analyze that this can be attributed to the way it leverages tools. In Chameleon, the process of deciding tool parameters is considered a form of tool invocation, and specialized few-shot prompts are designed to guide the model through this process. From this aspect, Chameleon, despite nominally a single-agent architecture, exhibits features resembling a multi-agent one, which does not contradict our initial conclusion. Indeed, we can also explain from the perspective of optimizing objectives. Another well-known principle, Goodhart’s Law (Goodhart, 1984), states that “When a measure becomes a target, it ceases to be a good measure”. This implies that optimizing one objective on the same agent will inevitably harm other optimization objectives to some extent. Therefore, it is not optimal to optimize all objectives on a single agent, and a multi-agent architecture happens to address this issue. However, we analyze in §5 that excessive fine-grained division-of-labor is not the best approach.

Approach Ablations.

Tab. 2 presents the performance of AutoAct on the 70b model after removing certain key processes. It can be observed that the least impactful removal is the - reflect. We investigate that in the zero-shot scenario, the model tends to be over-confident in its answers. It typically only recognizes its errors when there are obvious formatting mistakes or significant repetitions in the planning process. Consistent with previous findings, the removal of the - multi agents leads to a noticeable decrease in performance. A more exciting discovery is that the results of - multi are comparable to those of FireAct. This indirectly suggests that the trajectory quality generated by the 70b model may be no worse than that of GPT-4. As expected, the performance deteriorates after - fine-tuning, which once again confirms the previous conclusion. To demonstrate the necessity of filtering out planning error data, we specifically remove the filtering process (- filtering) to examine the performance of AutoAct. The results indicate that the damage caused by training on unfiltered data is even greater than that of - fine-tuning.

Analysis

We evaluate the influence of different training data scales on the performance of self-planning on HotpotQA in Fig. 3 (a-c). It can be observed that the overall performance of different models goes to stability with minimal waves once the data scale exceeds 200. We speculate that this may be due to the limited ability of naive self-instruct to boost internal knowledge of the language model. As the training data increases, the knowledge which can be extracted through self-instruct decreases. Despite our efforts to filter out duplicate data, the mindless increase can inevitably lead to a significant surge in similar data, which undermines the benefits of increasing the data scale and makes it challenging to improve model performance or even leads to over-fitting. To further confirm the role of training data, we decouple the models from the training data and evaluate their training results on trajectories synthesized by stronger models. From Fig. 3 (d-f), we can see consistent conclusions with previous findings. Therefore, maximizing the diversity of the synthesized data in the database may be a key improvement direction for AutoAct and we leave this for our future work.

Moderate division-of-labor benefits group planning performance.

To explore the impact of different granularity of self-differentiation, we further subdivide the tool agent, assigning dedicated agents to manipulate each specific tool. We compare the performance of One agent, Three agents (AutoAct), and the Tool-Specified setting on HotpotQA in Fig. 4. It can be observed that excessive differentiation (Tool-Specified) not only fails to achieve better results but can sometimes even be less effective than not differentiating (One) at all. This is consistent with the findings in Qiao et al. (2023a) which indicate that multi-tool joint learning often outperforms single-tool individual learning. Moreover, it appears that the performance loss of tool-specific agents compared to AutoAct is more significant on harder problems. This is because challenging problems typically require more planning steps and higher levels of collaboration among tools. By unifying tool invocations under one agent, it becomes possible to effectively learn the interconnectedness between tools, thereby compensating for potential information gaps arising from using tool-specific agents. Note the difference from Li et al. (2024), here we are discussing the granularity of division-of-labor among agents with different responsibilities, rather than the voting quantity among mutually equal agents.

Human Evaluation.

To get a deeper understanding of the capability of AutoAct, we manually compare the quality of trajectories generated by different methods from the number of planning rounds, the logical correctness of thoughts, action types, action parameters, and overall coherence. The detailed human evaluation process can be found in Appx. C. The evaluation results are depicted in Fig. 5&6. We can observe a clear advantage for AutoAct over other methods in the action type and action parameters. This indicates that decoupling the missions of planning and tool invocation can lead to better performance for both, alleviating the overwhelming pressure on a single agent. A more intuitive comparison can be observed in Fig. 5 (a)(b). AutoAct successfully addresses the failure in ReAct by employing a more scientific combination of tools and making more accurate tool invocations. Furthermore, AutoAct tends to consume more planning rounds than other methods. This allows AutoAct to perform better on harder problems. However, this characteristic can be a double-edged sword when it comes to simple problems. A surprising aspect is that AutoAct can validate its inner answers by continuing more rounds of verification (Fig. 5 (c)). But this can also lead to a longer context, gradually deviating AutoAct from the original question (Fig. 5 (d)).

Related Work

The rise of LLMs has positioned them as the most promising key to unlocking the door to Artificial General Intelligence (AGI), providing robust support for the development of LLM-centered AI agents Wang et al. (2023a); Xi et al. (2023); Wang et al. (2023c, d). Related works focus primarily on agent planning Yao et al. (2023); Song et al. (2022); Chen et al. (2023a), external tools harnessing Patil et al. (2023); Qiao et al. (2023a); Qin et al. (2023), collective intelligence among multi-agents Liang et al. (2023); Liu et al. (2023); Chen et al. (2023c), etc. However, despite their success, existing methods still face two major troubles. Firstly, most agents heavily rely on prompts for customization, which makes it difficult to precisely tailor the behavior of the agent, resulting in unexpected performance at times. Secondly, each agent is compelled to master all skills, making it challenging for the agent to achieve expertise in every domain. In response, our approach leverages a proper division-of-labor strategy and fine-tuning each sub-agent to equip different agents with distinct duties. These agents collaborate to accomplish tasks orderly and effectively.

Agent Fine-Tuning.

Despite the vast interest in LLM-powered agents, the construction of agents through fine-tuning has received limited attention. Most early works concentrate on fine-tuning to optimize the model’s reasoning capabilities Liu et al. (2022); Fu et al. (2023) or tool proficiency Patil et al. (2023); Qiao et al. (2023a); Qin et al. (2023). Recently, more works have emphasized endowing open-source LLMs with agent capabilities through fine-tuning Chen et al. (2023a); Zeng et al. (2023); Yin et al. (2023); Shen et al. (2024). However, these works suffer from at least one of the following issues: i) the requirement of one single model to be a generalist, ii) the need for a large amount of annotated data, iii) the need for trajectory annotation of closed-source models. Our approach enables the Meta-Agent to synthesize trajectories and achieve a division-of-labor strategy in a zero-shot manner, without relying on closed-source models.

Conclusion and Future Work

In this paper, we propose AutoAct, an automatic agent learning framework that does not rely on large-scale annotated data and synthetic trajectories from closed-source models, while alleviating the pressure on individual agents by explicitly dividing the workload. Interesting future directions include: i) expanding AutoAct to more realistic task scenarios (Puig et al., 2018; Zhou et al., 2023a; Xie et al., 2024), ii) boosting more knowledge via self-instruct (as analyzed in §5), iii) iteratively enhancing synthetic trajectories via self-improvement (Huang et al., 2023; Aksitov et al., 2023).

Limitations

In this paper, we focus on constructing an automatic agent learning framework dubbed AutoAct. Despite our best efforts, this paper may still have some remaining limitations.

For experimental convenience, we only evaluate our method on complex question-answering tasks with the Llama-2-chat model series. However, there are many other interactive scenarios and backbone models beyond these. Other complex tasks include web Yao et al. (2022); Zhou et al. (2023a), household Puig et al. (2018); Shridhar et al. (2021), traveling Xie et al. (2024), robotics Ichter et al. (2022), etc., and more backbone models include Vicuna Zheng et al. (2023), ChatGLM Du et al. (2022), Mistral Jiang et al. (2023), etc. We plan to conduct further research on applying AutoAct to a wider range of tasks and backbones in the future.

Boosting Knowledge via Self-Instruct.

As analyzed in §5, the planning performance of AutoAct can be limited by the model’s ability to access internal knowledge through self-instruct. While the current phenomenon allows us to achieve lightweight self-differentiation in terms of parameters and data, it is still necessary to research how to enrich knowledge as much as possible within the constraints of limited data.

Self-Improvement.

Recent research has shed light on self-improvement techniques that enhance LLMs by iteratively training them on self-synthesized data Zelikman et al. (2022); Huang et al. (2023); Gülçehre et al. (2023); Aksitov et al. (2023). This approach allows the model to continually learn and refine its performance on its own. Our approach also involves training on self-synthesized data and we believe that further using the iterative thinking of self-improvement will significantly enhance the performance of our method.

Ethics Statement

This research was conducted with the highest ethical standards and best practices in research. All our experiments use publicly available datasets (as detailed in §3), avoiding ethical concerns related to privacy, confidentiality, or misuse of personal biological information. The human evaluation process (as detailed in Appx. C) was carried out strictly with fairness and transparency. Consequently, this research is free from any ethical concerns.

References

Appendix A Comparison with Related Works

Appendix B Baselines and Training Setups

We choose the open-source Llama-2 models Touvron et al. (2023) as the backbones of our Meta-Agent and sub-agents. The compared baselines are as follows: 1) CoT Wei et al. (2022), the naive Chain-of-Thought reasoning method. 2) ReAct Yao et al. (2023), a well-known single-agent framework based on few-shot learning that performs planning and action iteratively. 3) Chameleon Lu et al. (2023), another few-shot single-agent framework that performs planning before action. 4) Reflexion Shinn et al. (2023), a single-agent framework to reinforce language agents through linguistic feedback. 5) BOLAA Liu et al. (2023), a multi-agent framework that customizes different agents through prompts. 6) ReWOO Xu et al. (2023), a multi-agent framework that decouples reasoning from observations. 7) FireAct Chen et al. (2023a), a single-agent framework with fine-tuning on diverse kinds of trajectories generated by GPT-4 OpenAI (2023). 8) GPT-3.5-Turbo OpenAI (2022). To ensure fairness, we maintain an equal training trajectory volume of 200 for FireAct and AutoAct (200 synthesized data). As Reflexion provides answer correctness labels during reflection but other methods including AutoAct do not, we test all the other methods twice and choose the correct one for evaluation. For all the prompt-based baselines, we uniformly provide two examples in the prompt.

Training Setups.

We fine-tune all our models with LoRA Hu et al. (2022) in the format proposed in Alpaca Taori et al. (2023). Our fine-tuning framework leverages FastChat Zheng et al. (2023) using DeepSpeed Rasley et al. (2020). We detail the hyper-parameters for training in Table 4.

Appendix C Detailed Process of Human Evaluation

To get a deeper understanding of the capability of AutoAct, we manually compare the quality of trajectories generated by different methods from five aspects. We ask five NLP volunteers to individually select the optimal trajectories generated by all methods in terms of the number of planning rounds, the logical correctness of thoughts, action types, action parameters, and overall coherence. The final results are determined based on major votes. During the evaluation, it is hidden for the evaluators of the correspondence between the trajectories and the methods. We delete the reflection-related parts from the trajectories generated by AutoAct and randomly shuffle the order of trajectories of each method in each data to minimize the potential bias as much as possible.

Appendix D Task Information

Task Name: HotpotQA Task Description: This is a question-answering task that includes high-quality multi-hop questions. It tests language modeling abilities for multi-step reasoning and covers a wide range of topics. Some questions are challenging, while others are easier, requiring multiple steps of reasoning to arrive at the final answer. Task Data Examples: Question: From 1969 to 1979, Arno Schmidt was the executive chef of a hotel located in which neighborhood in New York? Answer: Manhattan Question: Are both Shangri-La City and Ma’anshan cities in China? Answer: yes Task Name: ScienceQA Task Description: This is a multimodal question-answering task that necessitates a model to utilize tools for transforming image information into textual data. Simultaneously, this task incorporates substantial background knowledge, requiring the language model to acquire external information to enhance its comprehension of the task. Task Data Examples: Question: Which of these states is the farthest north? Options: (A) West Virginia (B) Louisiana (C) Arizona (D) Oklahoma Caption: An aerial view of a painting of a forest. Answer: A. West Virginia Question: Identify the question that Tom and Justin’s experiment can best answer. Context: The passage below describes an experiment. Read the passage and then follow the instructions below. Tom placed a ping pong ball in a catapult, pulled the catapult’s arm back to a 45 angle, and launched the ball. Then, Tom launched another ping pong ball, this time pulling the catapult’s arm back to a 30 angle. With each launch, his friend Justin measured the distance between the catapult and the place where the ball hit the ground. Tom and Justin repeated the launches with ping pong balls in four more identical catapults. They compared the distances the balls traveled when launched from a 45 angle to the distances the balls traveled when launched from a 30 angle. Figure: a catapult for launching ping pong balls. Options: (A) Do ping pong balls stop rolling along the ground sooner after being launched from a 30-angle or a 45-angle? (B) Do ping pong balls travel farther when launched from a 30-angle compared to a 45-angle? Caption: A wooden board with a wooden head on top of it. Answer: B. Do ping pong balls travel farther when launched from a 30 angle compared to a 45 angle?

Appendix E Tool Library

Appendix F Prompt

F.2 Prompt for Tool Selection

F.3 Prompt for Trajectories Synthesis

Appendix G Database Cases

HotpotQA: Question: The deepest part of the ocean, is located in which ocean? Answer: The Pacific Ocean Question: The famous scientist who discovered gravity, lived in which century? Answer: 17th century Question: The first successful flight of a power was made by which inventor? Answer: The Wright brothers Question: The highest mountain peak in the solar system is located on which planet? Answer: Mars Question: In the novel "Pride and Prejudice", what is the name of Mr. Darcy’s estate in Derbyshire, England? Answer: Pemberley ScienceQA: Question: Which of the following is a type of renewable energy? Options: (A) Coal (B) Oil (C) Natural gas (D) Solar power Caption: A picture of a solar cell Answer: D. Solar power Question: Which of the following is the term for the process by which the Earth’s weather patterns are influenced by the movement of air in the atmosphere? Options: (A) Weathering (B) Erosion (C) Deposition (D) Atmospheric circulation Caption: An image of air currents in the atmosphere Answer: D. Atmospheric circulation Question: Which of the following is a type of chemical reaction that involves the transfer of electrons between atoms? Options: (A) Combustion (B) Photosynthesis (C) Respiration (D) Electrolysis Caption: An image of a battery Answer: D. Electrolysis Question: Which of the following is an example of a type of weather phenomenon that occurs when warm air rises and cool air sinks? Options: (A) Thunderstorms (B) Hurricanes (C) Fog (D) Fronts Caption: An image of a front Answer": D. Fronts Question: Which of the following is the term for the process by which water is purified through the use of microorganisms that consume organic matter? Options: (A) Filtration (B) Sedimentation (C) Biodegradation (D) Disinfection Caption: An image of a water treatment plant Answer: C. Biodegradation