Hypothesis Search: Inductive Reasoning with Language Models

Introduction

Inductive reasoning – the ability to infer general principles from specific examples and apply them to novel situations – is a core aspect of human intelligence (Peirce, 1868). Recently, large-scale pre-trained language models have received significant interest for their performance across a diverse range of reasoning tasks such as commonsense, arithmetic and symbolic reasoning (Rajani et al., 2019, Shwartz et al., 2020, Nye et al., 2021, Wei et al., 2022, Marasović et al., 2021, Lampinen et al., 2022, Zelikman et al., 2022, Zhou et al., 2022). There has been extensive discussion of language models’ impressive “in-context learning” capabilities, a form of inductive reasoning. However, other work suggests that in-context learning of these models has a highly limited capacity to perform inductive reasoning tasks where precise behavior is required (Chollet, 2019, Johnson et al., 2021).

The Abstraction and Reasoning Corpus (ARC) is a particularly challenging visual inductive reasoning benchmark (Chollet, 2019). For each task in ARC, models are given a set of training input-output pairs with a shared transformation rule, and the goal is to predict the corresponding output(s) given the novel test input(s), as illustrated in Fig 2 (a). ARC is interesting because the answers are fairly natural for humans yet require a complex and precise transformation. Evaluations of LLMs on ARC (Xu et al., 2023b, Mirchandani et al., 2023, Gendron et al., 2023) have directly prompted LLMs to predict outputs by in-context learning, finding extremely poor performance relative to humans (Chollet, 2019, Johnson et al., 2021).

We instead take inspiration from Bayesian models of human inductive reasoning (Tenenbaum et al., 2006, Goodman et al., 2008). That research frames inductive reasoning as posterior prediction: an ideal Bayesian learner assumes a large hypothesis space of possible rules, uses Bayes’ rule to form a posterior distribution over hypotheses from examples, then responds accordingly with a posterior-predictive distribution. Studies of human inductive learning have found that people likely approximate the full posterior with just a few hypotheses (Vul et al., 2014). Furthermore, people often represent hypotheses of the world at multiple levels of abstraction (Tenenbaum et al., 2011), with more abstract hypotheses guiding the search for more specific ones (Goodman et al., 2011).

We thus propose an approach that improves the inductive reasoning ability of LMs by decomposing the task via hypothesis formation at two levels of abstraction: first by generating hypotheses in natural language and then by realizing these as specific programs that are used for making predictions. Natural language provides abstract representations that uncover key features but are difficult to verify and potentially ambiguous. Programmatic hypotheses are directly verifiable on examples via execution and can naively generalize to new inputs but involve many implementation details that can be distracting to a language model. In other words, we use particular programmatic implementations to act as a precise, generalizable representation of a given inductive hypothesis formulated in natural language. Our pipeline thus disentangles inductive reasoning tasks primarily into two capabilities: the ability to propose accurate natural language hypotheses about the underlying rules, and the ability to formalize them as programs.

However, in practice LLMs are not yet able to find a good hypothesis with one try. Sampling multiple hypotheses and multiple programs per hypothesis turns out to be sufficient, but can be extremely costly. Thus, we also investigate approaches to reduce the number of hypotheses that must be considered. First, we use an LLM to summarize multiple hypotheses into a smaller number of hypotheses. Second, we experiment with querying a human oracle to go through all hypotheses and indicate which can be ignored. The latter can be viewed as a lower bound on performance that would be achieved by our approach without filtering, because we also find that programs which are correct on all examples almost always generalize correctly, an interesting feature of complex inductive reasoning domains.

We conduct experiments on three inductive reasoning datasets: the Abstraction and Reasoning Corpus (ARC), the one-dimensional variant of ARC (1D-ARC), and the Syntax-Guided Synthesis (SyGuS) dataset. Our results indicate that explicit hypothesis formation substantially improves performance over the direct prompting (ICL) approach. Ablation studies suggest both levels of abstraction – natural-language hypothesis generation and programmatic hypothesis representations – are beneficial to performing inductive reasoning tasks.

Method

We consider inductive reasoning tasks that require discovering an underlying transformation rule given input-output examples that follow this unknown rule. More formally, we are given a set of training examples ${(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{n},y_{n})}$ where each $y_{i}=f(x_{i})$ for some unknown function $f$. Our goal is for the model to infer the outputs $y_{1}^{\prime},y_{2}^{\prime},\ldots,y_{n}^{\prime}$ for a list of novel inputs $x_{1}^{\prime},x_{2}^{\prime},\ldots,x_{n}^{\prime}$ that captures the transformation $f$. This formulation applies to all three datasets we consider in the experiment settings, as shown in Figure 2. This task is widely studied in program synthesis literature (Acquaviva et al., 2022, Odena et al., 2020, Ellis et al., 2023, Xu et al., 2023a), where a program written in a manually-designed Domain-Specific Language (DSL) is used to represent the transformation, which is applied to the test inputs to obtain the predicted outputs. Recently, there are also multiple works (Webb et al., 2022, Xu et al., 2023b, Mirchandani et al., 2023, Gendron et al., 2023) that do not predict the rule explicitly. Instead, large language models are used to predict the output for novel input examples directly given the training input-output pairs.

2 Overview

As illustrated in Figure 1, in our pipeline, we first prompt an LLM to generate hypotheses about the transformation rule shared across the input-output pairs in natural language. We then filter out a smaller set of hypotheses, using either an LLM or human annotator – the goal of this step is simply to reduce the computational cost of later steps. The filtered hypotheses are used to prompt an LLM to generate Python programs that take in an input example and output the transformed result. These programs are then tested against the initial training examples. Note that, in these domains, we observed that programs that successfully generated the outputs for the training pairs almost always generalized to the test items.

3 Generating Hypotheses

We first prompt GPT-4 to generate natural language hypotheses for inductive reasoning problems. For each problem, we provide GPT-4 with a description of the task setup and the problem-specific input-output examples and prompt it to generate hypotheses about possible underlying rules or patterns that could explain the transformation in the given examples. We also provide two held-out problems with human-annotated hypotheses as few-shot demonstrations in the prompt. When doing an ARC task, we provide GPT-4 with the input-output examples in the form of a grid of numbers and specify the corresponding colors for each number as part of the prompt. The exact prompt can be found in the Appendix A. We sample multiple responses from GPT-4, with a temperature of $1.0$, as the hypothesis candidates.

4 Reducing Number of Candidate Hypotheses

Ideally, we would like to directly test generated hypotheses by implementing them as Python programs. However, given a potentially large number of hypotheses, testing all of them can be expensive. Thus, we investigate several methods to identify the most promising hypotheses from a set of proposals. For an end-to-end approach, we investigate using LLMs to summarize the full set of hypotheses into a smaller number of hypotheses. Specifically, we directly present GPT-4 with all candidate hypotheses and ask it to produce a smaller number of hypotheses summarizing the given candidate hypotheses. In addition, to help estimate a lower bound on performance if we were to test all hypotheses, we ask a human annotator to go through candidate hypotheses and select correct ones, if any.

5 Implementing Python Programs From Hypotheses

The pseudocode for this stage is presented in Algorithm 1. After obtaining a set of candidate hypotheses for each problem, we individually use each hypothesis as the input for GPT-4 and prompt it to generate multiple Python programs that implement the described transformation. Then, we run these programs against the problem’s original input-output examples (while still holding out the test examples), determining whether they yield correct outputs for each case. If a code implementation correctly generates the outputs for each of the training examples, it is selected for generating the prediction on the test input example. If no implementation passes all of the training examples, we repeatedly ask GPT-4 to revise the implementations according to the execution results on the training set, including error messages and desired outputs, similar to Chen et al. (2023). If we still cannot achieve a program that passes all the training examples after a preset number of feedback rounds, we select the program that passes most examples for generating the prediction.

Experiments and Results

We evaluate our approach on three distinct datasets: the Abstraction and Reasoning Corpus (ARC), the one-dimensional variant of ARC (1D-ARC), and BUSTLE’s Syntax-Guided Synthesis (SyGuS) dataset. These datasets offer diverse and challenging reasoning tasks in the domains of 2D grids, number sequences and strings, enabling us to thoroughly assess the inductive reasoning capabilities of our method. We provide examples of tasks in these datasets in Figure 2.

The Abstraction and Reasoning Corpus (ARC), proposed by Chollet (2019), is a dataset designed to assess models’ generalizable reasoning capabilities. It is a dataset of 400 training and 400 evaluation problems. Each problem consists of a set of input-output 2D grids that capture a specific underlying rule or pattern such as geometric transformation and object counting. Each example is a grid with $1\times 1$ to $30\times 30$ pixels of any of ten colors – note that the input and output grid need not have the same shape. To allow us to effectively analyze this task despite the high cost of GPT-4, in our paper, we randomly select a subset of 40 problems from the 400 training problems as the evaluation dataset.

1D-ARC is a one-dimensional adaptation of the original ARC dataset proposed in (Xu et al., 2023b). It contains Although simpler than the two-dimensional ARC problems, 1D-ARC offers a more controlled setting to investigate the inductive reasoning abilities of language models as they are trained to handle sequential data. We once again select a random subset for evaluation, this time randomly choosing two tasks from each of 1D-ARC’s $18$ categories for a total of 36 problems.

The SyGuS dataset in the BUSTLE paper contains 89 tasks that require representing a mapping between pairs of strings as a program (Odena et al., 2020). This task represents the kinds of problems solved by FlashFill (Gulwani, 2011), a feature in Excel that has been widely cited as an influential real-world example of program synthesis (Le et al., 2017).

2 ARC

We measure the performance of different methods by computing the accuracy of models’ prediction on the test input casesNote that the official ARC challenge uses the top-3 accuracy, where models can predict up to 3 candidate answers and the prediction is treated as correct if the correct answer is in the candidate answers. Here, we only consider top-1 accuracy.. Although the input-output examples are typically visually presented in 2D pixel grids, we convert them to a text format in the style of NumPy arrays. We include the prompt templates in Appendix A.

2.1 Main Results

We compare the direct prompting baseline to different variants of our pipeline.

Direct Prompting. Similar to previous works (Xu et al., 2023b, Mirchandani et al., 2023, Gendron et al., 2023), we provide training examples in the prompt and ask GPT-4 to directly infer the output grid of the novel test inputs.

Program Only. This is an ablation of our pipeline where we directly prompt GPT-4 to output Python programs given the training examples. We generate $64$ programs for each task and select the program passing the most training examples for generating the test outputs.

Summarized Hypotheses. For each problem, we first use GPT-4 to generate 64 candidate hypotheses and then ask GPT-4 to summarize $8$ hypotheses from the $64$ candidates. We then generate $8$ programs for each hypothesis resulting in $64$ candidate programs perf problem. This is followed by $2$ rounds of execution feedback. Note that during our experiments, we found that GPT-4 only generates correct hypotheses for $21$ tasks, according to human annotators. To further save cost, we only attempt to generate programs for these $21$ tasks and treat other tasks as incorrect. So the reported performance should be treated as a lower bound of the method, since potentially the model can obtain correct programs even with wrong hypotheses.

Human-Selected Hypotheses. We first ask GPT-4 to generate 64 hypotheses and then ask a human to manually annotate all 64 hypotheses and select correct ones, if any exist. Then we generate 8 programs for each of these hypotheses followed by up to 3 rounds of execution feedback. Similar to the previous version, we only generate programs on the 21 tasks with plausible hypotheses.

Human-Written Hypotheses. For this version, we leverage the human language annotations from the LARC dataset (Acquaviva et al., 2022) as golden hypotheses. We then generate 8 programs for each hypothesis, followed by 2 rounds of execution feedback. We treat these human-written hypotheses as oracle solutions in order for us to better understand the extent to which this pipeline is separately bottlenecked by hypothesis generation as opposed to program generation.

The main results are shown in Table 1. Using a programmatic representation already boosts the performance over the direct prompting baseline by a large margin, from 12.5% to 17.5%. Leveraging the summarized hypotheses is also helpful, improving the performance from $17.5\%$ to $27.5\%$. We obtain the best accuracy $37.5\%$ when generating programs using human-selected hypotheses. This is on par with the version where we directly leverage the golden human-generated hypotheses. This indicates that GPT-4 is pretty good at both generating hypotheses and realizing them as programs, which enables it to be a strong inductive reasoner.

2.2 Qualitative Results

We show an example of generated hypotheses and the corresponding programs generated from the considered methods in Fig. 3. We observe that many of the correct hypotheses generated by GPT-4 are similar to the human-written hypotheses in terms of their specificity, although often less concise. Summarized hypotheses can often become vague and ambiguous, which is potentially the reason for degraded performance. Sometimes the correct hypothesis is omitted from the summarized hypotheses. As a side note, because we prompt GPT-4 to treat the grids as NumPy (Harris et al., 2020) arrays, we observe that GPT-4 tends to leverage various functions from the NumPy library to perform the desired transformation.

2.3 More Ablation Studies

GPT-3.5 vs GPT-4. In this ablation, we leverage GPT-3.5 instead of GPT-4 in our pipeline. Compared with GPT-4, we find GPT-3.5 mostly generates meaningless hypotheses given the inputs from ARC. We then test GPT-3.5’s ability to generate program implementations when given the human-written hypotheses. Because GPT-3.5’s context length is only 4096 tokens (GPT-4’s context length is 8096), only 33 tasks can fit into the prompt. Therefore, we treat the problems that do not fit in the context window as incorrect and do not leverage execution feedback. GPT-3.5 achieves an accuracy of 32.5% with 128 programs when given human-written hypotheses. (Howeverm GPT-3.5 is approximately 20 times cheaper than GPT-4.)

Execution Feedback. The results of models using different numbers of execution feedback iterations are summarized in Table 2. Execution feedback plays an important role regardless of how hypotheses are generated. However, the performance gain plateaus as the number of feedback iterations increases.

3 1D-ARC

In contrast to the ARC experiments, GPT-4’s performance on 1D-ARC was notably higher. We observed reasonably correct hypotheses by simply generating 16 hypothesis candidates. Therefore, we did not need to obtain a subset of hypotheses to reduce the cost of implementing programs. On the 1D-ARC dataset, we compare the direct prompting baseline with two variants of our method.

Direct Prompting For this experiment, we report the accuracy of direct prompting results from Xu et al. (2023b) on the selected 36 tasks.

Program Only. We directly prompt GPT-4 to output Python programs given the training examples. We generate $80$ programs for each task and select the program that passed most training examples.

Full. We first generate 16 different language hypotheses, then generate 4 programs for each, resulting in 64 programs per problem.

We summarize our results in Table 3. Generating hypotheses and implementing programs significantly improves the performance on 1D-ARC compared with the direct prompting method.

4 SyGuS

We use all $89$ tasks from the SyGuS dataset for evaluation. Unlike ARC and 1D-ARC datasets, we follow the convention in the program synthesis literature and treat all examples as training examples. The accuracy is computed by whether the program passes all training examples.

We find that GPT-4 can generate correct programs for $94.3\%$ of the SyGuS tasks using $8$ programs with two rounds of feedback without hypothesis generation, demonstrating strong performance in a direct program generation approach. Of the five remaining tasks, we find that three of the tasks have mistakes in their examples. As a result, when using natural language hypotheses to guide the code generation process, GPT-4’s performance does not meaningfully change, reaching $93.2\%$ by generating $4$ hypotheses and implementing $2$ program for each hypothesis. This performance is slightly below that of the direct program generation without language guidance, highlighting that language may not be useful if performance is already saturated. As a comparison, the state-of-the-art program induction approach CrossBeam Shi et al. (2022) can solve $74.8\%$ of the dataset using a domain-specific language with 50K program candidates. Our method significantly outperforms CrossBeam with many fewer programs tested.

Discussions

Currently, there are two types of failures in our pipeline. First, the model may be unable to generate a correct and sufficiently precise natural language hypothesis. Second, the model can still generate incorrect programs given a correct hypothesis.

Hypothesis generation is especially challenging for the ARC dataset as it involves recognizing visual patterns in 2D grids. While we observe that GPT-4 has a primitive ability to recognize points, lines, and rectangles, and to identify repetition and symmetry relationships, it has trouble understanding more complex shapes and visual relationships like translation, scaling, and containment. This is unsurprising as GPT-4 is trained primarily on text corpora, and the visual grid is input as text in our experiments. Furthermore, we observe that GPT-4 has difficulty proposing reasonable hypotheses for very large grids, possibly due to the limited context length. In contrast, GPT-4 was quite good at hypothesis generation on 1D-ARC. While the concepts may be easier for this dataset it is certainly the case that the visual encoding is easier. We thus tentatively conclude that current LMs are quite capable of hypothesis generation for inductive learning and anticipate that vision-language models (Driess et al., 2023) may close the remaining gap for visual tasks like ARC.

Even with correct hypotheses, difficulties may arise when the task is hard to implement in Python. For example, task 444801d8 shown in Figure 4 was one where the language model failed when given a correct hypothesis. The task is difficult to solve programmatically, even for humans, as it requires identifying an irregular shape and then filling it according to an irregular pattern. This suggests a limitation of using generic Python programs for solving visual inductive reasoning tasks. Natural language hypotheses may also contain ambiguous concepts that mismatch the biases of the program generator. The human-written hypothesis for task 363442ee in Figure 4 is: “In the input, you should see a color pattern on the left side and blue squares on the right. The output grid size same size as the input. To make the output, you have to use the blue square as the middle square and recreate the same pattern replacing the blue square with the same color in the middle as the pattern.” GPT-4 is unable to understand what “color pattern” refers to and generates an incorrect program by treating the first three columns as the pattern. On the other hand, GPT-4’s generated hypothesis mentions that "In the input, you should see a 3x3 colored square on the left side…", which yields the correct Python implementation. Thus a good match is needed between hypothesis generator and program synthesis, suggesting dircetions for future work.

2 Considering Every Candidate Hypothesis.

Currently, our pipeline does not consider many candidate hypotheses; we note that this is not a theoretical limitation of our method. In our experiments, we found that when the generated program passes all training cases, it almost always passed the test case (we only observe a single task that is an exception). Therefore, the performance of human-selected hypotheses can reasonably be treated as a lower bound for the performance if we consider every candidate hypothesis. However, we need to sample a large number of (64) hypotheses to have a reasonable hit rate of correct ones, and testing a single candidate hypothesis can take up to $1.5$$($8$ programs with two rounds of feedback) – leading us to evaluate summarizing and human filtering. This suggests that the effectiveness of our method will improve automatically as the inference cost of language models decreases.

3 Data Memorization.

While large language models have shown remarkable performance on numerous benchmarks, there are recurring concerns about whether these models have simply memorized the answers due to observing the problems during training, instead of actually solving the desired tasks. This is particularly true for closed-source models where details of the training set are not publicly available, such as GPT-4. Since the ARC (as well as the LARC dataset with human-written hypotheses) and SyGuS datasets are publicly available on the internet, there is a possibility that GPT-4’s training data contains these datasets, which might affect how we interpret these results. While differentiating between memorization and generalization for these close-sourced models remains an open problem, there are few pieces of evidence that show the effectiveness of our method. First, as far as we know, there are no public attempts to solve ARC or SyGuS datasets with Python programs. Second, we tried prompting GPT-4 with some examples in a task and asked it to output other examples in the same task, and GPT-4 failed to do so. Third, the substantial boost of our full pipeline over the direct prediction baseline cannot be simply explained by data memorization.

4 Combinatorial Search with Parsel

We also explore the application of Parsel, an efficient compositional program generation method (Zelikman et al., 2023), in combination with GPT-4 to enhance the model’s ability to generate and evaluate code implementations. This approach aims to capitalize on the benefits of compositional reasoning in problem-solving, by first decomposing a solution, generating multiple implementations of each part of the solution, and then searching over combinations of the implementations. This allows for many more programs to be tested with fewer LLM calls. For human-written hypotheses, this improved performance to 47.5%, but for language-model-generated hypotheses it had the reverse effect. Details can be found in Appendix B.

Related Works

Techniques to allow automatic inductive reasoning have been widely studied by the artificial intelligence community as well as the program synthesis community. Given a set of observations, these efforts aim to computationally infer the underlying rules for a set of observations that can be generalized to novel scenarios. Traditional methods usually rely on programs written in manually designed domain-specific languages to represent the rule space and perform searching on the space to obtain the desired program. A number of heuristics have been proposed to speed up the search process. BUSTLE (Odena et al., 2020) proposes a neural search algorithm that takes the intermediate results of partial programs into account during the search process. DreamCoder (Ellis et al., 2023) introduces a wake-sleep algorithm that will dynamically build library functions on top of the primitive operations for solving more complex tasks with less running time. These methods typically require training on a corpora of related tasks, and cannot generalize across different domains due to the limited DSL. Earlier work in this area showed that introducing linguistic knowledge and selecting relevant language descriptions allows for better classifiers (Andreas et al., 2018). Recently, multiple works (Mirchandani et al., 2023, Gendron et al., 2023) tried to evaluate large language models on inductive reasoning tasks. These works directly prompt models to predict the output given the novel input as well as training examples, which leads to poor performance. Our work draws inspiration from previous program synthesis literature to use programs as representations of the underlying rules. But we instead leverage a general programming language Python, which makes our method applicable to a wide range of different domains such as grid transformation and string transformation.

There has been a consistent effort to introduce program representations into different types of reasoning tasks such as visual reasoning (Andreas et al., 2016, Mao et al., 2019) and question answering (Dong & Lapata, 2016, Zhong et al., 2017). Programs provide various advantages over end-to-end methods, such as interpretability, generalizability, and efficiency. Mainstream approaches have focused on learning to parse natural language questions into programs of domain-specific languages that can be executed to obtain the answer; a program executor is often jointly learned to execute primitive functions (Andreas et al., 2016, Mao et al., 2019). Recently, LLMs have been shown to be capable of generating programs written in general-purpose programming languages. This inspired multiple works to leverage LLMs to reason with programmatic representations. Gao et al. (2022) introduced Program-Aided Language models (PAL), and Chen et al. (2022) proposed the "Program of Thoughts" (PoT) prompting, both of which prompt large language models to solve step-by-step math and symbolic reasoning tasks by proposing programs and offload the computation to a Python interpreter. Visprog (Gupta & Kembhavi, 2023) and ViperGPT (Surís et al., 2023) generated programs that can be executed using pretrained perception modules to tackle visual reasoning tasks. These approaches are superior in performance and require minimal data for in-context learning without the need for any training. Notably, the code generated by the models in these papers has primarily served as a computational aid, not a general task representation. In our case, programs serve as testable hypotheses for solving inductive reasoning tasks. Lastly, Clement et al. (1986) investigated the correlation between analogical reasoning ability and the programming skills of high school students, indicating a significant relationship between the ability to perform analogical reasoning and write compositional programs. Given previously observed parallels between language model behavior and cognitive psychology experiments (e.g., Dasgupta et al. (2022), Aher et al. (2022)), language models may exhibit a similar trend.

Conclusions

In this work, we propose a pipeline that facilitates better inductive reasoning in large language models. The core idea is to first prompt LLMs to generate hypotheses of the underlying rule in natural language, to then implement the hypotheses as Python programs, and to search for programs which can be verified on the training examples and executed on novel inputs for inference. We evaluate the effectiveness of our pipeline on three challenging datasets Abstraction and Reasoning Corpus (ARC), its variant 1D-ARC, and a string transformation dataset SyGuS. Our pipeline outperforms the baseline methods by a large margin on all three datasets.

References

Appendix A Experiment Details

For hypothesis generation, The prompts are shown in Figure 5, Figure 6 and Figure 7. We set the temperature to be $1.0$ and the maximum number of tokens in response to be $200$. For program generation and execution feedback, we use a temperature of $0.7$ and set the maximum number of tokens to be $1000$. We use gpt-4-0314 and gpt-3.5-turbo-0301 throughout the experiments.

After obtaining programs from language models, we directly execute them on training examples. If there are no programs passing all training examples, we prompt the language model again with the first example that the program fails to pass, and ask it to correct the program. To save cost for experiments on ARC where we generate more than 64 programs, we do not run execution feedback on every program. Instead, we cluster programs by their output on the training examples. Only one program is selected from each cluster for feedback execution.

Appendix B Extra Results

To enhance the performance of program generation, we also adapt a recently proposed method Parsel (Zelikman et al., 2023) to our settings. Instead of directly generating programs, we first generate an intermediate pseudocode program written in Parsel language from a given hypothesis, as shown in Figure 8. The Parsel language specifies the functions needed to be implemented by specifying the function name, arguments and its desired behavior in natural language. Then the Parsel program is passed to a language model for implementing individual functions.

To allow functions to be implemented with knowledge of their context, unlike the original Parsel paper, we implement all functions needed in a single API call. We then sample multiple trials and extract multiple implementations of each function specified in the Parsel program. Then we will recombine every implementation of each function to generate multiple programs. Using human-written hypotheses, we achieve an accuracy of 47.5% on the 40 randomly selected questions from ARC by generating 4 Parsel Programs for each hypothesis and 8 programs for each Parsel program without any feedback, surpassing the 37.5% accuracy obtained by directly generating programs from hypotheses. However, we found that this yields worse performance with LLM-generated hypotheses: on the 13 selected tasks that GPT-4 can generate correct hypotheses, directly generating 8 programs with 1 round of execution feedback yields an accuracy of 92% while 4 Parsel programs $\times$ 8 python programs with 1 round of feedback only yields an accuracy 69%. We suspect that this is due to Parsel introducing a new level of abstraction into our pipeline: given that error might accumulate during the transformation between different levels of abstraction, Parsel increases the probability of generating incorrect final programs. We believe leveraging better code generation techniques is a promising direction to improve our pipeline.

B.2 Pilot Experiments and Non-Systematic Findings

In the prompt we use for ARC, we indicate the grids are represented as NumPy arrays using Python type hint (numpy.ndarray[int]). The typing hint plays an important role in generating programs from language hypotheses, since it encourages LLMs to leverage NumPy functions that are suited for grid transformation, such as flipping, 2D indexing. If we change the typing hint to List[List[int]], LLMs will no longer leverage this library function, which makes the program longer and more error-prone. Using human-written hypotheses, $8$ programs and one round of execution feedback. GPT-4 can only achieve $32.5\%$, compared with the $37.5\%$ performance the using NumPy array signature.

We also explore to use LLMs to rank language hypotheses to throw away bad hypotheses. This is inspired by Zhang et al. (2023), which reranks code generated from a description based on its probability of generating the description. Because GPT-4 does not expose the log-probabilities of its generated items, and there is no clear way to extract the log-probabilities of the hypotheses, we instead use GPT-3 to rerank the hypotheses generated by GPT-4 by looking at their probabilities of generating the input-output examples, given the hypothesis. We evaluate this ranking method on the 21 tasks where GPT-4 is able to generate a correct language hypothesis from 64 candidates. After the ranking, there are only 10 tasks where the correct hypotheses are placed in the top 16 candidates. This prevents us from reducing the hypotheses needed to implement without sacrificing the overall performance.

We observed that many for many tasks in ARC, it is easy for LLMs to come up with reasonable hypotheses if the grids are parsed into a useful geometric representation, such as irregular shapes, diagnoal lines. As a result, we explored the possibility of using alternative geometric representations, similar to concurrent work (Xu et al., 2023b). In particular, we attempted to treat each grid as the result of a sequence of shape placements, for example:

We implemented an algorithm to identify the shortest possible sequence of shape placements that would result in the observed grid. While we observed that this allowed the model to propose more reasonable hypotheses for a subset of the problems, it harmed performance on more of them. This is due in part to the inherent difficulty of proposing a useful general representation for ARC tasks.