BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model

Introduction

The field of developing deep learning architectures for tabular data is recently experiencing rapid advancements (Somepalli et al., 2021; Gorishniy et al., 2021; Arik and Pfister, 2021; Huang et al., 2020). The primary driving force behind this trend is the limitations of the current dominant methods for tabular data: tree-based methods. Specifically, while tree-based methods excel in tabular learning, tree-based methods lack the capability to integrate with deep learning architectures. Therefore, the pursuit of deep tabular learning is not just a matter of enhancing performance but is also crucial to bridge the existing gap. However, a recent tabular benchmark study (Grinsztajn et al., 2022) reveals that tree-based methods still surpass deep learning models, underscoring two main challenges for deep tabular learning, as highlighted by Grinsztajn et al. (2022, Section 5.3 & 5.4):

Non-Rotationally Invariant Data Structure: The non-rotationally invariant structure of tabular data weakens the effectiveness of deep learning models that have rotational invariant learning procedures.

Feature Sparsity: Tabular datasets are generally sparser than typical datasets used in deep learning, which makes it challenging for deep learning models to learn from uninformative features.

To combat these, we introduce the Bi-Directional Sparse Hopfield Network, a Hopfield-based deep learning framework tailored for tabular data. To address the non-rotationally invariant data structure of tabular data (C1), our model adopts a dual-component design, named the Bi-directional Sparse Hopfield Module ($\mathtt{BiSHopModule}$). Specifically, our model employs bi-directional learning through two separate Hopfield models, focusing on both column-wise and row-wise patterns separately, thereby naturally assimilating the tabular data’s inherent structure as an inductive bias.

For tackling the features sparsity in tabular data (C2), we utilize the generalized sparse modern Hopfield model (Wu et al., 2024b). The generalized sparse modern Hopfield model is an extension to the sparse modern Hopfiled model (Hu et al., 2023) and modern Hopfiled model (Ramsauer et al., 2020) with the learnable sparsity. It offers robust representation learning and seamlessly integrates with existing deep learning architectures, ensuring focus on crucial information. Furthermore, inspired by brain’s multi-level organization of associative memory, we stack multiple layers of the generalized sparse modern Hopfield model within BiSHopModule. As a result, each layer learns representations at unique scales, adjusting its sparsity accordingly, adding (C2) as another inductive bias to the model.

At its core, BiSHop facilitates multi-scale representation learning, capturing both intra-feature and cross-feature dynamics while adjusting sparsity for each scale. In all directions, whether column-wise or row-wise, the model identifies representations across various scales. These refined representations, spanning all scales, are subsequently concatenated for downstream inference, ensuring a holistic Bi-Directional cellular learning tailored for tabular data.

Methodologically, we propose BiSHop, a novel deep-learning model for tabular data. BiSHop integrates with two inductive biases (C1, C2) in tabular learning using BiSHopModule and hierarchical learning structure. The BiSHopModule utilizes the generalized sparse modern Hopfiled model (Wu et al., 2024b) for tabular feature learning, enabling multi-scale sparsity learning with superior noise-robustness. We also present a hierarchical two-joint design to handle the intrinsic structure of tabular data with learnable sparsity and multi-scale cellular learning. Additionally, we adopt tabular embedding (Huang et al., 2020; Gorishniy et al., 2021, 2022) to enhance representation learning for both numerical and categorical features.

Experimentally, we conduct thorough experiments on diverse real-world datasets as well as a well-known tabular benchmark (Grinsztajn et al., 2022). This encompasses a total of 18 classification tasks and 11 regression tasks. We compare BiSHop with both SOTA tree-based and deep learning methods. Our results show that BiSHop outperforms baselines across most of tested datasets, including both regression and classification tasks.

1 Related Works

Tabular data is a common data types across various domains such as time series prediction, fraud detection, physics, and recommendation systems. The state-of-the-art machine learning models on tabular data are tree-based model such as the family of gradient boosting decision trees (GBDT) models (Chen et al., 2015; Prokhorenkova et al., 2018; Ke et al., 2017). Recent years, as deep learning model architectures thrive in the natural language processing (NLP) domain and the computer vision (CV) domain, there are many attempts to adapt and apply those successful deep learning architectures such as Multi-layer Perceptron (MLP) Kadra et al. (2021), Convolutional neural network (CNN) (Buturović and Miljković, 2020), and Transformer (Huang et al., 2020; Padhi et al., 2021; Somepalli et al., 2021) from these two domains to the domains using tabular data. Besides, another line of works using deep learning is to create differentiable tree-based models intending to bring extra power on top of current GBDT models (Arik and Pfister, 2021; Abutbul et al., 2020; Popov et al., 2019). However, unlike their dominance in NLP and CV, all these deep learning models have been struggling to surpass GBDTs’ dominant performance on tabular data (Borisov et al., 2021; Grinsztajn et al., 2022). A recent work, TabR (Gorishniy et al., 2023)), show some marginal advantage over GBDT on portion of the datasets. For small datasets, TabPFN (Hollmann et al., 2023) by utilizing Prior-Data Fitted Network performs better then tree-based method. However, the memory and runtime usage scale quadratically with the training inputs. T2G-FORMER (Yan et al., 2023) fails to surpass XGBoost, while performing better than other deep learning methods by feature relations learning. TANGOS (Jeffares et al., 2023) reduces the gap between deep-learning models to tree-based models by applying specific regularisation techniques during NN training. To this day, there is still no deep learning model for tabular data that can uniformly outperform tree-based model.

The classical Hopfield models (Hopfield, 1984, 1982; Krotov and Hopfield, 2016) are quintessential representations of the human brain’s associative memory. Their primary function is the storage and retrieval of specific memory patterns. Recently, a resurgence of interest in Hopfield models within the machine learning field is attributed to developments in understanding memory storage capacities (Krotov and Hopfield, 2016; Demircigil et al., 2017; Wu et al., 2024a), innovative architecture (Hoover et al., 2023; Seidl et al., 2022; Fürst et al., 2022; Ramsauer et al., 2020), and their biologically-grounded rationale (Kozachkov et al., 2022; Krotov and Hopfield, 2020). Notably, the modern Hopfield models (Ramsauer et al., 2020; Hu et al., 2023; Wu et al., 2024b; Hu et al., 2024b)For an in-depth tutorial, see (Brandstetter, 2021)., demonstrate not only a strong connection to the transformer attention mechanisms in deep learning, but also superior performance, and a theoretically guaranteed exponential memory capacity. In this regard, seeing the modern Hopfield models as an advanced extension of attention mechanisms opens up prospects for crafting Hopfield-centric architectural designs. Therefore, their applicability spans diverse areas like immunology (Widrich et al., 2020), time series forecasting (Wu et al., 2024b; Auer et al., 2023), reinforcement learning (Paischer et al., 2022), and large language models (Hu et al., 2024a; Fürst et al., 2022). In this context, this work emphasizes refining this line of research towards sparser models. Specifically, we improve our method’s ability to handle sparsity by incorporating a Generalized Sparse Modern Hopfield Network ($\mathtt{GSH}$) from Wu et al. (2024b). We posit that this effort is crucial in guiding future research toward Hopfield-driven design paradigms and bio-inspired computing systems.

Background: Dense and Generalized Sparse Modern Hopfield Model

This section provides a concise overview of the modern Hopfield model (Ramsauer et al., 2020) and the generalized sparse modern Hopfield model (Wu et al., 2024b). Wu et al. (2024b) presents an extension to (Hu et al., 2023; Ramsauer et al., 2020), utilizing the Tsallis $\alpha$-entropy (Tsallis, 1988)

Ramsauer et al. (2020) propose the (dense/vanilla) modern Hopfield model with a specific set of $E$ and $\mathcal{T}$, and integrate it into deep learning architectures via its connection with attention mechanism, offering enhanced performance, and theoretically guaranteed exponential memory capacity. Specifically, they introduce a Hopfield energy function:

and the corresponding memory retrieval dynamics

The $\mathcal{T}_{\text{Dense}}$ dynamics converge to memories provably and retrieve patterns accurately in just one step.

The modern Hopfield model from (2.1) possesses an exponential memory capacity in pattern size $d$.

Notably, the one-step approximation of $\mathcal{T}_{\text{Dense}}$ mirrors the attention mechanism in transformers, leading to a novel architecture design: the Hopfield layers.

2 Generalized Sparse Modern Hopfield Model

This section follows (Wu et al., 2024b, Section 3). For self-containedness, we also summarize the useful theoretical results of (Wu et al., 2024b) in Appendix B.

where $\Psi^{\alpha}(\cdot)$ is the Tsallis entropic regularizer

The corresponding memory retrieval dynamics is given as

Given $t$ as the iteration number, the generalized sparse modern Hopfield model exhibits a retrieval dynamic

which ensures a monotonic decrease of the energy (2.3).

This model also enjoys nice memory retrieval properties:

Given the energy function $E$ and retrieval dynamics $\mathcal{T}$ defined in (2.3) and (2.4), respectively. For any sequence $\{\mathbf{x}_{t}\}_{t=0}^{\infty}$ generated by the iteration $\mathbf{x}_{t^{\prime}+1}=\mathcal{T}(\mathbf{x}_{t^{\prime}})$, all limit points of this sequence are stationary points of $E$.

Lemma 2.2 ensures the (asymptotically) exact memory retrieval of this model ((2.3) and (2.5)), Thus, it serves as a well-defined associative memory model.

In essence, Wu et al. (2024b) present this sparse extension of the modern Hopfield model through a construction of both $E$ and $\mathcal{T}$ by convex conjugating the Tsallis entropic regularizers. This model not only adheres to the conditions for a well-defined modern Hopfield model, but also equips greater robustness (Corollary B.1.2) and retrieval speed (Theorem B.1 and Corollary B.1.1) than the modern Hopfield model (Ramsauer et al., 2020), see Section B.2 for details. In Figure 4, we also provide proof-of-concept experimental validations on tabular datasets for Theorem B.1, Corollary B.1.1 and Corollary B.1.2.

Importantly, the generalized sparse modern Hopfield model serves as a valuable component in deep learning due to its connection to the transformer attention mechanism akin to its cousins. Next, we review such connections and the Generalized Sparse Modern Hopfield (GSH) layers.

Following (Wu et al., 2024b; Hu et al., 2023; Ramsauer et al., 2020), $\mathbf{X}$ and $\bm{\Xi}$ are defined in the associative space, embedded from the raw query $\mathbf{R}$ and memory patterns $\mathbf{Y}$, respectively, using $\mathbf{X}^{\top}=\mathbf{R}\mathbf{W}_{Q}\coloneqq\mathbf{Q}$ and $\bm{\Xi}^{\top}=\mathbf{Y}\mathbf{W}_{K}\coloneqq\mathbf{K}$ with matrices $\mathbf{W}_{Q}$ and $\mathbf{W}_{K}$. By transposing $\mathcal{T}$ from (2.5) and applying $\mathbf{W}_{V}$ such that $\mathbf{V}\coloneqq\mathbf{K}\mathbf{W}_{V}$, we obtain:

This allows the seamless integration of the generalized sparse modern Hopfield model into deep learning architectures. Concretely, the $\mathtt{GSH}$ layer takes matrices $\mathbf{R}$, $\mathbf{Y}$ as inputs, with the weight matrices $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, $\mathbf{W}_{V}$. Depending on its configuration, it offers several functionalities:

Memory Retrieval: In this learning-free setting, weight matrices $\mathbf{W}_{K}$, $\mathbf{W}_{Q}$, and $\mathbf{W}_{V}$ are set as identity matrices. Here, $\mathbf{R}$ represents the query input, and $\mathbf{Y}$ denotes the stored memory patterns for retrieval.

$\mathtt{GSH}$: This configuration takes $\mathbf{R}$ and $\mathbf{Y}$ as inputs. Intending to substitute the attention mechanism, the weight matrices $\mathbf{W}_{K}$, $\mathbf{W}_{Q}$, and $\mathbf{W}_{V}$ are rendered learnable. Furthermore, $\mathbf{R}$, $\mathbf{Y}$, and $\mathbf{Y}$ serve as the sources for query, key, and value respectively. Achieving a self-attention-like mechanism requires setting $\mathbf{R}$ equal to $\mathbf{Y}$.

$\mathtt{GSHPooling}$: With inputs $\mathbf{Q}$ and $\mathbf{Y}$, this layer uses $\mathbf{Q}$ as a static prototype pattern, while $\mathbf{Y}$ contains patterns over which pooling is desired. Given that the query pattern is replaced by the static prototype pattern $\mathbf{Q}$, the only learnable weight matrices are $\mathbf{W}_{K}$ and $\mathbf{W}_{V}$.

$\mathtt{GSHLayer}$: The $\mathtt{GSHLayer}$ layer takes the query $\mathbf{R}$ as its single input. The layer equips with learnable weight matrices $\mathbf{W}_{K}$ and $\mathbf{W}_{V}$, which function as our stored patterns and their corresponding projections. This design ensures that our key and value are decoupled from the input. In practice, we set $\mathbf{W}_{Q}$ and $\mathbf{Y}$ as identity matrices.

In this work, we utilize $\mathtt{GSH}$ and $\mathtt{GSHPooling}$ layershttps://github.com/MAGICS-LAB/STanHop .

Methodology

As in Figure 1, BiSHop use three distinct parts to integrate two pivotal inductive biases in tabular data: non-rotationally invariant data structures (C1) and sparse information in features (C2) (Grinsztajn et al., 2022, Section 5.3 & 5.4):

A joint Tabular Embedding layer is designed to processing categorical and numerical data separately.

The Bi-Directional Sparse Hopfield Module (BiSHopModule) leverages the generalized sparse modern Hopfield model. This module incorporates the non-rotationally invariant bias through two interconnected $\mathtt{GSH}$ blocks for row-wise and column-wise learning.

Stacked BiSHopModules for hierarchical learning, addressing sparse features. Each layer in the stack module captures information at different scales, allowing for scale-specific sparsity.

We provide a detailed breakdown of each part as follows.

2 Bi-Directional Sparse Hopfield Module

By drawing parallels with the intricate interplay of different parts in the brain (Presigny and Fallani, 2022), we present the core design of the BiSHop framework, the Bi-Directional Sparse Hopfield Module ($\mathtt{BiSHopModule}$), as visualized in Figure 2 (c). The BiSHopModule incorporates the generalized sparse modern Hopfield model and integrate the inductive bias of tabular structure (C1) through a unique structure of stacked row-wise and column-wise $\mathtt{GSH}$ blocks. Specifically, the row-wise $\mathtt{GSH}$ focuses on capturing the embedding details for individual features, whereas the column-wise $\mathtt{GSH}$ aggregates information across all features. We denote $\mathbf{X}^{\text{patch}}_{n,p},n\in[N],p\in[P]$ as the element in $n$-th row (feature) and $p$-th column (embedding).

The column-wise $\mathtt{GSH}$ block (purple block on the LHS of Figure 2 (c)) is responsible for capturing embedding hidden information across the embedding dimension $P$ for each feature. The process begins by passing the patch embeddings of $n$-th row of $\mathbf{X}^{\text{patch}}$, $\mathbf{X}^{\text{patch}}_{n,:},n\in[N]$, to the $\mathtt{GSH}$ layer for self-attention, followed by the addition of the original patch embeddings (similar to the residual connection of the standard transformer). Next, we pass the output above through one $\mathtt{LayerNorm}$ layer, one Multi-Layer Perception (MLP) layer, and another $\mathtt{LayerNorm}$, and obtain the final output of the column-wise block $\mathbf{X}^{\text{col}}$:

This sequence of operations ensures the effective transformation of the embeddings, facilitating the extraction of meaningful information from the feature space.

This $\mathbf{Q}$ pooling matrix design aggregates information from all patch embedding dimensions, and by setting $C\ll N$, it significantly reduces computational complexity.

Together with the row-wise block, we summarize the entire BiSHopModule as a function

where input is $\mathbf{X}^{\text{patch}}$ and output is $\mathbf{X}^{\text{row}}$.

3 Stacked BiSHopModules for Multi-Scale Learning with Scale-Specific Sparsity

Motivated by the human brain’s multi-level organization of associative memory (Presigny and Fallani, 2022; Krotov, 2021), we utilize a hierarchical structure to learn multi-scale information similar to (Zhang and Yan, 2023; Zhou et al., 2021). This is illustrated in Figure 2 (d). This structure consists of two main components: the encoder and the decoder, both of which incorporate the $H$ layer of BiSHopModules. Specifically, the encoder captures coarser-grained information across different scales, while the decoder makes forecasts based on the information encoded by the encoder.

For the final prediction, we flatten $X^{\text{dec},H}$ and pass it to a new MLP predictor.

Drawing inspiration from the dynamic sparsity observed in the human brain (Stokes et al., 2013; Leutgeb et al., 2005; Willshaw et al., 1969), the parameter $\alpha$ for each $\mathtt{GSH}$ layer is a learnable parameter by design (Wu et al., 2024b; Correia et al., 2019), which allows BiSHopModule to adapt to different sparsity for different resolutions. Namely, the learned representations at each scale are equipped with scale-specific sparsity.

Experimental Studies

In this section, we compare BiSHop with SOTA tabular learning methods, following the tabular learning benchmark paper (Grinsztajn et al., 2022). We summarize our experimental results in Table 1 and Figure 3.

Our experiment consists of two parts: firstly, we benchmark commonly used datasets in the literature; secondly, we follow the tabular benchmark (Grinsztajn et al., 2022), applying it to a broader range of datasets on both classification and regression tasks.

In the first experimental setting, we evaluate BiSHop on 9 common classification datasets used in previous works (Grinsztajn et al., 2022; Somepalli et al., 2021; Gorishniy et al., 2021; Huang et al., 2020). These datasets vary in characteristics: some are well-balanced, and others show highly skewed class distributions; We set the train/validation/test proportion of each dataset as 70/10/20%. Please see Section C.1 for datasets’ details.

In the second experimental setting, we test BiSHop in the tabular benchmark (Grinsztajn et al., 2022). The datasets compiled by this benchmark consist of 4 OpenML suites:

Categorical Classification (CC, suite_id: 334),

Numerical Classification (NC, suite_id: 337),

Categorical Regression (CR, suite_id: 335),

Numerical Regression (NR, suite_id: 336).

Both CC and CR include datasets with numerical and categorical features, whereas NC and NR only contain numerical features. Due to limited computational resources, we randomly select one-third of the datasets from each suite for evaluation. We evaluate BiSHop on each suit with 3-6 different datasets and truncate to 10,000 training samples for larger datasets (corresponding to medium-size regimes in the benchmark). For these datasets, we allocate 70% of the data for the training set (7,000 samples). Of the remaining 30%, we allocate 30% for the validation set (900 samples), and the rest 70% for the test set (2,100 samples). All samples are randomly chosen from the original dataset and perform identical preprocessing steps of the previous benchmark (Grinsztajn et al., 2022).

We use the AUC score for the first experimental setting, aligned with literature. We use accuracy for classification task and R2 score for regression task in the second experimental setting, aligned with (Grinsztajn et al., 2022).

In the first experimental setting, we select 5 deep learning and 3 tree-based baselines, including (i) DL-based method such as MLP, TabNet, TabTransformer, FT-Transformer (Gorishniy et al., 2021), SAINT (Somepalli et al., 2021), TabPNF (Hollmann et al., 2023), TANGOS (Jeffares et al., 2023), T2G-FORMER (Yan et al., 2023) and (ii) tree-based methods such as LightGBM, CatBoost, and XGBoost (Chen et al., 2015). For each dataset, we conduct up to 200 random searches on BiSHop to report the score of the best hyperparameter configuration. We stop HPOs when observing the best result. Baselines and benchmark datasets’ results are quoted from competing papers when possible and reproduced otherwise. We report the reproduced results in Appendix C. Notably, we quote the best result from all baselines if multiple results are available.

In the second experimental setting, we reference baselines resultshttps://github.com/LeoGrin/tabular-benchmark from the benchmark paper (Gorishniy et al., 2021), comprising 4 deep learning methods and 3 tree-based methods, including (i) DL-based method such as MLP, ResNet (He et al., 2015), FT-Transformer (Gorishniy et al., 2021), SAINT (Somepalli et al., 2021) and (ii) tree-based methods such as RandomForest, GradientBoostingTree (GBDT), and XGBoost (Chen et al., 2015). We select the best results of each method from the benchmark (Grinsztajn et al., 2022). Notably, these best results take 400 HPOs according to Grinsztajn et al. (2022).

BiSHop’s default parameter settings are as follows: Embedding dimension $G$ = 32; Stride factor $L$ = 8; Number of pooling vector $C$ = 10; Number of BiSHopModules $H$ = 3; Number of aggregation in encoder $r$ = 4; Number of representation decoded $S$ = 24; Dropout = 0.2; Learning rate: $5\times 10^{-5}$. For numerical embedding, we only gather quantile information from training data to process the embedding function. For hyperparameter tuning, we use the “sweep” feature of Weights and Biases (Biewald et al., 2020). Notably, due to the computational constraints, we manually end the HPO once our method surpass the best performance observed in the benchmarks. We report search space for all hyperparameters in Table 6 and other training details in Section C.2. The optimization is conducted on training/validation sets, and we report the average test set scores over 3 iterations, using the best-performed configurations on the validation set. We show implementation and training details in the appendix.

We summarize our results of the Baselines I in Figure 3 and the results of the Baselines II in Table 1. In Figure 3, BiSHop outperforms both tree-based and deep-learning-based methods by a significant margin in most datasets. In Table 1, BiSHop achieves optimal or near-optimal results with less 10% numbers (on average) of HPO in a tabular benchmark (Grinsztajn et al., 2022).

2 Ablation Studies

We conduct the following sets of ablation studies on Datasets I align with Grinsztajn et al. (2022).

In Figure 3, we change feature sparsity on our datasets following Grinsztajn et al. (2022, Figure 4 & 5). Firstly, we compute the feature importance using Random Forest. Secondly, we remove features in both increasing (solid curves) and decreasing (dashed curves) order of feature importance. For each order, we report the average AUC score over all datasets at each percentage from BiSHop, XGBoost, and LighGBM. Our results (RHS of Figure 3) indicate that BiSHop has the capacity to handle sparse features.

In Table 18, we conduct experiments on rotating the datasets and BiSHopModule’s direction, both individual rotation and combined rotation:

Rotate the 2 directions (row-wise and column-wise)

Our results indicate (i) BiSHop is robust against column-row switch in BiSHopModule, and (ii) BiSHop addresses the Non-Rotationally Invariant Data Structure challenge (C1).

In detail, we report the average AUC score over all datasets at each rotation. We first assess (R1). The results for (R1) show a marginal ($<1\%$) performance drop across datasets. To discuss the rotational invariance problem, we access (R2) by following the same procedure as outlined in Grinsztajn et al. (2022, Section 5.4). The results for (R2) do not indicate a significant drop in performance. Furthermore, the results for (R3) provide further validation for both (R1) and (R2). Our results indicate that BiSHop addresses (C1).

In Table 19, we assess the impacts of stacking different layers of BiSHopModule. We report the average AUC over all datasets at different layers of BiSHopModule. Our results indicate that 4 layers of BiSHopModule marginally maximize the model performance.

We also conduct other ablation studies including:

Component Analysis. In Table 16, we remove each component at a time. We report the implementation details in Section D.1. For each removal, we report averaged AUC scores over all datasets. Overall, each component contributes to varying degrees of performance.

Comparison with the Dense Modern Hopfield Model. In Section D.2, we compare the performance of Sparse, Dense Hopfield Models, and Attention Mechanism. Our results indicate that the generalized Sparse Hopfield Mmdel outperforms the other two methods.

Convergence Analysis. In Section D.3, we compare the converging rate of Sparse and Dense Hopfield Models. Our results indicate that the generalized sparse modern Hopfield model converges faster than the Dense Model.

Appendix D includes all details of ablation experiments.

Conclusion

We address the gap highlighted by Grinsztajn et al. (2022) where deep learning methods trail behind tree-based methods. We present the Bi-Directional Sparse Hopfield Model (BiSHop) for deep tabular learning, inspired by the recent intersection of Hopfield models with attention mechanisms. Leveraging the generalized sparse Hopfield layers as its core component, BiSHop effectively handles the hardness of deep tabular learning, with the inclusion of two important inductive biases of tabular data (C1, C2).

Comparing with Existing Works. Empirically, our model consistently surpasses SOTA tree-based and deep learning methods by 3% across common benchmark datasets. Moreover, our model achieves optimal or near-optimal results within 16% number of HPOs, compared with methods in the tabular benchmark (Grinsztajn et al., 2022). We deem these results as closing the performance gap between DL-based and tree-based tabular learning methods, making BiSHop a promising solution for deep tabular learning.

Limitation. One notable limitation of our study is the non-utilization of the external memory capabilities inherent in modern Hopfield models. We see the integration of these capabilities, especially in memory augmented large models, as a compelling direction for future research.

Boarder Impact

Our work aim at addressing the long standing problem of tabular learning of DL-based model. We do not expect any negative social impact of our work.

While the focus is on tabular learning applications, the perspective isn’t confined to just tabular data. We believe this methodology also presents an opportunity to delve into large foundational models, including extensive language models, through a perspective shaped by contemporary neuroscience.

Acknowledgments

JH would like to thank Stephen Cheng, Dino Feng and Andrew Chen for enlightening discussions, the Red Maple Family for support, and Jiayi Wang for facilitating experimental deployments. The authors would also like to thank the anonymous reviewers and program chairs for their constructive comments.

JH is partially supported by the Walter P. Murphy Fellowship. HL is partially supported by NIH R01LM1372201, NSF CAREER1841569, DOE DE-AC02-07CH11359, DOE LAB 20-2261 and a NSF TRIPODS1740735. This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.

Appendix

Appendix B Supplementary Theoretical Backgrounds

To highlight the computational benefits of the generalized sparse modern Hopfield model, we quote relevant results from (Wu et al., 2024b) here.

We adopt the formal definition of memory storage and retrieval from (Ramsauer et al., 2020) for continuous patterns.

Assuming that every pattern ${\bm{\xi}}_{\mu}$ surrounded by a sphere $S_{\mu}$ with finite radius $R\coloneqq{\frac{1}{2}}\mathop{\rm Min}_{\mu,\nu\in[M]}\norm{{\bm{\xi}}_{\mu}-{\bm{\xi}}_{\nu}}$, we say ${\bm{\xi}}_{\mu}$ is stored if there exists a generalized fixed point of $\mathcal{T}$, $\mathbf{x}^{\star}_{\mu}\in S_{\mu}$, to which all limit points $\mathbf{x}\in S_{\mu}$ converge to, and $S_{\mu}\cap S_{\nu}=\emptyset$ for $\mu\neq\nu$. We say ${\bm{\xi}}_{\mu}$ is $\epsilon$-retrieved by $\mathcal{T}$ with $\mathbf{x}$ for an error.

Then we introduce the definition of pattern separation for later convenience.

Let’s consider a memory pattern ${\bm{\xi}}_{\mu}$ within a set of memory patterns $\bm{\Xi}$.

The separation metric $\Delta_{\mu}$ for ${\bm{\xi}}_{\mu}$ with respect to other memory patterns is the difference between its self-inner product and the maximum inner product with any other pattern:

B.2 Supplementary Theoretical Results for Generalized Sparse Modern Hopfield Model

Let $\mathcal{T}_{\text{Dense}}$ be the retrieval dynamics of the dense modern Hopfield model (Ramsauer et al., 2020). It holds $\norm{\mathcal{T}(\mathbf{x})-{\bm{\xi}}_{\mu}}\leq\norm{\mathcal{T}_{\text{Dense}}(\mathbf{x})-{\bm{\xi}}_{\mu}}$ for all $\mu$.

Theorem B.1 implies two computational advantages:

Computationally, Theorem B.1 suggests that $\mathcal{T}$ converges to fixed points using fewer iterations than $\mathcal{T}_{\text{dense}}$ for the same error tolerance. This means that $\mathcal{T}$ retrieves stored memory patterns more quickly and efficiently than its dense counterpart.

Corollary B.1.1 does not imply computational efficiency. The proposed model’s sparsity falls under the category of sparsity-inducing normalization maps (Tay et al., 2022; Peters et al., 2019; Correia et al., 2019; Krotov and Hopfield, 2016). This means that, during the forward pass, the space complexity remains at $\mathcal{O}(n^{2})$, on par with the dense modern Hopfield model.

Nevertheless, Corollary B.1.1 suggests a specific type of “efficiency" related to faster memory retrieval compared to the dense Hopfield model. In essence, a retrieval dynamic with a smaller error converges faster to the fixed points (stored memories), thereby enhancing efficiency.

Appendix C Experimental Details

All experiments are conducted on the platform with NVIDIA GEFORCE RTX 2080 Ti, A100 GPUs, and INTEL XEON SILVER 4214 @ 2.20GHz.

C.1 Additional Details on Datasets

We describe all the dataset used in our experiments in Table 3, as well as the download links to each dataset in Table 5.

The links to the four OpenML suites from (Grinsztajn et al., 2022) are CC: https://www.openml.org/search?type=benchmark&sort=date&study˙type=task&id=300, NChttps://www.openml.org/search?type=benchmark&study˙type=task&sort=tasks˙included&id=298, CRhttps://www.openml.org/search?type=benchmark&study˙type=task&sort=tasks˙included&id=299, NRhttps://www.openml.org/search?type=benchmark&study˙type=task&sort=tasks˙included&id=297

C.2 Baselines

We evaluate BiSHop by comparing it to state-of-the-art (SOTA) tabular learning methods, specifically choosing top performers in recent studies (Grinsztajn et al., 2022; Somepalli et al., 2021; Gorishniy et al., 2021).

TabPFN (Hollmann et al., 2023). We implement TabPFN using 32 data permutations for ensemble same as the original paper setting and truncate the training set to 1024 instances.

T2G-FORMER (Yan et al., 2023). We implement T2G-FORMER by applying quantile transformation from the Scikit-learn library to Baseline I datsets, aligning with the default setting in. The hyperparameter space is at Table 13.

TANGOS (Jeffares et al., 2023) We adapted the official TANGOS source code to include the datasets from Baseline I alongside the original datasets. The hyperparameter space is at Table 14.

Selection of Benchmark. We select Grinsztajn et al. (2022) as our benchmark for several reasons. Unlike other benchmarks that focus solely on tasks such as classification (Gardner et al., 2023), this benchmark encompasses both regression and classification tasks. This benchmark provides results from 400 hyperparameter optimization (HPO) trials, ensuring each model’s hyperparameter search is sufficient. In contrast, some methods, such as (McElfresh et al., 2023), restrict HPO to 10 hours on a specific GPU. As a deep-learning-based method, BiSHop requires more training time compared to tree-based methods. Moreover, the comparison under the same time constraints on different GPUs is unfair.

C.3 Implementation Details

We label encoded the categorical features, and keep the raw numerical features for further encoding.

For tree based method, we employ the build in categorical embedding method. For MLP we use one-hot encoding.

We implement Piece-wise Linear Encoding from (Gorishniy et al., 2021, 2022) which change the original scalar values of numerical features to a one-hot-like encoding.

For each model hyperparameter configuration, we run 3 experiments on the best configuration and report the average AUC score on the test set.

C.4 Training Details

We use $\mathtt{ReduceLROnPlateau}$ to fine tuning the learning rate to improve convergence and model training progress.

We use Adam optimizer to minimize cross-entropy. The coefficients of Adam optimizer, betas, are set to (0.9, 0.999).

We continue training till there are $\mathtt{Patience=20}$ consecutive epochs where validation loss doesn’t decrease or we reach 200 epochs. Finally, we evaluate our model on test set with the last checkpoint.

We report the number of hpo for each dataset from baseline I in Table 8. We report hyperparameter configurations for CatBoost in Table 9, LightGBM in LABEL:table:HPO_lightgbm, TabNet in LABEL:table:HPO_tabnet, XGBoost in Table 12, T2G-Former in Table 13, Tangos in Table 14. We follow the same procedure of HPOs for Tangos and T2G-Former in Yan et al. (2023) and Jeffares et al. (2023), including the number of trials. For other methods, we follow the same settings as BiSHop.

During random hyperparameter search, we observe that learning rate is the most important hyperparameter (see Table 7). We use WandB "sweep" features (Biewald et al., 2020) to calculate the importance of each hyperparameter. Our findings agree with (Grinsztajn et al., 2022) suggesting that learning rate is the most important hyperparameter for both neural network and gradient-boosted trees.

Appendix D Additional Numerical Experiments

We separately remove each component of BiSHop. We use the default hyperparameters in Table 6 for other components. We report the average AUC score of three runs using the default parameter for all datasets in Table 16.

Without Cat Emb: We remove both individual and shared embedding methods as described in the tabular embedding section, replacing them with PyTorch’s embedding layers (torch.nn.Embedding) and keep the embedding dimension unchanged.

Without Num Emb: We remove the Piecewise Linear Encoding method for numerical features, directly concatenating numerical features with the output of categorical embedding as detailed in Section 3.1.

Without Patch Embedding: We remove the patch embedding method by setting the stride factor $L$ to 1.

Without Decoder: We remove the decoder blocks in BiSHop and pass the encoded data directly to MLP predictor.

Without BiSHopModule: We replace the column-wise block and row-wise block in the BiSHop module with a MLP of hidden size 512.

The results demonstrate that each component contributes to varying degrees to the BiSHop model, with numerical embedding, decoder blocks, and the BiSHopModule being the most significant contributors.

D.2 Comparison with the Dense Modern Hopfield Model

Using the default hyperparameters of BiSHop, we evaluate its performance using three distinct layers: (i) the GSH (generalized sparse modern Hopfield model), (ii) the Hopfield (dense modern Hopfield model (Ramsauer et al., 2020)) and (iii) Attn (attention mechanism (Vaswani et al., 2017)). We report the average AUC score over 10 runs in LABEL:tab:abl_main.

D.3 Convergence Analysis

We calculate the validation loss and AUC score using the same default parameters and compare them with the dense modern Hopfield model. For ease of presentation, we only plot the results of six datasets (Blastchar, Shrutime, Income, Bank, Qsar and Jannis). We use the same hyperparameter for each dataset for both $\mathtt{GSH}$ and $\mathtt{Hopfield}$. We plot the results in Figure 4 with the mean of 30 runs. The result indicate that $\mathtt{GSH}$ converges faster and achieves an AUC score that is equal to or higher than $\mathtt{Hopfield}$.

D.4 Rotation Invariance

In Table 18, we conduct the following experiments on rotating the datasets and BiSHopModule’s direction, both individually and in combination:

Rotate the 2 directions (row-wise and column-wise). To validate the effectiveness of bi-directional design in BiSHop, we conduct experiments by rotating these directions and reporting the performance and average result. The results indicate that the direction of BiSHop is vital for performance.

Rotate the datasets. Following the experiment setup in (Grinsztajn et al., 2022), we randomly rotate datasets using a randomly generated special orthogonal matrix. The results indicate that BiSHop is robust against data rotation.

Rotate the 2 directions and the datasets. To further validate our findings, we then apply both (R1) and (R2). The results show a drop in performance across nearly every dataset and align with our findings in (R1) and (R2).

The average AUC score across all datasets is reported for each type of rotation.

D.5 Hierarchy of BiSHopModule

In Table 19, we assess the impacts of stacking different layers of BiSHopModule. We report the average AUC over all datasets for different layers of BiSHopModule.

We progressively increase the layers within BiSHopModule from 1 to 8 in the Encoder and Decoder layers and keep other parameters at their default setting.

Table 19 summarizes the performance in AUC and average over all datasets for various layers. The results suggest that 4 layers are the optimal setting to maximize the performance.

Appendix E Computational Time

We summarize the computational complexity for each function used in BiSHop in Table 20. Here we use the same notation as introduced in the main paper: $N^{\text{cat}}$ be the number of categorical features, $N^{\text{num}}$ be the number of numerical features, $N=N^{\text{num}}+N^{\text{cat}}$ be the total number of all features, $G$ be the embedding dimension. $P$ be the patch embedding dimension, $D^{\text{model}}$ be the hidden dimension, $\text{len}(Q)$ be the size of query pattern, $C$ be the number of pooling vectors, $\text{len}(Y)$ be the size of memory pattern.

As for the computational complexity of the GSH layers (Wu et al., 2024b), a theoretical analysis of the efficiency of modern Hopfield models can be found in (Hu et al., 2024c).

For each dataset and hyperparameter configuration, the average training time for BiSHop varies from 30 minutes to 2 hours. Based on different hyperparameter settings, number of our model parameters varies from $10^{7}$ to $10^{8}$.

References