TUDataset: A collection of benchmark datasets for learning with graphs

Introduction

Graph-structured data is ubiquitous across application domains ranging from chemo- and bioinformatics (Barabasi & Oltvai, 2004; Stokes et al., 2020) to image (Simonovsky & Komodakis, 2017) and social network analysis (Easley & Kleinberg, 2010). To develop successful machine learning models in these domains, we need techniques that can exploit the rich information inherent in the graph structure and the feature information contained within nodes and edges. In recent years, numerous approaches have been proposed for machine learning with graphs—most notably, approaches based on graph kernels (Kriege et al., 2020) and graph neural networks (GNNs) (Scarselli et al., 2009; Gilmer et al., 2017). However, most papers, even recent ones, evaluate newly proposed architectures or methods on a fixed set of small-scale, non-diverse benchmarks, using non-standardized experimental protocols and baselines, hindering the comparison of results from different publications.

Present work. Here, we give an overview of TUDataset. The benchmark collection consists of over 120 datasets from a wide range of domains for supervised learning with graphs, i.e., classification and regression. All datasets are provided in a standard dataset format at www.graphlearning.io and are easily accessible from popular graph learning frameworks such as Pytorch Geometric (Fey & Lenssen, 2019)https://pytorch-geometric.readthedocs.io/en/latest/modules/datasets.html and DGL (Wang et al., 2019)https://docs.dgl.ai/en/0.4.x/api/python/data.html. To facilitate a standard comparison of kernel and neural approaches, we provide implementations of standard algorithms and easy-to-use evaluation procedures. Moreover, we report results on an experimental study comparing graph kernels and GNNs on a subset of the TUDataset.

Related work. There exist two main approaches to supervised learning with graphs, graph kernels and graph neural networks (GNNs). Graph kernels have been studied extensively in the past 15 years, see (Kriege et al., 2020) for a thorough overview. Important approaches include random-walk and shortest paths based kernels (Gärtner et al., 2003; Sugiyama & Borgwardt, 2015; Borgwardt & Kriegel, 2005; Kriege et al., 2019), as well as the Weisfeiler-Lehman subtree kernel (Shervashidze et al., 2011; Morris et al., 2017). Further recent works focus on approaches based on assignments (Kriege et al., 2016; Nikolentzos et al., 2017), spectral properties (Kondor & Pan, 2016), graph decomposition (Nikolentzos et al., 2018), randomized binning (Heimann et al., 2019), and the extension of kernels based on the Weisfeiler-Leman algorithm (Togninalli et al., 2019; Rieck et al., 2019). For a theoretical investigation of graph kernels, see (Kriege et al., 2018b). Recently, graph neural networks (Gilmer et al., 2017; Scarselli et al., 2009) emerged as an alternative to graph kernels. Notable instances of this architecture include, e.g., (Duvenaud et al., 2015; Hamilton et al., 2017; Velickovic et al., 2018), and the spectral approaches proposed in, e.g., (Bruna et al., 2014; Defferrard et al., 2016; Kipf & Welling, 2017; Monti et al., 2017)—all of which descend from early work in (Kireev, 1995; Merkwirth & Lengauer, 2005; Sperduti & Starita, 1997; Scarselli et al., 2009). A survey of recent advancements in GNN techniques can be found, e.g., in (Chami et al., 2020; Wu et al., 2019; Zhou et al., 2018).

The papers (Fey & Lenssen, 2019; Chen et al., 2019b; Errica et al., 2019; Dwivedi et al., 2020) evaluate GNNs using a unified evaluation procedure, however, they only use small- or medium-scale datasets. Recently, ogb.stanford.edu (Hu et al., 2020) launched, however, the provided datasets for graph classification focus on chemistry and bioinformatic applications, and the number is quite limited at this point. Moreover, the datasets proposed in (Ferber et al., 2019) focuses on instances from planning competitions. Recent efforts to implement graph kernels in a common framework such as the GraKeL library (Siglidis et al., 2018) foster comparability, but do not solve the dataset related issues discussed above, and only focus on kernel approaches.

The TUDataset collection

The TUDataset collection contains over 120 datasets provided at www.graphlearning.io. The datasets, baseline methods and experimental evaluation tools can be conveniently accessed from the Python interface, see Appendix A for further details. Dataset statistics and further documentation is available at our website.

Our collection of datasets covers graphs from various domains, contributed by different authors. Therefore, they differ regarding the used graph model even within the same domain and the provided annotations, e.g., discrete or continuous node and edge attributes. Here, we give an overview of some representative domains and graph models.

Small molecules. A common class of graph datasets consists of small molecules with class labels representing, e.g., toxicity or biological activity determined in drug discovery projects. Here, a graph represents a molecule, i.e., nodes take the places of atoms and edges that of chemical bonds. Consequently, the labels encode atom and bond types, possibly with additional chemical attributes. The graph models differ, e.g., in whether hydrogen atoms are represented explicitly by nodes, and bonds in aromatic rings are annotated accordingly.

Our collection contains small datasets commonly used in the early graph kernel literature such as Mutag (Debnath et al., 1991) and Ptc (Helma et al., 2001), medium-sized datasets, e.g., Nci1 and Nci109 (Wale et al., 2008; Shervashidze et al., 2011), as well as several large datasets derived from the Tox21 challenge 2014 or PubChem (Kim et al., 2018). This includes the eleven datasets from anticancer screen tests with different cancer cell lines used by Yan et al. (2008) to demonstrate the efficacy of classifiers based on significant graph patterns. These datasets, the largest of which contains more than 79k graphs, are typically not balanced and contain far more small molecules that are identified as inactive against cancer cells. Moreover, our collection also contains large-scale molecular regression tasks such as Alchemy (Chen et al., 2019a), Qm9 (Ramakrishnan et al., 2014), and Zinc (Dwivedi et al., 2020; Jin et al., 2018). The first two contain 3D coordinates of the nodes, which should be taken into account in a rotation-invariant manner to benefit from the geometrical information.

Bioinformatics. The datasets DD, Enzymes and Proteins represent macromolecules. Borgwardt et al. (2005) introduced a graph model for proteins, where nodes represent secondary structure elements and are annotated by their type, i.e., helix, sheet, or turn, as well as several physical and chemical information. An edge connects two nodes if they are neighbors along the amino acid sequence or one of three nearest neighbors in space. Using this approach, the dataset Enzymes was derived from the Brenda database (Schomburg et al., 2004). Here, the task is to assign enzymes to one of the 6 EC top-level classes, which reflect the catalyzed chemical reaction. Similarly, the dataset Proteins was derived from (Dobson & Doig, 2003), and the task is to predict whether a protein is an enzyme. The dataset DD used by Shervashidze et al. (2011) is based on the same data, but contains graphs, where nodes represent individual amino acids and edges their spatial proximity.

Computer vision. Graph-based methods are widely used in computer vision for various tasks using diverse graph models. Our collection provides several datasets originating from the IAM Graph Database (Riesen & Bunke, 2008) such as Letter and Fingerprint. Other datasets represent Cuneiform signs (Kriege et al., 2018a), 3D point clouds for robot grasping tasks (FirstMM_DB) and semantic image processing (Msrc) (Neumann et al., 2016).

Social networks. Yanardag & Vishwanathan (2015) introduced several graph classification datasets derived from social networks. In the Reddit datasets, each graph represents a discussion thread, where nodes correspond to users, two of which are connected by an edge if one responded to a comment of the other. This graph model is used to derive several datasets, where the classification task is to distinguish either discussion-based and question/answer-based subreddits (Reddit-Binary) or predict the subreddit, where the thread was posted (Reddit-Multi-5K and Reddit-Multi-12K). Collab are datasets derived from scientific collaboration networks. Each graph is the ego-network of a researcher, and the task is to predict their research field, i.e., high energy, condensed matter, or astrophysics. Similarly, the Imdb datasets consist of ego-networks derived from actor collaborations, and the task is to predict the genre, e.g., Action vs. Romance. Rozemberczki et al. (2020) used similar approaches to obtain more massive social network datasets. reddit_threads contains more than 200k graphs with the task to predict whether a thread is discussion-based. deezer_ego_nets and twitch_egos contain ego-networks derived from online services, and the task is to predict the gender and play behavior (single or multiple games) of the central user. github_stargazers contains graphs representing the social networks of GitHub users divided into those who starred popular machine learning and web development repositories.

Recently, temporal graphs were considered by Oettershagen et al. (2020), where edges represent the contact or interaction between two individuals at a certain point in time. These graphs are of interest when studying dissemination processes such as the spreading of epidemics, rumours or fake news. We provide temporal graph classification datasets derived from Tumblr (Rozenshtein et al., 2016), Dblp and Facebook (Viswanath et al., 2009) as well as contacts between students at the MIT (Eagle & Pentland, 2006), in a Highschool and visitors at the Infectious exhibition (Isella et al., 2011).

Synthetic. Several graph datasets were generated to demonstrate the strengths or weaknesses of specific methods. The datasets SyntheticNew and Synthie were created by Feragen et al. (2013) (see Erratum) and Morris et al. (2016), respectively, to demonstrate the ability of kernels to operate on graphs with continuous attributes. Knyazev et al. (2019) introduced the datasets Colors and Triangles, where the task is to count the number of nodes with a given one-hot-encoded color and the number of triangles, respectively.

2 Baselines methods

To provide meaningful baselines, we provide implementations of common graph kernels as well as GNN architectures. We have implemented the Weisfeiler-Lehman Subtree (Shervashidze et al., 2011), Shortest-path (Borgwardt & Kriegel, 2005), Graphlet (Shervashidze et al., 2009) (using labeled subgraphs with three nodes), Weisfeiler-Lehman Optimal Assignment (Kriege et al., 2016) kernel in $\text{C}^{{}_{{}_{{}_{++}}}}$and made them accessible through the Python interface of TUDataset. Moreover, all GNN architectures provided by PyTorch Geometric can be conveniently used as a baseline as well.

3 Evaluation methods

To ensure a fair and meaningful comparison between methods, we propose the following evaluation procedures for kernels and GNNs. First, for kernels, we propose the established $C$-SVM implementation LibSvm (Chang & Lin, 2011) for kernels that compute a Gram matrix, and the linear $C$-SVM implementation LibLinear (Fan et al., 2008) for kernels that can be computed based on sparse, explicit feature maps. We optimize GNNs end-to-end using Adam (Kingma & Ba, 2015). To compute classification accuracies, we propose to use $10$-fold cross-validation, where we select $10\%$ of each training fold uniformly at random as validation set to optimize hyperparameters, e.g., the number of iterations, $C$ parameter, number of layers, feature dimension. We repeat the above evaluation ten times and report standard deviations over all ten repetitions, and additionally across all one hundred runs (i.e., ten repetitions with ten folds each). See Appendix B in the appendix for examples. For the large-scale molecule learning tasks, we either use random splits (80%/10%/10%) or the provided, fixed splits and report MAE (mean std. MAE, mean std. logMAE for multi-target regression, see (Klicpera et al., 2020)), over five runs.

Experimental evaluation

Our intent here is to provide baseline experiments and compare graph kernels and GNNs. We used the following datasets, graph kernels, and GNN baselines.

Datasets. We used the deezer_ego_nets, github_stargazers, Enymes, Imdb-Binary, Imdb-Multi, Mcf-7, Molt-4, NCI1, Proteins, Reddit-Binary, reddit_threads, twitch_egos, Uacc257 graph classification datasets. Moreover, we used the Alchemy, QM9, Zinc (multi-target) regression datasets. See the website and Table 4 in the appendix for dataset statistics. We opted for not using continuous node features of the small datasets (if available) and the 3D-coordinates of the Alchemy dataset to solely provide baseline results based on graph structure and discrete labels. In case of the QM9 dataset, we closely replicated the (continuous) node and edge features of Gilmer et al. (2017).

Graph kernels. As kernel baselines we used the Weisfeiler-Lehman Subtree ($1$-WL) (Shervashidze et al., 2011), Shortest-path (SP) (Borgwardt & Kriegel, 2005), Graphlet (GR) (Shervashidze et al., 2009), Weisfeiler-Lehman Optimal Assigment (WL-OA) (Kriege et al., 2016) included in the TUDataset package. The $C$-parameter was selected from $\{10^{-3},10^{-2},\dotsc,10^{2},$ $10^{3}\}$ from the validation set. For the larger datasets, we computed sparse feature vectors for each graph and used the linear $C$-SVM implementation of LibLinear (Fan et al., 2008). The number of iterations of the $1$-WL and WL-OA were selected from $\{0,\dotsc,5\}$.As already shown in (Shervashidze et al., 2011), choosing the number of iterations too large will lead to overfitting.

GNNs. For comparison with kernel methods, we used Gin-$\varepsilon$ (Xu et al., 2019) and Gin-$\varepsilon$-JK with jumping knowledge networks as neural baselines (Xu et al., 2018). For data with (continuous) edge features, we used a $2$-layer MLP to map them to the same number of components as the node features and combined them using summation (Gine-$\varepsilon$ and Gine-$\varepsilon$-JK). We used mean pooling to pool the learned node embeddings to a graph embedding and used a $2$-layer MLP for the final classification, using a dropout layer with $p=0.5$ after the first layer of the MLP. For the smaller datasets of Table 1, we optimized the number of hidden units from $\{32,64,128\}$, the number of layers from $\{1,2,3,4,5\}$ using the validation set. For the mid-scale datasets, due to computation time constraints, we set the number of hidden units to $64$ and the number of layers to $3$. Moreover, for both, we use a learning rate decay of $0.5$ with a patience parameter of $5$, a starting learning rate of $0.01$ and a minimum of $10^{-6}$, and a maximum epoch number of $200$. For both methods, we used the evaluation procedure described in Section 2.3 to optimize hyperparameters and compute accuracies. See the appendix for details on the hyperparameter and evaluation protocols used for the larger molecular regression tasks (Zinc, and Alchemy, QM9).

Results and discussion. Tables 1, 2 and 3 summarize the results. On the small-scale datasets, see Table 1, the WL-OA performs best overall. However, it does not scale to large datasets, Table 2, as it relies on Gram matrix computation. Here, the $1$-WL performs well on all datasets, excluding github_stargazers, where the neural baselines perform best overall.For the neural baselines unlike the kernel baselines, we used one-hot degree features for datasets that do not provide node labels. Our results show that despite the extensive research on GNNs in recent years, classical graph kernels in combination with SVMs are still highly competitive in graph classification tasks.

On the large-scale molecular learning tasks, see Table 3, it becomes apparent that specialized architectures such as MPNN result in significant gains over the generic Gine-$\varepsilon$ baseline.

Conclusion

We gave an overview of the TUDataset collection, and reported on the results of an experimental study comparing graph kernels and GNNs on a subset of the data. We believe that our dataset collection will spark further progress in graph representation learning, and that our unified evaluation procedures will improve the comparability of results. We are looking forward to adding more datasets and are excited about contributions from the community, researchers, and practitioners from other areas. Future work includes a more extensive comparision of kernel and neural approaches on large-scale molecular regression tasks with continuous node and edge features.

Acknowledgement

We thank everybody who provided datasets for the TUDataset collection.

This work has been partially funded by the Deutsche Forschungsgemeinschaft (DFG) within the Collaborative Research Center SFB 876 “Providing Information by Resource-Constrained Data Analysis”, project A6 “Resource-efficient Graph Mining”. Nils Kriege has been supported by the Vienna Science and Technology Fund (WWTF) through project VRG19-009.

References

Appendix A Evaluation examples

See www.graphlearning.io for further documentation.

Kernelized SVM for graph kernels based on Gram matrices

Linear SVM for graph kernels based on sparse feature maps

Appendix B Experimental protocol and hyperparameters for Zinc, Alchemy, QM9

For the larger molecular regression tasks, Zinc and Alchemy,Note that the full dataset is different from the contest dataset, e.g., it does not provide normalized targets, see https://alchemy.tencent.com/. we closely followed the hyperparameters found in (Dwivedi et al., 2020) and (Chen et al., 2019a), respectively, for the Gine-$\varepsilon$ layers. That is, for Zinc, we used four Gine-$\varepsilon$ layers with a hidden dimension of 256 followed by batch norm and a $4$-layer MLP for the joint regression of the twelve targets, after applying mean pooling. For Alchemy and QM9, we used six layers with 64 (hidden) node features and a set2seq layer (Vinyals et al., 2016) for graph-level pooling, followed by a $2$-layer MLP for the joint regression of the twelve targets.

For Zinc, we used the given train, validation split, test split, and report the MAE over the test set. For the Alchemy and QM9 datasets, we uniformly and at random sampled 80% of the graphs for training, and 10% for validation and testing, respectively. Moreover, following (Chen et al., 2019a; Gilmer et al., 2017), we normalized the targets of the training split to zero mean and unit variance. We used a single model to predict all targets. Following (Klicpera et al., 2020), we report mean standardized MAE and mean standardized logMAE. We repeated each experiment five times (with different random splits in case of Alchemy and QM9) and report average scores and standard deviations. Moreover, we use a learning rate decay of $0.5$ with a patience parameter of $5$, and a starting learning rate of $0.001$ with a minimum of $10^{-6}$.

All neural experiments were conducted on a workstation with four Nvidia Tesla V100 GPU cards with 32GB of GPU memory running Oracle Linux Server 7.7.

Appendix C Dataset statistics