PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations

Introduction

In recent years, Graph Convolutional Networks (GCNs) have drawn the attention of researchers and practitioners in a variety of domains and applications, ranging from computer vision and graphics to computational biology , recommendation systems and social network analysis . However, GCNs still suffer from two main problems. First, they are usually shallow as opposed to the concept of deep convolutional neural networks (CNNs) due to the over-smoothing phenomenon , where the node feature vectors become almost identical, such that they are indistinguishable, which yields non-optimal performance. Furthermore, GCNs are typically customized to a specific domain and application. That is, as we demonstrate in Sec. 4.1, a successful point-cloud classification network can perform poorly on a citation graph node-classification problem , and vice-versa. Furthermore, because many GCNs lack theoretical guarantees, it is difficult to reason about their success on one problem and lack on others. These observations motivate us to develop a profound understanding of graph networks and their dynamics.

To this end, we suggest a novel, universal approach to the design of GCN architectures. Our inspiration stems from the similarities and equivalence between Partial Differential Equations (PDEs) and deep networks explored in . Furthermore, as GCNs can be thought of as a generalization of CNNs, and a standard convolution can be represented as a combination of differential operators on a structured grid , we adopt this interpretation to explore versions of GCNs as PDEs on graphs or manifolds. We therefore call our network architectures PDE-GCN, and demonstrate that our approach is general with respect to the given task. That is, our architectures behave similarly for different problems, and their performance is on par or better than other domain-specific GCNs. Furthermore, our family of architectures are backed by theoretical guarantees that allow us to explain the behavior of the GCNs in some of the results that we present. To be more specific, our contribution is as follows:

We introduce and implement general graph convolution operators, based on graph gradient and divergence. This abstraction of the spatial operation on the graph leads to a more general and flexible approach for architecture design.

We propose treating a variety of graph related problems as discretized PDEs, and formulate the dynamics that match different problems such as node-classification and dense shape-correspondence. This is in direct effort to propose a family of networks that can solve multiple problems, instead of GCNs which are tailored for a specific application.

Our method allows constructing a deep GCN without over-smoothing, with theoretical guarantees.

We validate our method by conducting numerous experiments on multiple datasets and applications, achieving significantly better or similar accuracy compared to state-of-the-art models.

Related work

GCNs are typically divided into spectral and spatial categories. Most of those can be implemented using the Message-Passing Neural Network paradigm , where each node aggregates features (messages) from its neighbors, according to some scheme. The works use polynomials of the graph Laplacian to parameterize the convolution operator. DGCNN constructs a k-nearest-neighbors graph from point-clouds and dynamically updates it. MoNet learns a Gaussian mixture model to weight the edges of meshes for shape analysis tasks. Works like GraphSAGE and GAT propose methods for inductive and transductive learning on non-geometrical graphs.

Several of the methods above suffer from over-smoothing , leading to undesired node features similarity for deep networks. To overcome this problem, some approaches rely on imposing regularization and augmentation. For example, PairNorm introduces a novel normalization layer, and DropEdge randomly removes edges to decrease the degree of nodes. Other methods prevent over-smoothing by dedicated construction. For instance, JKNet combines all intermediate representations at each stage of the network. APPNP replaces learnt convolutional layers with a pre-defined kernel based on PageRank, yielding a shallow network which preserves locality, making it robust to over-smoothing. GCNII proposes to add an identity residual where the initial features of the networks are added to the features of the $l$-th layer, scaled by some coefficient.

Another approach is to construct a network that inherently does not over-smooth, as we suggest in this work. Our network is based on discretized PDEs, hence we are able to motivate our choices by well studied theory and numerical experiments . On a similar note, the recent DiffGCN also makes use of discretized operators to approximate the graph gradient and Laplacian. However, DiffGCN is specifically tailored for geometric tasks since it projects its components on the $x,y,z$ axes, and is applied using a ResNet (diffusion) structure only. The recent GRAND applies attention mechanism with diffusive dynamics, using several integration schemes. Here we propose a network for both geometric and non-geometric tasks, and also utilize both the diffusion or hyperbolic layer dynamics, including a mixture of the two.

PDEs and CNNs:

In a recent series of works, the connection between PDEs and CNNs was studied . It was shown that it is possible to treat a deep neural network as a dynamical system driven by some PDE, where each convolution layer is considered a time step of a discretized PDE. The connection between PDEs and CNNs was also used to reduce the computational burden . Besides the interpretation of CNNs as a PDE solver, it was shown that it is also possible to learn a symbolic representation of PDEs in a task-and data driven approach in . In the context of GCNs and PDEs, it was recently shown that GCNs can be utilized to enhance the solution of PDEs for problems like fluid flow dynamics. However, in this work, we harness PDE concepts to design and construct GCNs for a variety of applications.

Methods

We show now that GCNs can be viewed as discretizations of PDEs on manifolds, similarly to CNNs that are viewed as discretized PDEs on regular grids . Consider a general manifold ${\cal M}$ where a vector function $f$ resides (also dubbed the features function), along with its continuous differential operators such as the gradient $\nabla$, divergence $\nabla\cdot$ and the Laplacian $\Delta$ that reside on the manifold ${\cal M}$.

Given these differential operators, one can model different processes on ${\cal M}$. In particular, we consider two PDEs – the non-linear diffusion and the non-linear hyperbolic equations

respectively, equipped with appropriate boundary conditions. Here $K$ is a coefficient matrix that can change in time and represents the propagation over the manifold ${\cal M}$, $K^{*}$ is its conjugate transpose and $\sigma(\cdot)$ is a non-linear activation function. Eq. (1)-(2) define a non-linear operator that takes initial feature vectors $f^{0}$ at time and propagates them to time $T$, yielding $f^{T}$ where they can be used for different tasks. We now provide two theorems that characterize the behavior of Eq. (1)-(2), based on ideas from See proofs in Appendix A..

If the activation function $\sigma(\cdot)$ is monotonically non-decreasing and sign-preserving, then the forward propagation through the diffusive PDE in (1) for $t\in[0,\infty)$ yields a non-increasing feature norm, that is,

Assume that the activation function $\sigma(\cdot)$ is monotonically non-decreasing, sign-preserving and satisfies $|\sigma(x)|\leq|x|$, and define energy

then the forward propagation through the hyperbolic PDE in (2) satisfies ${\cal E}_{net}\leq c_{K}$, where $c_{K}$ is a constant that depends on $K$ but independent of time.

The outcome of those theorems is that the dynamics described in Eq. (1) is smoothing, while the one in Eq. (2) is bounded by a conserving mapping. An illustration of this behavior is presented in Fig. 1.

In the physical world, diffusion and hyperbolic equations are used for different applications. Similarly, many computational models for image segmentation , denoising and deblurring are based on anisotropic diffusion which are similar to the model in Eq. (1). On the other hand, applications that require conservation such as volume/distance preservation as in the dense shape correspondence task and protein folding , are typically better treated using a hyperbolic equation as in Eq. (2). Those insights motivate us to construct two types of layers according to Eq. (1)-(2) using discretized differential operators on graphs.

2 Discretized differential operators on graphs

where nodes $i$ and $j$ are connected via the $(i,j)$-th edge, ${\bf W}_{ij}$ is an edge weight matrix which can be learnt, and ${\bf f}_{i}$ and ${\bf f}_{j}$ are the features on the $i$-th and $j$-th nodes, respectively. The gradient operator can be thought of as a weighted directional derivative of the function $f$ in the direction defined by the nodes $i$ and $j$. Furthermore, if we choose the scaled identity matrix ${\bf W}_{ij}=d_{ij}^{-1}{\bf I}$, where $d_{ij}$ is the distance between the two nodes, then the discrete gradient is a second order approximation to the true gradient of the function ${f}$ on the edges of the graph. In this work, we use ${\bf W}_{ij}=\gamma_{ij}{\bf I}$, where the scale $\gamma_{ij}$ is the geometric mean of the degree of the nodes $i,j$. Note, that the gradient operator is a mapping from the vertex space to the edge space.

Given the gradient matrix, it is possible to define the divergence matrix , which is an approach that is used extensively in mimetic discretizations of PDEs. To this end, we define the inner product between an edge feature vector ${\bf q}$ and the gradient of a node feature vector ${\bf f}$ as

The divergence is naturally defined as the operator that maps edge operator $\bf q$ to the node space, that is $\nabla\cdot\approx-{\bf G}^{\top}$. As usual, the graph Laplacian operator can be obtained by taking the divergence of the gradient. In graph theory it is defined as a positive matrix that is, $\Delta\approx{\bf G}^{\top}{\bf G}$ In the field of numerical PDEs, the Laplacian is defined as $-{\bf G}^{\top}{\bf G}$, but the combinatorial Laplacian is defined as ${\bf G}^{\top}{\bf G}$. .

We also define the weighted line integral over an edge. Similarly to Eq. (3)-(4), we define

The operator $\bf A$ approximates the mass operator on the edges. The right equation in Eq. (5) suggests that an appropriate averaging operator for edge features is the transpose of the nodal edge average.

The advantage of defining such operators is that we are able to design networks with architectures that mimic the continuous operators (1) and (2) on the discrete level, as we show in the next section.

3 PDE-GCN: Graph Convolutional Networks by Partial Differential Equations

In order to use the computational models in Eq. (1)-(2), we form their discrete versions:

Here, in Eq. (6) we use the forward Euler to discretize Eq. (1), and in Eq. (7) we discretize the second order time derivative in Eq. (2), using the leapfrog method. In both cases, ${\bf f}^{(l)}$ are the node features and ${\bf K}_{l}$ is a $1\times 1$ trainable convolution of the $l$-th layer. The similarity between (6) and ResNet is well documented in the context of CNNs . The hyper-parameter $h$ is the step-size, and it is chosen such that the stability of the discretization is kept. We use $\sigma=\tanh$ for the activation function as it yields slightly better results in our experiments, although other functions such as ReLU can also be used, as reported in Sec. 4.7. Each of Eq. (6)-(7) defines a PDE-GCN block. We denote the former (diffusive equation) by PDE-GCND and the latter (hyperbolic equation) by PDE-GCNH.

To complete the description of our network, a few more details are required, as follows:

The (opening) embedding layer. The input vertex features ${\mathbf{u}}_{\cal V}$ are fed through an embedding ($1\times 1$ convolution) layer ${\bf K}_{o}$ to obtain the initial features ${\bf f}_{0}$ of our PDE-GCN network: ${\bf f}_{0}={\bf K}_{o}{\bf u}_{\cal V}$. In cases where input edge attributes (features) $\mathbf{u}_{\cal E}$ are available (as in the experiment in Sec. 4.6), we transform them to the vertex space by taking their divergence and average, and concatenate them to the input of the embedding layer ${\bf K}_{o}$ as follows: ${\bf f}_{0}={\bf K}_{o}({\bf u}_{\cal V}\oplus{\bf A}^{\top}{\bf u}_{\cal E}\oplus{\bf G}^{\top}{\bf u}_{\cal E})$.

The (closing) embedding layer. Given the final vertex features ${\bf f}^{(L)}$ of our PDE-GCN with $L$ layers, we obtain the output of the network by performing: ${\bf u}_{out}={\bf K}_{c}{\bf f}^{(L)}$. Here ${\bf K}_{c}$ is a $1\times 1$ convolution layer mapping the hidden feature space to the output shape.

Initialization. The $1\times 1$ convolutions ${\bf K}_{l}$ in Eq. (6)-(7) are initialized as identity. This way, the network begins from a diffusion/hyperbolic equation, which serves as a prior and guides the network to initially behave like classical methods and to further improve during training.

The choice of dynamics. For some applications, anisotropic diffusion is appropriate, while for others conservation is more important. However, in some applications this may not be clear. To this end, it is possible to combine Eq. (1)-(2) to obtain the continuous process

where $\alpha=sigmoid(\beta)$, meaning $0\leq\alpha\leq 1$, and $\beta$ is a single trainable parameter. The discretization of this PDE leads to the following network dynamics:

where ${\bf f}^{(l+1)}$ is updated by the known ${\bf f}^{(l)}$ and ${\bf f}^{(l-1)}$. We denote a layer that is governed by Eq. (9) by PDE-GCNM (where M stands for mixture). Note, that it is also possible to learn a combination coefficient $\alpha_{i}$ per layer, although we did not read a benefit from such scheme.

As we show in our numerical experiments, learning $\alpha$ yields results that are consistent with our understanding of the problem, that is, graph node-classification gravitates towards no second order derivatives while applications that require conservation gravitate towards the hyperbolic equation.

Experiments

In this section we demonstrate our approach on various problems from different fields and applications ranging from 3D shape-classification to protein-protein interaction and node-classification . The experiments also vary in their output type. Node classification is similar to segmentation problems that are typically solved by anisotropic diffusion while the dense shape correspondence problem is conservative and therefore can be thought of as a problem that does not require smoothing.

In all experiments, we use the suitable PDE-GCN (D, H or M) block as described in Sec. 3, with various depths (number of layers) and widths (number of channels), as well as the appropriate final convolution steps, depending on the task at hand. A detailed description of the architectures used in our experiments is given in Appendix B. We use the Adam optimizer in all experiments, and perform grid search over the hyper-parameters of our network. The selected hyper-parameters are reported in Appendix C. Our objective function in all experiments is the cross-entropy loss, besides inductive learning on PPI where we use the binary cross-entropy loss. Our code is implemented using PyTorch , trained on an Nvidia Titan RTX GPU.

We show that for all the considered tasks and datasets, our method is either remarkably better or on par with state-of-the-art models.

To gain deeper understanding about the effectiveness and generalization of various GCN methods to different tasks, we start by picking two datasets - ModelNet-10 for 3D shape-classification, and Cora for semi-supervised node classification. Those datasets are not only different in terms of application (global versus local classification), but also stem from different domains. While in ModelNet-10 the data has geometrical meaning, the data in Cora has no obvious geometrical interpretation. Therefore, we suggest that success in both applications should be obtained from a generalizable GCN. We compare our PDE-GCND with two recent and popular networks - DGCNN and GCNII . For ModelNet-10 shape-classification, we randomly sample 1,024 points from each shape to form a point cloud, and connect its points using k-nearest-neighbors (k-nn) algorithm with $k=10$ to obtain a graph and follow the training scheme of . On Cora, we follow the same procedure as in . We evaluate all models with 4 layers, as well as 2 layers for DGCNN on Cora.

Our results, reported in Tab. 1 suggest that while each of the considered methods obtains high accuracy on the dataset it originally was tested on (Cora for GCNII and ModelNet-10 for DGCNN), obtaining a similar measure of success on a different dataset was not possible when using the very same networks. Additionally, while our attempts to add a diffusion-equation dynamics to DGCNN (i.e., updating features as in Eq. (6)) showed an increase in performance – a large gap to state-of-the-art model still exists. On top of that, we also see that DGCNN suffers from over-smoothing, as its accuracy significantly decreases when adding more layers. Last but not least, we observe that our PDE-GCND obtains high accuracy on both datasets, similar or better than state-of-the-art models.

2 Learning PDE network dynamics

In this experiment, we delve on the ability to learn the appropriate PDE that better models a given problem. To this end, we use the mixture model from Eq. (9) so that the resulting PDE is a combination of the diffusion and hyperbolic dynamics. We use a 8 layer mixed PDE-GCN, starting with $\alpha=0.5$, such that it is balanced between a PDE-GCND and PDE-GCNH. By learning the parameter $\alpha$ in (9), we allow to choose a mixed PDE between a purely conservative network and a diffusive one. We consider two problems: semi-supervised node classification on Cora, and dense shape correspondence on FAUST .

Our results, reported in Fig. 2 suggest that just as in classical works , problems like node-classification obtain better performance with an anisotropic diffusion like in Eq. (6), and for problems involving dense-correspondences like in that tend to conserve the energy of the underlying problem, a hyperbolic equation type of PDE as in Eq. (7) is more appropriate.

3 Semi-supervised node classification

In this set of experiments we use three datasets – Cora, Citeseer and PubMed . For all datasets we use the standard training/validation/testing split as in , with 20 nodes per class for training, 500 validation nodes and 1,000 testing nodes and follow the training scheme of . The statistics of the datasets are reported in Tab. 2. We compare our results using PDE-GCND with recent and popular models like GCN , GAT , APPNP , JKNet and DropEdge . We note that our network does not over-smooth, as an increase in the number of layers does not cause performance degradation. For example, on CiteSeer, we obtain $75.6\%$ accuracy with 32 layers, compared to $74.6\%$ with two layersNote that this result is also a new state-of-the-art accuracy.. Overall, our results in Tab. 3 show that our PDE-GCND achieves similar or better accuracy than the considered methods.

4 Fully-supervised node classification

We follow and use 7 datasets: Cora, CiteSeer, PubMed, Chameleon, Cornell, Texas and Wisconsin. We also use the same train/validation/test splits of $60\%,20\%,20\%$, respectively. In addition, we report the average performance over 10 random splits from . We fix the number of channels to 64 and perform grid search to determine the hyper-parameters parameters, which are reported in Appendix C. We compare our network with GCN, GAT, three variants of Geom-GCN , APPNP, JKNet, Incep and GCNII in Tab. 4. Our experiments read either similar or better than the state-of-the-art on Cora, CiteSeer and PubMed datasets. On Chameleon , Cornell, Texas and Wisconsin datasets, we improve state-of-the-art accuracy by a significant margin. For example, we obtain $93.24\%$ accuracy on Texas with our PDE-GCNM, compared to $77.84\%$ with GCNII*. Similar improvements hold for Cornell and Wisconsin datasets. The common factor for these datasets is their small size, as depicted from Tab. 2. We argue that the success of our network stems from its capability of apriori extracting features from graphs, due to its utilization of discretized differential operators and PDE guided construction. On Chameleon , using PDE-GCNM, we improve the current state-of-the-art accuracy of GCNII* from $62.48\%$ to $66.01\%$. Also, we note that unlike in the semi-supervised case, where some of the labels are missing, here it is possible to obtain meaningful results with the hyperbolic equation based PDE-GCNH as we do not have unknown nodes in the fully-supervised case, which would be otherwise preserved using the hyperbolic equation dynamics.

5 Inductive Learning

We follow and employ the PPI dataset for the inductive learning task. We use a 8 layer PDE-GCND network, without dropout or weight-decay, and a learning rate of 0.001. We compare our results with methods like GraphSAGE, GAT, JKNet, GeniePath, GCNII and others. As reported in Tab. 6, our PDE-GCN D achieves $99.07$ Micro-averaged F1 score, superior to methods like GAT, JKNet and GeniePath, also close to state-of-the-art GCNII* with a score of $99.58$.

6 Dense shape correspondence

7 Ablation study

Our method has two main dynamics – diffusion and hyperbolic. To verify our proposal in Sec. 3.1, we examine the performance of the hyperbolic PDE-GCNH on semi-supervised node-classification on Cora and CiteSeer datasets. Our results in Tab. 7 show that indeed for problems where we wish to obtain a piecewise-constant prediction, the diffusive formulation of our network, PDE-GCND, is more suitable. Furthermore, we study the importance of the positive-semi definiteness of our learnt operator as described in Eq. (6)-(7), by removing the ${\bf K}_{l}^{T}$ term from the dynamics equations. This yields a non-symmetric operator that does not guarantee positive-semi-definiteness. We note that the enforcement of the latter is important to obtain higher accuracy which is improved by up to $3\%$ with the introduction of a positive semi-definite operator. Also, we report on the use of $\sigma=ReLU$, from the discussion in Sec. 3.3, where we favor $\tanh$ as an activation function, and the ability of our PDE-GCNM to reproduce the results of PDE-GCND in the case of semi-supervised learning.

Summary

In this paper we explored new architectures for graph neural networks. Our motivation stems from the similarities between graph networks and time dependent partial differential equations that are discretized on manifolds and graphs. By adopting an appropriate PDE, and embedding the finite graph in an infinite manifold, we are able to define networks that are either diffusive, conservative, or a combination of both.

Not all natural phenomena are solved using the same PDE and we should not expect that all graph problems should be solved by the same network dynamics. To this end we allow the data to choose which type of network is appropriate for the solution of the problem (diffusive or hyperbolic). Indeed, numerical experiments show that the network gravitates towards a hyperbolic one for problems where conservation is required, and towards a diffusive one when anisotropic diffusion is favorable.

Finally, we showed that the proposed networks can be made deep without over-smoothing and, can deliver the state-of-the-art performance or improve it for virtually every problem we worked with. In particular, our network dramatically improved the state-of-the-art for problems that are data-poor. We believe that for such problems the structure imposed by our dynamics and operators regularizes the network and therefore yields implicit regularization.

Acknowledgments and Disclosure of Funding

The research reported in this paper was supported by grant no. 2018209 from the United States - Israel Binational Science Foundation (BSF), Jerusalem, Israel. ME is supported by Kreitman High-tech scholarship.

References

Appendix A Theorems and proofs

We repeat the theorems presented in Sec. 3 and provide their proofs below. The theorems hold for Neumann boundary conditions, which we use in our implementation—this is achieved by the construction of the differential operators. The proofs follow the ones presented in .

If the activation function $\sigma(\cdot)$ is monotonically non-decreasing and sign-preserving, then the forward propagation through the diffusive PDE in (1) for $t\in[0,\infty)$ yields a non-increasing feature norm, that is,

Let us examine the following inner product following Eq. (1):

From integration by parts it holds that :

Plugging the definition of an inner product, together with the assumption that $\sigma$ is a sign-preserving function, it follows that:

Therefore, the following is non-positive:

Assume that the activation function $\sigma(\cdot)$ is monotonically non-decreasing, sign-preserving and satisfies $|\sigma(x)|\leq|x|$, and define energy

then the forward propagation through the hyperbolic PDE in (2) satisfies ${\cal E}_{net}\leq c_{K}$, where $c_{K}$ is a constant that depends on $K$ but independent of time.

This energy is associated with the linear hyperbolic (wave-like) equation:

Assuming $K$ is constant in time, we obtain:

This means that the energy ${\cal E}_{lin}$ is constant in time, i.e. there exists some $c_{K}$ such that ${\cal E}_{lin}=c_{K}$.

Also, given our assumption that $\sigma$ is sign-preserving and $|\sigma(x)|\leq|x|$ (i.e., it does not increase the norm of its input), we show that ${\cal E}_{net}\leq{\cal E}_{lin}$:

Therefore, we conclude that ${\cal E}_{net}\leq c_{K}$.

Appendix B Architectures in details

In this section we elaborate on the specific architectures that were used in our experiments in Sec. 4. As discussed in Sec. 3.3, all our network architectures are comprised of an opening layer ($1\times 1$ convolution), a sequence of PDE-GCN layers, and a closing layer ($1\times 1$ convolution), and possibly additional final convolution steps which serve as the classifier. In total, we have three types of architectures in our experiments, which differ in their classifier layers. Throughout the following tables, $c_{in}$ and $c_{out}$ denote the input and output channels, respectively, and $c$ denotes the number of features in hidden layers (which is a hyper-parameter, as given in Appendix C.) We denote the number of PDE-GCN blocks by $L$, and the dropout probability by $p$.

Our first architecture is described in Tab. 8 and includes only a closing layer as a final step. The architecture is used for the semi- and fully supervised node classification tasks (i.e., the experiment on Cora in Sec. 4.1 – 4.2, the experiments in Sec. 4.3 – 4.4 and the ablation study in Sec. 4.7), as well as the inductive learning task on PPI in Sec. 4.5. Note, the high-level architecture is the same as in GCNII , and only differs in the employed GCN-block, which is our PDE-GCN.

The second architecture is described in Tab. 9, and is used for the ModelNet-10 in Sec. 4.1. The difference between this architecture and the one presented in Tab. 8 is that here we perform a global-max pooling operation to obtain a global shape class prediction. Following this pooling operation, we add two multi-layer perceptron (MLP) layers, where each consists of a $1\times 1$ convolution, $ReLU$ activation, batch normalization and dropout with probability of $0.5$. Finally, a fully connected convolution layer is applied to obtain the prediction.

The third architecture is used for the dense-shape correspondence task on FAUST in Sec. 4.6 is given in Tab. 10. In addition to the closing $1\times 1$ convolution layer, it also includes a layer of a $1\times 1$ convolution and an $ELU$ activation, followed by another final $1\times 1$ convolution which classifies the point-to-point correspondence. In the case of the FAUST dataset, each mesh has $n=6890$ vertices.

Appendix C Hyper-parameters details

We provide the selected hyper-parameters in our experiments, besides for the inductive learning on PPI (Sec. 4.5) and dense shape correspondence (Sec. 4.6) which are reported in the main paper. We denote the learning rate of our PDE-GCN layers by by $LR_{GCN}$, and the learning rate of the $1\times 1$ opening and closing as well as any additional classifier layers by $LR_{oc}$. Also, the weight decay for the opening and closing layers is denoted by $WD_{oc}$. For the PDE-GCN layers, no weight decay is used throughout all experiments.

For semi-supervised node-classification on Cora, for GCNII we used the same settings as in the original paper of GCNII. For DGCNN and our PDE-GCNH we used the same hyper-parameters as reported in Tab. 12.

For the ModelNet-10 classification we used a learning rate of $0.01$ without weight decay, for all parameters, on all considered networks, and a hidden feature space of size $c=64$.

C.2 Learning PDE dynamics

In this experiment we used a $8$ layers mixed PDE-GCNM, starting with $\alpha=0.5$, such that it is balanced between a PDE-GCND and a PDE-GCNH. We report the hyper-parameters for this experiment in Tab. 11.

C.3 Semi-supervised node-classification

The hyper-parameters for this experiment are summarized in Tab. 12.

C.4 Fully-supervised node-classification

The hyper-parameters for this experiment are summarized in Tab. 13.

C.5 Ablation study

In this experiment we used the same hyper-parameters as reported in Tab. 12.