Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU

Introduction

The signature transform, sometimes referred to as the path signature or simply signature, is a central object in rough path theory (Lyons 1998; Lyons 2014). It is a transformation on differentiable paths And may be extended to paths of bounded variation, or merely finite $p$-variation (Lyons et al. 2004)., and may be thought of as loosely analogous to the Fourier transform. However whilst the Fourier transform extracts information about frequency, treats each channel separately, and is linear, the signature transform exacts information about order and area, explicitly considers combinations of channels, and is in a precise sense ‘universally nonlinear’ (Bonnier et al. 2019, Proposition A.6).

The logsignature transform (Liao et al. 2019) is a related transform, that we will also consider. In both cases, by treating sequences of data as continuous paths, then the (log)signature transform may be applied for use in problems with sequential structure, such as time series. Indeed there is a significant body of work using the (log)signature transform in machine learning, with examples ranging from handwriting identification to sepsis prediction, see for example Morrill et al. 2019; Fermanian 2019; Király & Oberhauser 2019; Toth & Oberhauser 2020; Morrill et al. 2020b.

Earlier work often used the signature and logsignature transforms as a feature transformation. See Levin et al. 2013; Chevyrev & Kormilitzin 2016; Yang et al. 2016a; Yang et al. 2016b; Kormilitzin et al. 2016; Li et al. 2017; Perez Arribas et al. 2018 for a range of examples. In this context, when training a model on top, it is sufficent to simply preprocess the entire dataset with the signature or logsignature transform, and then save the result.

However, recent work has focused on embedding the signature and logsignature transforms within neural networks. Recent work includes Bonnier et al. 2019; Liao et al. 2019; Moor et al. 2020; Morrill et al. 2020a; Kidger et al. 2020 among others. In this context, the signature and logsignature transforms are evaluated many times throughout a training procedure, and as such efficient and differentiable implementations are crucial. Previous libraries (Lyons 2017; Reizenstein & Graham 2018) have been CPU-only and single-threaded, and quickly become the major source of slowdown when training and evaluating these networks.

We introduce Signatory, a CPU- and GPU-capable library for calculating and performing functionality related to the signature and logsignature transforms. To our knowledge it is the first GPU-capable library for these operations. The focus is on machine learning applications.

Signatory is significantly faster than previous libraries (whether run on the CPU or the GPU), due to a combination of parallelism and novel algorithmic improvements. In particular the latter includes both uniform and asymptotic rate improvements over previous algorithms. Additionally, Signatory provides functionality not available in previous libraries, such as precomputation strategies for efficient querying of the (log)signature transform over arbitrary overlapping intervals.

The library integrates with the open source PyTorch ecosystem and runs on Linux or Windows. Documentation, examples, benchmarks and tests form a part of the project.

Much of the code is written in C++ primitives and the CPU implementation utilises OpenMP. The backward operations are handwritten for both speed and memory efficiency, and do not rely on the autodifferentiation provided by PyTorch.

The source code is located at https://github.com/patrick-kidger/signatory, documentation and examples are available at https://signatory.readthedocs.io, and the project may be installed directly via pip.

This paper is not a guide to using Signatory—for that we refer to the documentation. This is meant as a technical exposition of its innovations.

2 Applications

Signatory has already seen a rapid uptake amongst the signature community. Recent work using Signatory include Morrill et al. 2020b; Perez Arribas et al. 2020 who involve signatures in neural differential equations, or Moor et al. 2020; Min & Ichiba 2020 who study deep signature models (Bonnier et al. 2019). Meanwhile Ni et al. 2020 apply Signatory to hybridise signatures with GANs, and Morrill et al. 2020a create a generalised framework for the “signature method”. As a final example, Signatory is now itself a dependency for other libraries (Kidger 2020).

Background

We begin with some exposition on theory of the signature and logsignature transforms. We begin with definitions and offer intuition afterwards. Also see Reizenstein & Graham 2018 for an introduction focusing on computational concerns, and Lyons et al. 2004 and Hodgkinson et al. 2020 for pedagogical introductions to the motivating theory of rough paths.

Most texts define the signature transform using the notation of stochastic calculus. Here, we sacrifice some generality (that is not needed in this context) in favour of more widely-used notation. Additionally, many texts also include a $k=0$ term, which is defined to equal one. We omit this as it does not carry any information, and is therefore irrelevant to the task of machine learning.

The signature transform may naturally be extended to sequences of data.

The space of sequences of data over a set $V$ is

An interval of $(x_{1},\ldots,x_{L})\in\mathcal{S}\left(V\right)$ is $(x_{i},\ldots,x_{j})\in\mathcal{S}\left(V\right)$ for some $1\leq i<j\leq L$.

Note that the choice of $\tfrac{i-1}{L-1}$ is unimportant; any $L$ points in $$ would suffice, and in fact the definition is invariant to this choice (Bonnier et al. 2019, Definition A.10).

2 The grouplike structure

Furthermore the signature of a sequence of length two may be computed explicitly from the definition. Letting

With Chen’s identity, this implies that the signature transform may be computed by evaluating

3 The logsignature, inverted signature, and inverted logsignature

The group inverse we denote $\hbox{}^{-1}$. Additionally a notion of logarithm may be defined (Liao et al. 2019), where

This then defines the notions of inverted signature transform, logsignature transform and inverted logsignature transform as

respectively. We emphasise that the inverted signature or logsignature transforms are not the inverse maps of the signature or the logsignature transforms.

4 Signatures in machine learning

All of these transforms are in fact differentiable with respect to $\mathbf{x}$, and so may be backpropagated through. These transforms may thus be thought of as differentiable operations between tensors, in the way usually performed by machine learning frameworks.

5 Intuition

The (inverted) signature and logsignature transforms all have roughly the same intuition as one another. (They all represent the same information, just in slightly different ways.) Given a sequence of data $(x_{1},\ldots,x_{L})$, then these transforms may be used as binning functions, feature extractors, or nonlinearities, to give summary statistics over the data.

Code example

Signatory is designed to be Pythonic, and offer operations working just like any other PyTorch operation, outputting PyTorch tensors. A brief example is:

Algorithmic improvements

We present several noveral algorithmic improvements for computing signatures and logsignatures.

Recall from equation (3) that the signature may be computed by evaluating several exponentials and several $\boxtimes$. We begin by finding that it is beneficial to compute

The bulk of a signature computation may then be sped up by writing it in terms of this fused operation. See equation (3): a single exponential is required at the start, followed by a reduction with respect to this fused multiply-exponentiate. This gives substantial real-world speedups; see the benchmarks of Section 6.

at which point the $k$-th term may be computed by a scheme in the style of Horner’s method:

See Appendix A.1 for the the mathematics, including proofs of both the asymptotic complexity and the uniformly fewer multiplications.

2 Improved precomputation strategies

with a similar relation for the inverted signature. Then, at inference time use the group-like structure

followed by a $\log$ if it is a logsignature that is desired. As a single operation this is $\mathcal{O}(1)$ in $L$.

We do remark that this should be used with caution, and may suffer from numerical stability issues when used for large $i,j$.

3 More efficient logsignature basis

The logsignature transform of a path has multiple possible representations, corresponding to different possible bases of the ambient space, which is typically interpreted as a free Lie algebra (Reutenauer 1993). The Lyndon basis is a typical choice (Reizenstein & Graham 2018).

We show that there exists a more computationally efficient basis. It is mathematically unusual, as it is not constructed as a Hall basis. But if doing deep learning, then the choice of basis is (mostly) unimportant if the next operation is a learnt linear transformation.

The Lyndon basis uses Lyndon brackets as its basis elements. Meanwhile our new basis uses basis elements that, when written as a sum of Lyndon brackets and expanded as a sum of words, have precisely one word as a Lyndon word. This means that the coefficient of this basis element can be found cheaply, by extracting the coeffient of that Lyndon word from the tensor algebra representation of the logsignature. See Appendix A.2 for the full exposition.

New features

Signatory provides several features not available in previous libraries.

There are two main levels of parallelism. First is naïve parallelism over the batch dimension. Second, we observe that equation (3) takes the form of a noncommutative reduction with respect to the fused multiply-exponentiate. The operation $\boxtimes$ is associative, and so this may be parallelised in the usual way for reductions, by splitting the computation up into chunks.

Parallelism on the CPU is implemented with OpenMP. For speed the necessary operations are written in terms of C++ primitives, and then bound into PyTorch.

2 GPU Support

An important feature is GPU support, which is done via the functionality available through LibTorch. It was a deliberate choice not to write CUDA code; this is due to technical reasons for distributing the library in a widely-compatible manner. See Appendix B.

There are again two levels of parallelism, as described in the previous section.

When the (log)signature transform is part of a deep learning model trained on the GPU, then GPU support offers speedups not just from the use of a GPU over the CPU, but also obviates the need for copying the data to and from the GPU.

3 Backpropagation

Crucial for any library used in deep learning is to be able to backpropagate through the provided operations. Signatory provides full support for backpropagation through every provided operation. There has previously only been limited support for backpropagation through a handful of simple operations, via the iisignature library (Reizenstein & Graham 2018).

The backpropagation computations are handwritten, rather than being generated autodifferentiably. This improves the speed of the computation by using C++ primitives rather than high-level tensors, and furthermore allows for improved memory efficiency, by exploiting a reversibility property of the signature (Reizenstein 2019, Section 4.9.3). We discuss backpropagation in more detail in Appendix C.

4 Inverted signatures and logsignatures

Signatory provides the capability to compute inverted signatures and logsignatures, via the optional inverse argument to the signature and logsignature functions. This is primarily a convenience as

5 Exploiting the grouplike structure

It is often desirable to compute (inverted) (log)signatures over multiple intervals of the same sequence of data. These calculations may jointly be accomplished more efficiently than by evaluating the signature transform over every interval separately. In some cases, if the original data has been discarded and only its signature is now known, exploiting this structure is the only way to perform the computation.

Here we detail several notable cases, and how Signatory supports them. In all cases the aim is to provide a flexible set of tools that may be used together, so that wherever possible unecessary recomputation may be elided. Their use is also discussed in the documentation, including examples.

Recall equation (2). If the two signatures on the right hand side of the equation are already known, then the signature of the overall sequence of data may be computed using only a single $\boxtimes$ operation, without re-iterating over the data.

This operation is provided for by the multi_signature_combine and signature_combine functions.

This is handled by the optional stream argument to the signature function and to the logsignature function.

This could done by computing the signatures over $(x_{L+1},\ldots,x_{L+M})$, and combining them together as above. However, if these signatures (over $(x_{L+1},\ldots,x_{L+M})$) are not themselves of interest, then this approach may be improved upon, as this only exploits the grouplike structure, but not the fused multiply-exponentiate described in Section 4.1.

Computing them in this more efficient way may be handled via the basepoint and initial arguments to the signature function, and via the update method of the Path class.

Benchmark performance

We are aware of two existing software libraries providing similar functionality, esig (Lyons 2017) and iisignature (Reizenstein & Graham 2018). We ran a series of benchmarks against the latest versions of both of these libraries, namely esig 0.6.31 and iisignature 0.24. The computer used was equipped with a Xeon E5-2960 v4 and a Quadro GP100, and was running Ubuntu 18.04 and Python 3.7.

For esig and iisignature we report run time on the CPU, whilst for Signatory we report run time on the GPU, CPU with parallelism, and CPU without parallelism. As it is in principle possible to parallelise these alternative libraries using Python’s multiprocessing module Subject to nontrivial overhead., the most important comparisons are to Signatory on the CPU without parallelism (representing like-for-like computational resources), and on the GPU (representing the best possible performance).

We begin with a benchmark for the forward operation through the signature transform. We consider a batch of 32 sequences, of length 128. We then investigate the scaling as we vary either the number of channels (over 2–7) in the input sequences, or the depth (over 2–9) of the signature transform. For varying channels, the depth was fixed at 7. For varying depths, the channels was fixed at 4.

Every test case is repeated 50 times and the fastest time taken. See Figure 2. Note the logarithmic scale.

We observe that iisignature is Signatory’s strongest competitor in all cases. Signatory and iisignature are comparable for the very smallest of computations. As the computation increases in size, then the CPU implementations of Signatory immediately overtake iisignature, followed by the GPU implementation.

For larger computations, Signatory can be orders of magnitude faster. For example, to compute the signature transform with depth and number of channels both equal to $7$, then iisignature takes 20.9 seconds to perform the computation. In contrast, running on the CPU without parallelism, Signatory takes only 3.8 seconds, which represents a 5.5$\times$ speedup. We emphasise that the same computational resources (including lack of parallelism) were used for both. To see the benefits of a GPU implementation over a CPU implementation—the primary motivation for Signatory’s existence—then we observe that Signatory takes only 0.16 seconds to compute this same operation. Compared to the best previous alternative in iisignature, this represents a 132$\times$ speedup.

Next, we consider the backward operation through the signature transform. We vary over multiple inputs as before. See Figure 2. We again observe the same behaviour. iisignature is Signatory’s strongest competitor, but is still orders of magnitude slower on anything but the very smallest of problems. For example, to backpropagate through the signature transform, with depth and number of channels both equal to $7$, then Signatory on the CPU without parallelism takes 13.7 seconds. Meanwhile, iisignature takes over 2 minutes – 128 seconds – to perform this computation on like-for-like computational resources. Running Signatory on the GPU takes a fraction of a second, specifically 0.772 seconds. These represent speedups of 9.4$\times$ and 166$\times$ respectively.

For further speed benchmarks, a discussion on memory usage benchmarks, the precise numerical values of the graphs presented here, and code to reproduce these benchmarks, see Appendix D. We observe the same consistent improvements on these additional benchmarks.

2 Deep learning example

To emphasise the benefit of Signatory to deep learning applications, we consider training a deep signature model (Bonnier et al. 2019) on a toy dataset of geometric Brownian motion samples. The samples have one of two different volatilities, and the task is to perform binary classification.

The model sweeps a small feedforward network over the input sequence (to produce a sequence of hidden states), applies the signature transform, and then maps to a binary prediction via a final learnt linear map. This model has learnt parameters prior to the signature transform, and so in particular backpropagation through the signature transform is necessary.

The signature transform is computed either using Signatory, or using iisignature. We train the model on the GPU and plot training loss against wall-clock time.

Both models train successfully, but the model using Signatory trains 210 times faster than the one using iisignature. This makes clear how signatures have previously represented the largest computational bottleneck. The improvement of 210$\times$ is even larger than the improvements obtained in the previous section. We attribute this to the fact that iisignature necessarily has the additional overhead copying data from the GPU to the CPU and back again.

Conclusion

We have introduced Signatory, a library for performing functionality related to the signature and logsignature transforms, with a particular focus on applications to machine learning. Notable contributions are the speed of its operation, its GPU support, differentiability of every provided operation, and its novel algorithmic innovations.

This work was supported by the Engineering and Physical Sciences Research Council [EP/L015811/1].

References

Appendix A Further details of algorithmic improvements

The conventional way to compute a signature is to iterate through the computation described by equation (3): for each new increment, take its exponential, and $\boxtimes$ it on to what has already been computed; repeat.

Our proposed alternate way is to fuse the exponential and $\boxtimes$ into a single operation, and then iteratively perform this fused operation.

We now count the number of multiplications required to compute

We will establish that the fused operation uses fewer multiplications for all possible $d\geq 1$ and $N\geq 1$. We will then demonstrate that it is in fact of a lower asymptotic complexity.

Note that every tensor in the exponential is symmetric, and so in principle requires less work to compute than its number of elements would suggest. For the purposes of this analysis, to give the benefit of the doubt to a competing method, we shall assume that this is done (although taking advantage of this in practice is actually quite hard (Reizenstein & Graham 2018, Section 2)). This takes

scalar multiplications, using the formula for unordered sampling with replacement (Reizenstein & Graham 2018, Section 2), under the assumption that each division by a scalar costs the same as a multiplication (which can be accomplished by precomputing their reciprocals and then multiplying by them).

Next, we need to count the number of multiplications to perform a single $\boxtimes$.

To the authors’ knowledge there has been no formal analysis of a lower bound on the computational complexity of $\boxtimes$, and there is no better way to compute it than naïvely following this definition.

Thus the overall cost of the conventional way involves

A.1.2 The fused operation

where the $k$-th term may be computed by a scheme in the style of Horner’s method:

As before, we assume that the reciprocals $\frac{1}{2},\ldots,\frac{1}{N}$ have been precomputed, so that each division costs the same as a multiplication.

Then we begin by computing $z/2,\ldots,z/N$, which takes $d(N-1)$ multiplications.

Computing the $k$-th term as in equation (A.1.2) then involves $d^{2}+d^{3}+\cdots+d^{k}$ multiplications. This is because, working from innermost bracket to outermost, the first $\otimes$ produces a $d\times d$ matrix as the outer product of two size $d$ vectors, and may thus be computed with $d^{2}$ multiplications; the second $\otimes$ produces a $d\times d\times d$ tensor from a $d\times d$ matrix and a size $d$ vector, and may thus be computed with $d^{3}$ multiplications; and so on.

Thus the overall cost of a fused multiply-exponentiate is

A.1.3 Comparison

We begin by establishing the uniform bound $\mathcal{F}(d,N)\leq\mathcal{C}(d,N)$ for all $d\geq 1$ and $N\geq 1$.

Now suppose $d\geq 2$ and $N\geq 3$. Then

Thus we see that it suffices to show that

At this point $d=2$, $N=3$ must be handled as a special case, and may be verified by direct evaluation of equation (14). So now assume $d\geq 2$, $N\geq 3$, and that $d=2$, $N=3$ does not occur jointly. Then we see that equation (14) is implied by

The second condition is trivially true. The first condition rearranges to $N/(N-2)\leq d$, which is now true for $d\geq 2$, $N\geq 3$ with $d=2$, $N=3$ not jointly true.

This establishes the uniform bound $\mathcal{F}(d,N)\leq\mathcal{C}(d,N)$.

Checking the asymptotic complexity is much more straightforward. Consulting equations (12) and (13) shows that $\mathcal{F}(d,n)=\mathcal{O}(d^{N})$ whilst $\mathcal{C}(d,N)=\Omega(Nd^{N})$. And in fact as $\binom{d+k-1}{k}\leq d^{k}$ then equation (9) demonstrates that $\mathcal{C}(d,N)=\mathcal{O}(Nd^{N})$.

A.2 Logsignature bases

We move on to describing our new more efficient basis for the logsignature.

Let $\mathcal{A}=\{a_{1},\ldots,a_{d}\}$ be a set of $d$ letters. Let $\mathcal{A}^{+N}$ be the set of all words in these letters, of length between $1$ and $N$ inclusive. For example $a_{1}a_{4}\in\mathcal{A}^{+N}$ is a word of length two.

Impose the order $a_{1}<a_{2}<\cdots<a_{d}$ on $\mathcal{A}$, and extend it to the lexicographic order on words in $\mathcal{A}^{+N}$ of the same length as each other, so that for example $a_{1}a_{2}<a_{1}a_{3}<a_{2}a_{1}$. Then a Lyndon word (Lalonde & Ram 1995) is a word which comes earlier in lexicographic order than any of its rotations, where rotation corresponds to moving some number of letters from the start of the word to the end of the word. For example $a_{2}a_{2}a_{3}a_{4}$ is a Lyndon word, as it is lexicographically earlier than $a_{2}a_{3}a_{4}a_{2}$, $a_{3}a_{4}a_{2}a_{2}$ and $a_{4}a_{2}a_{2}a_{3}$. Meanwhile $a_{2}a_{2}$ is not a Lyndon word, as it is not lexicographically earlier than $a_{2}a_{2}$ (which is a rotation). Denote by $\mathcal{L}\left(\mathcal{A}^{+N}\right)$ the set of all Lyndon words of length between $1$ and $N$.

Given any Lyndon word $w_{1}\cdots w_{n}$ with $n\geq 2$ and $w_{i}\in\mathcal{A}$, we may consider its longest Lyndon suffix; that is, the smallest $j>1$ for which $w_{j}\cdots w_{n}$ is a Lyndon word. (It is guaranteed to exist as $w_{n}$ alone is a Lyndon word.) It is a fact (Lalonde & Ram 1995) that $w_{1}\cdots w_{j-1}$ is then also a Lyndon word. Given a Lyndon word $w$, we denote by $w^{b}$ its longest Lyndon suffix, and by $w^{a}$ the corresponding prefix.

where $wz$ denotes concatenation of words, distributed over the addition, as $w$ and $z$ belong to a span and thus may be linear combinations of words. For example $w=2a_{1}a_{2}+a_{1}$ and $z=a_{1}+a_{3}$ gives $wz=2a_{1}a_{2}a_{1}+2a_{1}a_{2}a_{3}+a_{1}a_{1}+a_{1}a_{3}$.

by $\phi(w)=w$ if $w$ is a word of only a single letter, and by

is a linear map between finite dimensional real vector spaces, from a lower dimensional space to a higher dimensional space.

is a linear map between finite dimensional real vector spaces, from a higher dimensional space to a lower dimensional space.

A.2.2 A basis for signatures

Recall that the signature transform maps between spaces as follows.

A.2.3 Bases for logsignatures

However, it turns out that this is unnecessarily expensive. In deep learning, it is typical to apply a learnt linear transformation after a nonlinearity - in which case we largely do not care in what basis we represent the logsignature, and it turns out that we can find a more efficient one.

But now by simply applying $\psi$ to both sides:

Appendix B LibTorch vs CUDA

LibTorch is the C++ equivalent to the PyTorch library. GPU support in Signatory was provided by using the operations provided by LibTorch.

It was a deliberate choice not to write custom CUDA kernels. The reason for this is as follows. We have to make a choice between distributing source code and distributing precompiled binaries. If we distribute source code, then we rely on users being able to compile CUDA, which is far from a guarantee.

Meanwhile, distributing precompiled binaries is unfortunately not feasible on Linux. C/C++ extensions for Python are typically compiled for Linux using the ‘manylinux’ specification, and indeed PyPI will only host binaries claiming to be compiled according to this specification. Unfortunately, based on our inspection of its build scripts, PyTorch appears not to conform to this specification. It instead compiles against a later version of Centos than is supported by manylinux, and then subsequently modifies things so as to seem compatible with the manylinux specification.

Unpicking precisely how PyTorch does this so that we might duplicate the necessary functionality (as we must necessarily remain compatible with PyTorch as well) was judged a finickity task full of hard-to-test edge cases, that is an implementation detail of PyTorch that should not be relied upon, and that may not remain stable across future versions.

Appendix C Backpropagation

Backpropagation is calculated in the usual way, mathematically speaking, by treating the signature and logsignature transforms as a composition of differentiable primitives, as discussed in Section 2.2.

The backpropagation computations are handwritten, and are not generated autodifferentiably. This improves the speed of the computation by using C++ primitives, rather than high-level tensors.

Moreover, it allows us to exploit a reversibility property of the signature (Reizenstein 2019). When backpropagating through any forward operation, then typically the the forward results are stored in memory, as these are used in the backward pass.

However, recall the grouplike structure of the signature; in particular this means that

We remark in Section 2.5 that the signature may be interpreted as the solution to a differential equation. This recomputation procedure actually corresponds to the adjoint method for backpropagating through a differential equation, as popularised in machine learning via Chen et al. 2018.

Importantly however, this does not face reconstruction errors in the same way as neural differential equations (Gholami et al. 2019). Because the driving path $f$ is taken to be piecewise affine in Definition 4, then the differential equation defining the signature may be solved exactly, without numerical approximations.

Equation (18) uses the same basic operations as the forward operation, and can be computed using the same subroutines, including the fused multiply-exponentiate.

C.2 Speed versus memory trade-offs

The reversibility procedure just described introduces the additional cost of recomputing the path (rather than just holding it in memory). In principle this need not be performed by holding partial results in memory.

For simplicity we do not offer this an alternative with Signatory. Signature-based techniques are often applied to long or high-frequency data (Lyons et al. 2014; Morrill et al. 2020b), for which the large size of multiple partially computed signatures can easily become a memory issue. Nonetheless this represents an opportunity for further work.

C.3 Parallelism

The use of parallelism in the gradient computation depends upon whether to use reversibility as discussed.

Consider first the case in which reversibility is not used, and all intermediate results are held in memory. As discussed in Section 5.1, the forward operation may be computed in parallel as a reduction. The computation graph (within the signature computation) then looks like a balanced tree, and so the backward operation through this computation graph may be performed in parallel as well.

As Signatory uses reversibility then backpropagation is not performed in parallel.

This represents an opportunity for further work, but practically speaking we expect that its impact is only moderate. Backpropagation is typically performed as part of a training procedure over batches of data; thus the available parallelism may already saturated by parallelism over the batch, and by the intrinsic parallelism available within each primitive operation.

Appendix D Further benchmarks

The benchmarks may be reproduced with the following code on a Linux system. First we install the necessary packages.

pip install numpy==1.18.0 matplotlib==3.0.3 torch==1.5.0

pip install iisignature==0.24 esig==0.6.31 signatory==1.2.1.1.5.0

git clone https://github.com/patrick-kidger/signatory.git

Note that numpy must be installed before iisignature, and PyTorch must be installed before Signatory. The unusually long version number for Signatory is necessary to specify both the version of Signatory, and the version of PyTorch that it is for. The git clone is necessary as the benchmarking code is not distributed via pip.

for further details on how to run any particular benchmark. For example,

python command.py benchmark -m time -f sigf -t channels -o graph

D.2 Memory benchmarks

Our benchmark scripts offer some limited ability to benchmark memory consumption, via the -m memory flag to the benchmark scripts.

The usual approach to such benchmarking, using valgrind’s massif, necessarily includes measuring the set-up code. As this includes loading both the Python interpreter and PyTorch, measuring the memory usage of our code becomes tricky.

As such we use an alternate method, in which the memory usage is sampled at intervals, using the Python package memory_profiler, which may be installed via pip install memory_profiler. This in turn has the limitation that it may miss a peak in memory usage; for small calculations it may miss the entire calculation. Furthermore, the values reported are inconsistent with those reported in Reizenstein & Graham 2018.

Nonetheless, when compared against iisignature using memory_profiler, on larger computations where peaks are less likely to go unobserved, then Signatory typically uses at an order of magnitude less memory. However due to the limitations above, we have chosen not report quantitative memory benchmarks here.

D.3 Signature transform benchmarks

The precise values of the points of Figures 2 and 2 are shown in Tables 1–4.

For convenience, the ratio between the speed of Signatory and the speed of iisignature is also shown.

D.4 Logsignature transform benchmarks

See Figure 4 for the graphs of the benchmarks for the logsignature transform.

The computer and runtime environment used was as described in Section 6.

We observe similar behaviour to the benchmarks for the signature transform. iisignature is slightly faster for some very small computations, but that as problem size increases, Signatory swiftly overtakes iisignature, and is orders of magnitude faster for larger computations.

The precise values of the points on these graphs are shown in Tables 5–8. Times are given in seconds. Also shown is the ratio between the speed of Signatory and the speed of iisignature. A dash indicates that esig does not support that operation.

D.5 Single-element-batch benchmarks

The benchmarks so far considered were for a batch of samples (of size 32). Whilst this is of particular relevance for training, it is sometimes less relevant for inference. We now repeat all the previous benchmarks (forward and backward through both signature and logsignature, varying both depth and channels), except that the batch dimension is reduced to size 1. See Figures 5 and 6. Numerical values are presented in Tables 9–16.

Here we see on very small problems that iisignature now outperforms Signatory by about a millisecond, but that once again Signatory overtakes iisignature on reasonably-sized problems, and is still orders of magnitude faster on larger problems.

We do not regard the performance on very small single-element problems as a drawback of Signatory. If performing very few very small calculations, then the difference of a millisecond is irrelevant. If performing very many very small calculations, then these can typically be batched together.