Data-driven reduced-order models via regularized operator inference for a single-injector combustion process

Introduction

The emerging field of scientific machine learning brings together the perspectives of physics-based modeling and data-driven learning. In the field of fluid dynamics, physics-based modeling and simulation have played a critical role in advancing scientific discovery and driving engineering innovation in domains as diverse as biomedical engineering (Yin et al. 2010; Nordsletten et al. 2011), geothermal modeling (O’Sullivan et al. 2001; Cui et al. 2011), and aerospace (Spalart and Venkatakrishnan 2016). These advances are based on decades of mathematical and algorithmic developments in computational fluid dynamics (CFD). Scientific machine learning builds upon these rigorous physics-based foundations while seeking to exploit the flexibility and expressive modeling capabilities of machine learning (Baker et al. 2019). This paper presents a scientific machine learning approach that blends data-driven learning with the theoretical foundations of physics-based model reduction. This creates the capability to learn predictive reduced-order models (ROMs) that provide approximate predictions of complex physical phenomena while exhibiting several orders of magnitude computational speedup over CFD.

Projection-based model reduction considers the class of problems for which the governing equations are known and for which we have a high-fidelity (e.g., CFD) model (Antoulas 2005; Benner et al. 2015). The goal is to derive a ROM that has lower complexity and yields accurate solutions with reduced computation time. Projection-based approaches define a low-dimensional manifold on which the dominant dynamics evolve. This manifold may be defined as a function of the operators of the high-fidelity model, as in interpolatory methods that employ a Krylov subspace (Bai 2002; Freund 2003), or it may be determined empirically from representative high-fidelity simulation data, as in the proper orthogonal decomposition (POD) (Lumley 1967; Sirovich 1987; Berkooz et al. 1993). The POD has been particularly successful in fluid dynamics, dating back to the early applications in unsteady flows and turbulence modeling (Sirovich 1987; Deane et al. 1991; Gatski and Glauser 1992), and in unsteady fluid-structure interaction (Dowell and Hall 2001).

Model reduction methods have advanced to include error estimation (Veroy et al. 2003; Veroy and Patera 2005; Grepl and Patera 2005; Rozza et al. 2008) and to address parametric and nonlinear problems (Barrault et al. 2004; Astrid et al. 2008; Chaturantabut and Sorensen 2010; Carlberg et al. 2013), yet the intrusive nature of the methods has limited their impact in practical applications. When legacy or commercial codes are used, as is often the case for CFD applications, it can be difficult or impossible to implement classical projection-based model reduction. Black-box surrogate modeling instead derives the ROM by fitting to simulation data; such methods include response surfaces and Gaussian process models, long used in engineering, as well as machine learning surrogate models. These methods are powerful and often yield good results, but since the approximations are based on generic data-fit representations, they are not equipped with the guarantees (e.g., stability guarantees, error estimators) that accompany projection-based ROMs. Nonlinear system identification techniques seek to illuminate the black box by discovering the underlying physics of a system from data (Brunton et al. 2016). However, when the governing dynamics are known and simulation data are available, reduced models may be directly tailored to the specific dynamics without access to the details of the large-scale CFD code.

This paper presents a non-intrusive alternative to black-box surrogate modeling. We use the Operator Inference method of Peherstorfer and Willcox 2016 to learn a ROM from simulation data; the structure of the ROM is defined by known governing equations combined with the theory of projection-based model reduction. Our approach may be termed ‘glass-box modeling’, which we define as the situation where the form of the targeted dynamics is known (here via the partial differential equations that define the problem of interest), but we do not have internal access to the CFD code that produces the simulation data. That is, we know what dynamics to expect, but we may only calibrate our models using outputs of the full-order CFD model. This glass-box setting is in contrast to black-box modeling approaches which do not exploit knowledge of the governing equations. We build on our prior work in Swischuk et al. 2020 by formally introducing regularization to the Operator Inference approach. Regularization is critical to avoid overfitting for problems with complex dynamics, as is the case for the combustion example considered here. A second contribution of this paper is a scalable implementation of the approach, which is available via an open-source implementation. Section 2 presents the methodology and regularization approach and describes the scalable implementation. Section 3 presents numerical results for a single-injector combustion problem and Section 4 concludes the paper.

Methodology

This section begins with an overview of the Operator Inference approach in Section 2.1. Section 2.2 augments Operator Inference with a new regularization formulation, posed as an optimization problem, and presents a complete algorithm for regularization selection and model learning. In Section 2.3, we discuss a scalable implementation of the algorithm, which can then be applied to CFD problems of high dimension.

We target problems governed by systems of nonlinear partial differential equations. Consider the governing equations of the system of interest written, after spatial discretization, in semi-discrete form

The non-intrusive Operator Inference (OpInf) approach proposed by Peherstorfer and Willcox 2016 parallels the intrusive projection-based ROM setting, but learns ROMs from simulation data without direct access to the FOM operators. Recognizing that the intrusive ROM has the same polynomial form as Eq. (1), OpInf uses a data-driven regression approach to derive a ROM of Eq. (1) as

OpInf solves a regression problem to find reduced operators that yield the ROM that best matches projected snapshot data in a minimum-residual sense. Mathematically, OpInf solves the least-squares problem

Eq. (3) can also be written in matrix form as

The OpInf approach permits us to compute the ROM operators $\widehat{\mathbf{c}}$, $\widehat{\mathbf{A}}$, $\widehat{\mathbf{H}}$, and $\widehat{\mathbf{B}}$ without explicit access to the original high-dimensional operators $\mathbf{c}$, $\mathbf{A}$, $\mathbf{H}$, and $\mathbf{B}$. This point is key since we apply variable transformations only to the snapshot data, not to the operators or the underlying model. Thus, even in a setting where deriving a classical intrusive ROM might be possible, the OpInf approach enables us to work with variables other than those used for the original high-fidelity discretization. In Section 3 we will see the importance of this for a reacting flow application. We also note that under some conditions, OpInf recovers the intrusive POD ROM (Peherstorfer and Willcox 2016; Peherstorfer 2020).

2 Regularization

The problem in Eq. (4) is generally overdetermined (i.e., $k>d(r,m)$), but is also noisy due to errors in the numerically estimated time derivatives $\mathbf{R}$, model mis-specification (e.g., if the system is not truly quadratic), and truncated POD modes that leave some system dynamics unresolved. The ROMs resulting from Eq. (4) can thus suffer from overfitting the operators to the data and therefore exhibit poor predictive performance over the time domain of interest $[t_{0},t_{f}]$.

To avoid overfitting, we introduce a Tikhonov regularization (Tikhonov and Arsenin 1977) to Eq. (4), which then becomes

which admit a unique solution since $\mathbf{D}^{\top}\mathbf{D}+\boldsymbol{\Gamma}^{\top}\boldsymbol{\Gamma}$ is symmetric positive definite.

An $L_{2}$ regularizer $\boldsymbol{\Gamma}=\lambda\mathbf{I}$, $\lambda>0$ and $\mathbf{I}$ the identity matrix, penalizes each entry of the inferred ROM operators $\widehat{\mathbf{c}}$, $\widehat{\mathbf{A}}$, $\widehat{\mathbf{H}}$, and $\widehat{\mathbf{B}}$, thereby driving the ROM toward the globally stable system $\frac{\textrm{d}}{\textrm{d}t}\widehat{\mathbf{q}}(t)=\mathbf{0}$. Since the entries of the quadratic operator $\widehat{\mathbf{H}}$ have a different scaling than entries of the other operators, we construct a diagonal regularizer $\boldsymbol{\Lambda}(\lambda_{1},\lambda_{2})$, with $\lambda_{1},\lambda_{2}>0$, such that the operator entries are penalized by $\lambda_{1}$, except for the entries of $\widehat{\mathbf{H}}$, which are penalized by $\lambda_{2}$. That is, with $\boldsymbol{\Gamma}=\boldsymbol{\Lambda}(\lambda_{1},\lambda_{2})$, Eq. (5) can be expressed as

The scalar hyperparameters $\lambda_{1}$ and $\lambda_{2}$, which balance the minimization between the data fit and the regularization, must be chosen with care. The ideal regularizer produces a ROM that minimizes some error metric over the full time domain $[t_{0},t_{f}]$; however, since data are only available for the smaller training domain $[t_{0},t_{k-1}]$, we choose $\lambda_{1}$ and $\lambda_{2}$ so that the resulting ROM minimizes error over $[t_{0},t_{k-1}]$ while maintaining a bound on the integrated POD coefficients over $[t_{0},t_{f}]$. That is, we require the state \widehat{\mathbf{q}}(t)=\left[\begin{array}[]{cccc}\hat{q}_{1}(t)&\hat{q}_{2}(t)&\cdots&\hat{q}_{r}(t)\end{array}\right]^{\top} produced by integrating Eq. (2) to satisfy $|\hat{q}_{i}(t)|\leq B$, $i=1,\ldots,r$ and $t\in[t_{0},t_{f}]$, for some $B>0$. This in turn ensures a bound on the magnitude of the entries of the high-dimensional state $\mathbf{q}(t)=\mathbf{V}\widehat{\mathbf{q}}(t)$:

where $V_{ij}$ is the $i$th element of the $j$th POD basis vector. Note that the bound $B$ may be chosen with the intent of imposing a particular bound on the $|q_{i}|$ since the sums $\sum_{j=1}^{r}|V_{ij}|$ can be precomputed. For example, our regularization strategy provides a computationally efficient way to impose the temperature limiters proposed by Huang et al. 2019.

Algorithm 1 details our regularized OpInf procedure, in which we choose $B$ as a multiple of the maximum absolute entry of the projected training data $\widehat{\mathbf{Q}}$. This particular strategy for selecting $\lambda_{1}$ and $\lambda_{2}$ could be replaced with a cross-validation or resampling grid search technique, but our experiments in this vein did not produce robust results. The training error $\|\widehat{\mathbf{Q}}-\widetilde{\mathbf{Q}}_{:,:k}\|$ in step 13 may compare $\widehat{\mathbf{Q}}$ and $\widetilde{\mathbf{Q}}_{:,:k}$ directly in the reduced space (e.g., with a matrix norm or an $L^{p}([t_{0},t_{k-1}])$ norm), or it may be replaced with a more targeted comparison of some quantity of interest.

The regularization approach of Algorithm 1 may be viewed as a stabilization method since it selects a ROM with reasonable behavior over a given time domain. It should be noted, however, that this method does not modify an existing ROM to achieve stability, different from other stabilization methods such as eigenvalue reassignment (Kalashnikova et al. 2014; Rezaian and Wei 2020). The optimization is driven by a penalization but has no built-in constraints, which is a major advantage in terms of the computational cost, but the resulting ROMs are not guaranteed to preserve properties such as energy conservation. Adding constraints to Operator Inference to target conservation, similar to the work in Carlberg et al. 2018, is a subject for possible future work.

3 Scalable Implementation

Finally, the minimization in step 14 is carried out with a derivative-free search method, which enables fewer total evaluations of the subroutine than a fine grid search. However, a coarse grid search is useful for identifying appropriate initial guesses for $\lambda_{1}$ and $\lambda_{2}$.

Results

This section applies regularized OpInf to a single-injector combustion problem, studied previously by Swischuk et al. 2020, on the two-dimensional computational domain shown in Figure 1. Section 3.1 describes the governing dynamics, a set of high-fidelity data obtained from a CFD code, and the variable transformations used to produce training data for learning reduced models with Algorithm 1. The resulting OpInf ROM performance is analyzed in Section 3.2 and compared to a state-of-the-art intrusive model reduction method in Section 3.3.

The combustion dynamics for this problem are governed by conservation laws

At the downstream end of the combustor, we impose a non-reflecting boundary condition while maintaining the chamber pressure via

where $p_{\textrm{back,ref}}=10^{6}$ Pa and $f=5{,}000$ Hz. The top and bottom wall boundary conditions are no-slip conditions, and for the upstream boundary we impose a constant mass flow at the inlets.

To generate high-fidelity training data, we use the finite-volume based General Equation and Mesh Solver (GEMS) (Harvazinski et al. 2015) to solve for the variables \left[\begin{array}[]{cccccccc}p&v_{x}&v_{y}&T&Y_{1}&Y_{2}&Y_{3}&Y_{4}\end{array}\right] over $n_{x}=38{,}523$ cells, resulting in snapshots with $8n_{x}=308{,}184$ entries each. The snapshots are computed for 60,000 time steps beyond the initial condition with a temporal discretization of $\delta t=10^{-7}$ s, from $t_{0}=0.015$ s to $t_{f}=0.021$ s. The computational cost of computing this dataset is approximately 1,200 CPU hours on two computing nodes, each of which contains two Haswell CPUs at $2.60$ GHz and $20$ cores per node.

Scaling is an essential aspect of successful model reduction and is particularly critical for this problem due to the wide range of scales across variables. After transforming the GEMS snapshot data to the learning variables $\vec{q}$, the species molar concentrations are scaled to $$, and all other variables are scaled to$$. This scaling ensures that null velocities and null molar concentrations are preserved. For example, some upstream regions of the injector have zero methane concentration at all times. By construction, the POD basis vectors and thus the ROM predictions will preserve those zero concentration values.

2 Sensitivity to Training Data

Figure 3 plots the GEMS and OpInf ROM results for pressure and $x$-velocity predictions over time at two of the monitor locations in Figure 1. See https://github.com/Willcox-Research-Group/ROM-OpInf-Combustion-2D for additional results. While it can be misleading to assess accuracy based on predictions at a single spatial point, these plots reveal several representative insights. First, each OpInf ROM faithfully reconstructs the training data but has some discrepancies in the prediction regime. Second, the pressure and $x$-velocity frequencies are well captured throughout the time domain, but the amplitudes are sometimes less accurate in the prediction regime. The effects of the 5,000 Hz downstream pressure forcing are clearly visible in the pressure. Third, we see the importance of the training data—as the amount of training data increases, the ROM predictions change significantly and generally (but not always) improve. This is yet another indication of the complexity of the dynamics we are aiming to approximate.

respectively. Figure 4 plots these errors against time for pressure and temperature using a POD basis with $r=43$ vectors computed from the first $k=20{,}000$ training snapshots. While both error measures for pressure remain low throughout the full time interval, both the projection errors and the ROM prediction errors for temperature increase significantly at the end of the training regime. The temperature profile is influenced by both the advective flow dynamics and the local chemical reactions, which in combination lead to a highly nonlinear and multiscale behavior that is difficult to represent with the POD basis after the training period. However, while the ROM struggles to accurately predict the detailed temperature variations pointwise, it does adequately predict the general trends of temperature evolution beyond the training horizon. Figure 4 shows the time-averaged temperature profiles for the GEMS data and for an OpInf ROM, suggesting that the ROM captures the time-averaged behavior of the temperature dynamics.

Figure 5 plots CH4 and CO2 concentrations integrated over the spatial domain. These measures give a more global sense of the ROM predictive accuracy and the predicted chemical reaction rate. In each case, the ROMs are able to accurately re-predict the training data and capture much of the overall system behavior in the prediction phase, with slightly more training error as the number of snapshots increases.

3 Comparison to POD-DEIM

We now compare regularized OpInf to a state-of-the-art nonlinear model reduction method that uses a least-squares Petrov-Galerkin POD projection coupled with the discrete empirical interpolation method (DEIM) (Chaturantabut and Sorensen 2010), as implemented for the same combustion problem in Huang et al. 2019; Huang et al. 2018. This POD-DEIM method is intrusive—it requires nonlinear residual evaluations of the GEMS code at sparse discrete interpolation points. This also increases the computational cost of solving the POD-DEIM ROM in comparison to the OpInf ROM: integrating a POD-DEIM ROM with $r=70$ for 6,000 time steps of size $\delta t=10^{-6}$ s takes approximately $30$ minutes on two nodes, each with two Haswell CPUs processors at $2.60$ GHz and $20$ cores per node; for OpInf, using Python $3.6.9$ and a single CPU on an AMD EPYC 7,702 64-core processor at $3.3$ GHz with $2.1$ TB RAM, we solve Eq. (5) with $k=20{,}000$ training snapshots and $r=43$ POD modes in approximately $0.6$ s and integrate the resulting OpInf ROM for 60,000 time steps of size $\delta t=10^{-7}$ s in approximately $0.4$ s. While these measurements are made on different hardware, and though the execution time for POD-DEIM can be improved with optimal load balancing, the difference in execution times (30 minutes versus 1 second) is representative and illustrates one of the advantages over POD-DEIM of the polynomial form employed in the OpInf approach.

Figure 6 compares select GEMS outputs to POD-DEIM and OpInf ROM outputs, with each ROM trained on $k=20{,}000$ training snapshots. As before, the OpInf ROM dimension $r=43$ is chosen such that $\mathcal{E}_{r}>0.985$; the POD-DEIM ROM, which uses an entirely different basis than the OpInf approach, requires $r=70$ vectors to achieve the same level of cumulative energy. Both approaches maintain appropriate pressure oscillation frequencies, and while neither model accurately predicts the global species concentration dynamics after the training period, the OpInf ROM reconstructs the training data more faithfully than the POD-DEIM ROM. Huang et al. 2019 show similar results for the same POD-DEIM model with 1 ms of training and 1 ms of prediction; here we are using 2 ms of training and 4 ms of prediction. Note from Figures 3 and 5 that the OpInf ROMs achieve excellent prediction results for the 1 ms period following the training.

Figures 7 and 8 show, respectively, full-domain results for the temperature and molar concentration of CH4. The figures show the solution at time instants within the training regime, at the end of the training regime, and into the prediction regime. As with the point traces shown earlier, we see that the ROMs have impressive accuracy over the training region, but lose accuracy as they attempt to predict dynamics beyond the training horizon. However, many of the coherent features are reasonably predicted, especially the recirculation zone dynamics near the dump plane ($x=0$ in Figure 1) shown in the temperature fields. Significantly, both ROMs maintain appropriate temperature ranges throughout the prediction phase. The POD-DEIM ROM explicitly enforces such limits by reconstructing the full solution at each time step, constraining the temperature to a desired range, and projecting the result back to the reduced space (see Section IV.G of Huang et al. 2019); in contrast, the OpInf ROM selects a regularization that results in bounded behavior due to the criteria $|\hat{q}_{i}(t)|\leq B$ (see Eq. (7)). In other words, POD-DEIM limits the temperature in the online phase, while OpInf builds a similar constraint into the offline phase.

Conclusions

The presented scientific machine learning approach is broadly applicable to problems where the governing equations are known but access to the high-fidelity simulation code is limited. The approach is computationally as accessible as black-box surrogate modeling while achieving the accuracy of intrusive projection-based model reduction. While the conclusions drawn from the numerical studies apply to the single-injector combustion example, they are relevant and likely apply to other problems. First, the quality and quantity of the training data are critical to the success of the method. Second, regularization is essential to avoid overfitting. Third, achieving a low error over the training regime is not necessarily indicative of a reduced model with good predictive capability. This emphasizes the importance of the training data. Fourth, physical quantities that exhibit large-scale coherent structures (e.g., pressure) are more accurately predicted by a reduced-order model than quantities that exhibit multiscale behavior (e.g., temperature, species concentrations). Fifth, a significant advantage of the data-driven learning aspects of the approach is that the reduced model may be derived in any variables. This includes the possibility to include redundancy in the learning variables (e.g., to include both pressure and temperature). Overall, this paper illustrates the power and effectiveness of learning from data through the lens of physics-based models as a physics-grounded alternative to black-box machine learning.

Acknowledgements

This work has been supported in part by the Air Force Center of Excellence on Multi-Fidelity Modeling of Rocket Combustor Dynamics under award FA9550-17-1-0195, and the US Department of Energy AEOLUS MMICC center under award DE-SC0019303.

References