Controllable Pareto Multi-Task Learning

Introduction

Multi-task learning (MTL) is important for many real-world applications, such as in computer vision (Kokkinos, 2017), natural language processing (Subramanian et al., 2018), and reinforcement learning (Van Moffaert & Nowé, 2014). In these problems, multiple tasks are needed to be learned at the same time. An MTL system usually builds a single model to learn several related tasks together, in which the positive knowledge transfer could improve the performance for each task. In addition, using one model to conduct multiple tasks is also good for saving storage costs and reducing the inference time, which could be crucial for many applications (Standley et al., 2020).

However, with a fixed learning capacity, different tasks could be conflicted with each other, and can not be optimized simultaneously (Zamir et al., 2018). The practitioners might need to carefully assign the tasks into different groups to achieve the best performance (Standley et al., 2020). A considerable effort is also needed to find a suitable way to balance the performance of each task (Kendall et al., 2018; Chen et al., 2018b; Sener & Koltun, 2018). Recently, a few methods have been proposed to train and store multiple models with different trade-offs among the tasks (Lin et al., 2019; Mahapatra & Rajan, 2020; Ma et al., 2020).

In many real-world MTL applications, the system needs to make a trade-off among different tasks in real time, and it is desirable to have the whole set of optimal trade-off solutions. For example, in a self-driving system, multiple tasks must be conducted simultaneously but also compete for a fixed resource (e.g., fixed total inference time threshold), and their preferences could change in real time for different scenarios (Karpathy, 2019). A recommendation system needs to balance multiple criteria among different stakeholders simultaneously, and making trade-off adjustment would be a crucial component (Milojkovic et al., 2020). Consider the huge storage cost, it is far from ideal to train and store multiple models to cover different trade-off preferences, which is also not good for real-time adjustment.

In this paper, we propose a novel controllable Pareto multi-task learning framework, to learn the whole trade-off curve for all tasks with a single model. As shown in Fig.1, at the inference time, MTL practitioners can easily control the trade-off among tasks based on their preferences. The main contributions of this work are:

We formulate solving an MTL problem as a preference-conditioned multiobjective optimization problem, and propose a novel solution generator to learn the whole trade-off curve for the given problem.

We propose a general hypernetwork-based multi-task neural network framework, and develop an efficient end-to-end algorithm to simultaneously optimize different trade-off preferences via a single model.

Experiments on different MTL problems validate that the proposed method can successfully learn the trade-off curves and support real-time trade-off control.

Related Work

Multi-Task Learning. The current works on deep multi-task learning mainly focus on designing novel network architecture and constructing efficient shared representation among tasks (Zhang & Yang, 2017; Ruder, 2017). Different deep MTL networks, with hard or soft parameters sharing structures, haven been proposed in the past few years (Misra et al., 2016; Long et al., 2017; Yang & Hospedales, 2017). However, how to properly combine and learn different tasks together remains a basic but challenging problem for MTL applications. Although it has been proposed for more than two decades, the simple linear tasks scalarization approach is still the current default practice to combine and train different tasks in MTL problems (Caruana, 1997).

Some adaptive weight methods have been proposed to better combine all tasks in MTL problems with a single model (Kendall et al., 2018; Chen et al., 2018b; Liu et al., 2019; Yu et al., 2020). However, analysis on the relations among tasks in transfer learning (Zamir et al., 2018) and multi-task learning (Standley et al., 2020) show that some tasks might conflict with each other and can not be optimized at the same time. Sener and Koltun (Sener & Koltun, 2018) propose to treat MTL as a multiobjective optimization problem, and find a single Pareto stationary solution among different tasks. Pareto MTL (Lin et al., 2019) generalizes this idea, and proposes to find a set of solutions with different trade-off preferences. Recent works focuses on generating diverse and dense Pareto stationary solutions (Mahapatra & Rajan, 2020; Ma et al., 2020). These methods need to train and store multiple models to cover the trade-off curve for a given problem, which is undesirable for many real-world applications.

Very recently, there is a concurrent work (Navon et al., 2021) that also independently proposes to learn the entire trade-off curve for MTL problems by hypernetwork. Their work emphasizes the runtime efficiency on training for multiple preferences, and validate their models on small scale problems. We highlight our method’s advantage on supporting real-time preference control for inference, and show it can scale well for large scale MTL model.

Multi-Objective Optimization. Multi-Objective optimization itself is a popular research topic in the optimization community. Many gradient-based and gradient-free algorithms have been proposed in the past decades (Fliege & Svaiter, 2000; Désidéri, 2012; Miettinen, 2012). The closest method to our approach is the decomposition-based multi-objective evolutionary algorithm (MOEA/D (Zhang & Li, 2007)), which decomposes a multi-objective optimization problem into finite preference-based subproblems and solves them at the same time. We propose to learn the whole trade-off curve for a given problem with a single model, which might contain infinite trade-off solutions.

In addition to MTL, multi-objective optimization algorithms also can be used in reinforcement learning (Van Moffaert & Nowé, 2014) and neural architecture search (NAS) (Elsken et al., 2019; Lu et al., 2020). However, most methods directly use or modify well-studied multi-objective algorithms to find a single solution or a finite number of Pareto solutions, and do not support real-time trade-off control. Parisi et al. (2016) proposed to learn the Pareto manifold for a multi-objective reinforcement learning problem. However, since this method does not consider the preference for generation, it does not support real-time preference adjustment. Recently, some methods have been proposed to learn preference-based solution adjustment for multi-objective reinforcement learning (Yang et al., 2019) and image generation (Dosovitskiy & Djolonga, 2020) with simple linear combinations and model adaptions. This paper uses a hypernetwork to generate all parameters for the main multi-task neural network conditioned on different preferences.

HyperNetworks. The hypernetwork is initially proposed for dynamic modeling and model compression (Schmidhuber, 1992; Ha et al., 2017). It also leads to various novel applications such as hyperparameter optimization (Brock et al., 2018; MacKay et al., 2019), Bayesian inference (Krueger et al., 2017; Dwaracherla et al., 2020), and transfer learning (von Oswald et al., 2020; Meyerson & Miikkulainen, 2019). Recently, some discussions have been made on its initialization method (Chang et al., 2020), the optimization dynamic (Littwin et al., 2020), and its relation to other multiplicative interaction methods (Jayakumar et al., 2020).

MTL as Multi-Objective Optimization

An MTL problem involves learning multiple related tasks at the same time. For training a deep multi-task neural network, it is to minimize the losses for multiple tasks:

where $\theta$ is the neural network parameters and $\mathcal{L}_{i}(\theta)$ is the empirical loss of the $i$-th task. For learning all $m$ tasks together, an MTL system usually aims to minimize all losses at the same time. However, in many problems, it is impossible to find a single best solution to optimize all the losses simultaneously. With different trade-offs among the tasks, the problem (1) could have a set of Pareto solutions which satisfy the following definitions (Zitzler & Thiele, 1999):

Pareto dominance. Let $\theta_{a},\theta_{b}$ be two solutions for problem (1), $\theta_{a}$ is said to dominate $\theta_{b}$ ($\theta_{a}\prec\theta_{b}$) if and only if $\mathcal{L}_{i}(\theta_{a})\leq\mathcal{L}_{i}(\theta_{b}),\forall i\in\{1,...,m\}$ and $\mathcal{L}_{j}(\theta_{a})<\mathcal{L}_{j}(\theta_{b}),\exists j\in\{1,...,m\}$.

Pareto optimality. $\theta^{\ast}$ is a Pareto optimal solution if there does not exist $\hat{\theta}$ such that $\hat{\theta}\prec\theta^{\ast}$. The set of all Pareto optimal solutions is called the Pareto set. The image of the Pareto set in the objective space is called the Pareto front.

Finite Solutions Approximation. The number of Pareto solutions could be infinite, and their objective values are on the boundary of the valid value region (Boyd & Vandenberghe, 2004). Under mild conditions, the whole Pareto set and Pareto front would be $(m-1)$-dimensional manifolds in the solution space and objective space, respectively (Miettinen, 2012). For a general multiobjective optimization problem, no method can guarantee to find the Pareto front. Traditional multiobjective optimization algorithms aim at finding a set of finite solutions to approximate the whole Pareto set and Pareto front:

where $\hat{S}$ is the approximated Pareto set of $K$ estimated solutions, and $\hat{F}$ is the set of corresponding objective vectors in the objective space. The current multiobjective optimization based MTL methods aim to find a single ($K=1$) (Sener & Koltun, 2018) or a set of multiple ($K>1$) Pareto stationary solutions (Lin et al., 2019; Mahapatra & Rajan, 2020; Ma et al., 2020) for a given problem. For an MTL problem, each solution is a large deep neural network. To well cover a trade-off curve, the number of required solutions might grow exponentially with the number of tasks (Lin et al., 2019; Ma et al., 2020). The huge training and storage cost make these methods less practical for real-world applications.

Preference-Based Solution Generator

Instead of training multiple models, we propose to directly learn the whole trade-off curve for a MTL problem with a single model. Similar to other multi-objective optimization algorithms, our method can not guarantee to find the ground truth Pareto front, but we show that it can find a good approximated trade-off curve for various problems.

As shown in Fig. 2, we want to build a solution generator to map a preference vector $\boldsymbol{p}$ to its corresponding solution $\boldsymbol{\theta}_{\boldsymbol{p}}$. If an optimal generator $\boldsymbol{\theta}_{\boldsymbol{p}}=g(\boldsymbol{p}|\boldsymbol{\phi}^{*})$ is obtained, MTL practitioners can assign their preference via the preference vector $\boldsymbol{p}$, and directly obtain the corresponding solution $\boldsymbol{\theta}_{\boldsymbol{p}}$ with the specific trade-off among tasks. With the solution generator, we can obtain the approximated Pareto set/front:

where $\boldsymbol{P}$ is the set of all valid preference vectors, $g(\boldsymbol{p}|\boldsymbol{\phi}^{*})$ is the solution generator with optimal parameters $\phi*$. Once we have a proper generator $g(\boldsymbol{p}|\boldsymbol{\phi}^{*})$, we can reconstruct the whole approximated Pareto set $\hat{S}$ and the approximated Pareto front $\hat{F}$ by going through all possible preference vector $\boldsymbol{p}$. In the rest of this section, we discuss two approaches to define the form of preference vector $\boldsymbol{p}\in\boldsymbol{P}$ and its connection to the corresponding solution $\boldsymbol{\theta}_{p}=g(\boldsymbol{p}|\boldsymbol{\phi}^{*})$.

Preference-Based Linear Scalarization: A simple and straightforward approach is to define the preference vector $\boldsymbol{p}$ and the corresponding solution $\boldsymbol{\theta}_{\boldsymbol{p}}$ via the weighted linear scalarization:

Although this approach is straightforward, it is not optimal for multiobjective optimization. Linear scalarization can not find any Pareto solution on the non-convex part of the Pareto front (Das & Dennis, 1997; Boyd & Vandenberghe, 2004). In other words, unless the problem has a convex Pareto front, the generator defined by linear scalarization cannot cover the whole Pareto set manifold.

Preference-Based Multiobjective Optimization: To better approximate the Pareto set, we generalize the idea of decomposition-based multiobjective optimization (Zhang & Li, 2007; Liu et al., 2014) and Pareto MTL (Lin et al., 2019) to connect the preference vector and the corresponding Pareto solution. To be specific, we define the preference vector $\boldsymbol{p}$ as an $m$-dimensional unit vector in the loss space, and the corresponding solution $\boldsymbol{\theta}_{\boldsymbol{p}}$ is the one on the Pareto front which has the smallest angle with $\boldsymbol{p}$.

The idea is illustrated in in Fig. 3. With a set of randomly generated unit reference vectors $\boldsymbol{U}=\{\boldsymbol{u}^{(1)},\cdots,\boldsymbol{u}^{(K)}\}$ and the preference vector $\boldsymbol{p}$, an MTL problem is decomposed into different regions in the loss space. We call the region closest to $\boldsymbol{p}$ as its preferred region. The corresponding Pareto solution $\boldsymbol{\theta}_{\boldsymbol{p}}$ is the solution that belongs to the preferred region and on the Pareto front. Formally, we can define the corresponding Pareto solution as:

where $\mathcal{L}(\boldsymbol{\theta})$ is the loss vector and $\Omega(\boldsymbol{p},\boldsymbol{U})$ is the constrained preference region conditioned on the preference and reference vectors $\Omega(\boldsymbol{p},\boldsymbol{U})=\{\boldsymbol{v}\in R^{m}_{+}|\angle(\boldsymbol{v},\boldsymbol{p})\leq\angle(\boldsymbol{v},\boldsymbol{u}^{(j)}),\forall j=1,...,K\}$.

Controllable Pareto Multi-Task Learning

In this section, we propose a hypernetwork-based multi-task neural network framework, along with an efficient end-to-end optimization procedure to solve the MTL problem.

The proposed controllable Pareto multi-task network is shown in Fig. 4(a). As discussed in the previous section, we use a preference vector to represent a practitioner’s trade-off preference among different tasks. The hypernetwork takes the preference vector $\boldsymbol{p}$ as its input, and generates $\boldsymbol{\theta}_{\boldsymbol{p}}=g(\boldsymbol{p}|\boldsymbol{\phi})$ as the corresponding parameters for the main MTL network. The trainable parameters to be optimized are the hypernetwork parameters $\boldsymbol{\phi}$. Practitioners can easily control the MTL network’s performances on different tasks in real time, by simply adjusting the preference vector $\boldsymbol{p}$.

Main MTL Model Structure: In our proposed model, the main MTL network always has a fixed structure with a hard-shared encoder (Ruder, 2017; Vandenhende et al., 2021) as shown in Fig. 4(a). Once the parameters $\theta_{\boldsymbol{p}}$ are generated, an input $\boldsymbol{x}$ to the main MTL network will first go through the shared encoder, and then all task-specific heads to obtain the outputs $f_{i}(\boldsymbol{x}|\theta_{p})$. Different tasks share the same encoder, and they are regularized by each other as in the traditional MTL model. Our proposed model puts the cross-tasks regularization on the hypernetwork, where it should generate a set of good encoder parameters that work well for all tasks with a given preference.

Hypernetwork and Scalability: With the model compression ability powered by the hypernetwork, our proposed method can scale well to large scale models. Chunking (Ha et al., 2017; von Oswald et al., 2020) is a commonly used method to reduce the number of parameters for the hypernetwork. As shown in Fig. 4(b), a hypernetwork can separately generate small parts of the main network $\theta_{p}=[\theta_{p,1},\theta_{p,2},\cdots,\theta_{p,K}]$, with a reasonable model size and multiple trainable chunk embedding $\{\boldsymbol{c}_{k}\}_{k=1}^{K}$, where $\theta_{p,k}=g(\boldsymbol{p},\boldsymbol{c}_{k}|\phi)$. In this way, the hypernetwork can scale well for large MTL models. We use fully-connected networks as the hypernetwork, and the main MTL neural networks have the same structures as the models used in current MTL literature (Sener & Koltun, 2018; Liu et al., 2019; Vandenhende et al., 2021).

We follow the hypernetwork design proposed by Ha et al. (2017). The parameter generation process for a chunk of parameters is illustrated in Fig. 5. The hypernetwork first takes the preference vector $\boldsymbol{p}$ and a chunk embedding $\boldsymbol{c}_{k}$ as the input to a set of fully connected layers to obtain an output vector $\boldsymbol{a}_{\boldsymbol{p},k}\in R^{d}$. Then it use a linear project operation to generate the parameters $\boldsymbol{\theta}_{\boldsymbol{p},k,j}=\boldsymbol{W}_{j}\boldsymbol{a}$, where $\boldsymbol{W}_{j}\in R^{n_{j,1}\times n_{j,2}\times d}$ contains $n_{j,1}\times n_{j,2}\times d$ trainable parameters and $\boldsymbol{\theta}_{\boldsymbol{p},k,j}\in R^{n_{j,1}\times n_{j,2}}$. In the proposed hypernetwork, the fully connected layers are reusable to generate different $\boldsymbol{a}$ based on different chunk embedding. Most hypernetwork parameters are in the parameter tensors $\boldsymbol{W}_{j}$. By sharing the same $\boldsymbol{W}$ to generate different chunks of parameters with different $\boldsymbol{a}_{\boldsymbol{p},k}$, the number of parameters for the hypernetwork can be significantly compressed (Ha et al., 2017). In this work, we keep our hypernetwork-based model to have a comparable size with the corresponding MTL model. For inference, the fully connected layers can take multiple chunk embedding to generate different $\boldsymbol{a}$ in batch, and most computation is on the linear projection. It leads to an acceptable inference latency overhead, and supports real-time control for different trade-off preferences.

2 Optimization: Learning the Generator

Since we want to control the trade-off preference at the inference time, the proposed model should learn to perform well for all valid preference vectors rather than a single one. Suppose we have a probability distribution $P_{\boldsymbol{p}}$ for all valid preference vector $\boldsymbol{p}$, a general goal would be:

However, it is hard to optimize the trainable parameters $\boldsymbol{\phi}$ within the expectation directly. We use Monte Carlo method to sample the preference vectors, and use the stochastic gradient descent algorithm to train the hypernetwork-based MTL model. We can sample one preference vector $\boldsymbol{p}$ at a time and optimize the loss function:

At each iteration $t$, if we have a valid gradient direction $\boldsymbol{d}_{t}$ for the multi-objective loss $\mathcal{L}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))$, we can simply update the parameters with gradient descent $\boldsymbol{\phi}_{t+1}=\boldsymbol{\phi}_{t}-\eta\boldsymbol{d}_{t}$. For the preference-conditioned linear scalarization case as in problem (4), the calculation of $\boldsymbol{d}_{t}$ is straightforward:

For the preference-conditioned multiobjective optimization problem (5), a valid descent direction should simultaneously reduce all losses and activated constraints. One valid descent direction can be written as a linear combination of all tasks with dynamic weight $\boldsymbol{\alpha}_{i}(t)$:

where the coefficients $\boldsymbol{\alpha}_{i}(t)$ is depended on both the loss functions $\mathcal{L}_{i}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))$ and the activated constraints $\mathcal{G}_{j}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))$. We give the detailed derivation in the Appendix due to page limit.

The algorithm framework is shown in Algorithm 1. We use a simple uniform distribution on all valid preferences and references vector in this paper. The proposed algorithm simultaneously optimizes the hypernetwork for all valid preference vectors that represent different trade-offs among tasks. As shown in Fig. 6, it continually learns the trade-off curve during the optimization process. We obtain a preference-conditioned generator at the end, and can directly generate a solution $\boldsymbol{\theta}_{\boldsymbol{p}}=g(\boldsymbol{p}|\boldsymbol{\phi}_{T})$ from any preference vector $\boldsymbol{p}$. By taking all valid preference vectors as input, we can construct the whole approximated trade-off curve.

Synthetic Multi-Objective Problem

In this paper, we propose to use a hypernetwork to generate the parameters for an MTL network with different trade-off preferences. The MTL network parameters are in a high-dimensional decision space, and the optimization landscape is complicated with an unknown Pareto front. To better analyze the proposed algorithm’s behavior and convergence performance, we first use it to learn the Pareto front for a low-dimensional multiobjective optimization problem.

The experimental results are shown in Fig. 7. This synthetic problem has a sine-curve-like Pareto set in the solution space, and its Pareto front is a concave curve in the objective space. The detailed definition is given in the Appendix. Our proposed controllable Pareto MTL method can successfully learn and reconstruct the whole Pareto set/front from the preference vectors, while the traditional methods can only find a set of finite solutions. The simple preference-based linear scalarization method has poor performance for the problem with a totally concave Pareto front.

Experiments

In this section, we validate the performance of the proposed controllable Pareto MTL method to generate trade-off curves for different MTL problems. We compare it with the following MTL algorithms: 1) Linear Scalarization: simple linear combination of different tasks with fixed weights; 2) Uncertainty (Kendall et al., 2018): adaptive weight assignments with balanced uncertainty; 3) DWA (Liu et al., 2019): dynamic weight average for the losses; 4) MGDA (Sener & Koltun, 2018): multiple gradient descent algorithm, to find one Pareto stationary solution; 5) Pareto MTL (Lin et al., 2019): to find a set of wildly distributed Pareto stationary solutions; and 6) Single Task: the single task learning baseline.

We conduct experiments on the following widely-used MTL problems: 1) MultiMNIST (Sabour et al., 2017): This problem is to simultaneously classify two digits on one image. 2) CityScapes (Cordts et al., 2016): This dataset has street-view RGB images, and involves two tasks to be solved, which are pixel-wise semantic segmentation and depth estimation. 3) NYUv2 (Silberman et al., 2012): This dataset is for indoor scene understanding with two tasks: a $13$-class semantic segmentation and indoor depth estimation. 4) CIFAR-100 with 20 Tasks: Follow a similar setting in (Rosenbaum et al., 2018), we split the original CIFAR-100 dataset (Krizhevsky & Hinton, 2009) into 20 five-class classification tasks. An overview of the models we used for each problem can be found in Table.1, and the details for these problems can be found in the Appendix.

Result Analysis: The experiment results on MultiMNIST, CityScapes and NYUv2 are shown in Fig. 8(a)(b)(c) respectively, where all models are trained from scratch. In all problems, our proposed algorithm can learn the trade-off curve with a single model, while the other methods need to train multiple models and cannot cover the entire curve. In addition, the proposed preference-conditioned multiobjective optimization method can find a better trade-off curve that dominates the curve found by the simple linear scalarization. These results validate the effectiveness of the proposed model and the end-to-end optimization method.

For all experiments, the single-task learning is a strong baseline in performance with a larger model size ($m$ full models). However, it cannot dominate most of the approximated Pareto front learned by our model. Our models can provide diverse optimal trade-offs among tasks (e.g., in the upper left and lower right area) for different problems and support real-time trade-off adjustment.

The experimental result on the $20$-tasks CIFAR100 classification problem is shown in Fig. 9. Our proposed model can achieve the best overall performance by making a real-time preference adjustment for predicting different tasks. It also outperforms the balanced hypernet approach, which has the same hypernetwork-based model but optimizes different tasks with an equal preference. This result shows the advantage of our proposed model on preference-based modeling and real-time preference adjustment for inference.

Scalability and Latency: Table.1 summarizes the information of all models we use in different problems. As discussed in the previous sections, we build the hypernetwork such that the proposed models have a comparable number of parameters with the corresponding single MTL model. Given the current works (Lin et al., 2019; Mahapatra & Rajan, 2020; Ma et al., 2020) need to train and store multiple MTL models to approximate the Pareto front, our proposed model is more parameter-efficient while it can learn the whole trade-off curve. In this sense, it is much more scalable than the current methods. The size of hypernetwork-based models can be further reduced due to its compression ability (Ha et al., 2017). Designing more efficient preference-based MTL models is a promising research direction in the future.

The inference latency for all models are also reported. The latency is measured on a 1080Ti GPU with the training batch size for each problem. We report the mean and standard deviation over $100$ independent runs. For our proposed model, we randomly adjust the preference for each batch. Our model has an affordable overhead on the inference latency, which is suitable for real-time preference adjustment.

Experiment with Pretrained Backbone: We conduct an experiment on the NYUv2 problem with a large-scale ResNet-50 backbone (He et al., 2016) following the experimental setting in Vandenhende et al. (2021). Each model has a ResNet-50 backbone pretrained on ImageNet, and task-specific heads with ASPP module (Chen et al., 2018a). The results are shown in Fig.10, where the results for MTL models are from Vandenhende et al. (2021) with the best finetuned configurations. Our proposed model also uses the pretrained ResNet-50 backbone, and the hypernetwork generates the encoder’s last few layers and all task-specific heads. Our model can learn a good trade-off curve between the two tasks, and its performance could be further improved with task balancing methods and better decoder architecture design as in Vandenhende et al. (2021).

More results and discussions can be found in the Appendix.

Conclusion

In this paper, we proposed a novel controllable Pareto multi-task learning framework for solving MTL problems. With a preference-based hypernetwork, our method can learn the whole trade-off curve for all tasks with a single model. It allows practitioners to easily make real-time trade-off adjustment among tasks at the inference time. Experimental results on various MTL applications demonstrated the usefulness and efficiency of the proposed method.

References

Appendix

We provide more discussion and analysis in this Appendix, which can be summarized as follows:

Preference-based Multiobjective Gradient: We give a detailed derivation for preference-based multiobjective gradient descent in Section A.

Discussion on the Convergence Behavior: We discuss the convergence behavior of the proposed method in Section B.

More Experiments: We compare our proposed methods with the concurrent proposed hypernetwork-based method in Section C.

Experimental Setting: The detailed settings for all experiments are provided in Section D.

Code: We will make our code publicly available.

Appendix A Preference-based Multiobjective Gradient Descent

In this section, we give a detailed derivation for the preference-based multiobjective gradient direction and a batched preferences optimization variant for training the hypernetwork-based MTL model.

As mentioned in Algorithm 1 in the main paper, we use gradient descent to update the solution generator at each iteration. For the case of linear scalarization, calculating the gradient direction is straightforward. In this subsection, we discuss how to obtain a valid gradient direction for the preference-conditioned multiobjective optimization. Similar to the previous work (Sener & Koltun, 2018; Lin et al., 2019), we use multiobjective gradient descent (Fliege & Svaiter, 2000; Désidéri, 2012) to solve the MTL problem.

The key difference between our proposed method and previous approaches is the parameters to be optimized. In our algorithm, the optimization parameter is $\boldsymbol{\phi}$ for the solution generator $\boldsymbol{\theta}_{\boldsymbol{p}}=g(\boldsymbol{p}|\boldsymbol{\phi})$ considering all preferences rather than $\boldsymbol{\theta}$ for a single solution.

A simple illustration of the preference-based multiobjective gradient descent for a problem with two tasks is shown in Fig. 11. At each iteration $t$, the algorithm samples a preference vector $\boldsymbol{p}$ and obtains its current corresponding solution $\boldsymbol{\theta}_{\boldsymbol{p},t}$. A valid gradient direction should reduce all the losses and guide the generated solution $\boldsymbol{\theta}_{\boldsymbol{p},t}=g(\boldsymbol{p}|\boldsymbol{\phi}_{t})$ toward the preference region $\Omega(\boldsymbol{p},\boldsymbol{U})$ around the preference vector $\boldsymbol{p}$. The preference-based multiobjective optimization problem can be written as:

The generator’s parameter $\boldsymbol{\phi}_{t}$ is the only trainable parameters to be optimized. With a sampled preference $\boldsymbol{p}$, what we need is to find a valid gradient direction $\boldsymbol{d}_{t}$ for updating $\boldsymbol{\phi}_{t}$ to reduce all the losses $\mathcal{L}_{i}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))$ and activated constraints $\mathcal{G}_{j}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t})|\boldsymbol{p},\boldsymbol{U})$.

We follow the methods proposed in (Fliege & Svaiter, 2000; Gebken et al., 2017), and calculate a valid descent direction $\boldsymbol{d}_{t}$ by solving the optimization problem:

where $\boldsymbol{d}_{t}$ is the obtained gradient direction, $\alpha$ is an auxiliary parameter for optimization, and $I(\theta)=\{j\in I|\mathcal{G}_{j}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))\geq 0\}$ is the index set of all activated constraints. By solving the above problem, the obtained direction $\boldsymbol{d}_{t}$ and parameter $\alpha_{t}$ will satisfy the following lemma (Gebken et al., 2017):

Lemma 1: Let $(\boldsymbol{d}_{t},\alpha_{t})$ be the solution of problem (11), we either have:

A non-zero $\boldsymbol{d}_{t}$ and $\alpha_{t}$ with

A.2 Adaptive Linear Scalarization

As mentioned in the main paper, we can rewrite the gradient direction $\boldsymbol{d}_{t}$ as a dynamic linear combination with the gradient of all losses $\nabla_{\boldsymbol{\phi}}\mathcal{L}_{i}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))$. Similar to the approach in (Fliege & Svaiter, 2000; Lin et al., 2019), we reformulate the problem (11) in its dual form:

By solving this problem, we obtain the valid gradient direction $\boldsymbol{d}_{t}=\sum_{i=1}^{m}\lambda_{i}\nabla_{\boldsymbol{\phi}_{t}}\mathcal{L}_{i}(\boldsymbol{\theta}_{\boldsymbol{p},t})+\sum_{j}\beta_{j}\nabla_{\boldsymbol{\phi}_{t}}\mathcal{G}_{j}(\boldsymbol{\theta}_{\boldsymbol{p},t})$, where $\lambda_{i}$ and $\beta_{j}$ are the Lagrange multipliers for the linear inequality constraints in problem (11).

Based on the definition in problem (10), we can rewrite the constraint $\mathcal{G}_{j}(g(\boldsymbol{p}|\boldsymbol{\phi}_{t}))$ as a linear combination of all losses:

Similarly, the gradient of the constraint is also a linear combination of the gradients for all losses:

Therefore, we can rewrite the valid descent direction as a linear combination of the gradients for all tasks with dynamic weight $\boldsymbol{\alpha}_{i}(t)$:

The coefficient $\lambda_{i}$ and $\beta_{j}$ are obtained by solving the dual problem (A.2).

We use the Frank-Wolfe algorithm (Jaggi, 2013) to solve the problem (A.2) as in the previous work (Sener & Koltun, 2018; Lin et al., 2019). We use simple uniform distribution to sample both the unit preference vector $\boldsymbol{p}$ and unit reference vectors $\boldsymbol{U}$ in this paper. The number of preference vector is $1$ in the previous discussion, and the number of reference vectors is a hyperparameter, which we set it as $3$ for all experiments.

A.3 Batched Preferences Update

In the main paper, we sample one preference vector at each iteration to calculate a valid direction $\boldsymbol{d}_{t}$ to update the Pareto generator. A simple and straightforward extension is to sample multiple preference vectors to update the Pareto solution generator.

At each iteration, we can simultaneously sample and optimize multiple preferences:

where $\boldsymbol{p}_{1},\boldsymbol{p}_{2},\cdots,\boldsymbol{p}_{K}$ are $K$ randomly sampled preference vectors and each $\mathcal{L}(g(\boldsymbol{p}_{k}|\boldsymbol{\phi}_{t}))$ is a multiobjective optimization problem. Therefore, we now have a hierarchical multiobjective optimization problem. If we do not have a specific preference among the sampled preference vectors, the above problem can be expanded as a problem with $Km$ objectives:

For the linear scalarization case, the calculation of valid gradient direction at each iteration $t$ is straightforward:

where we assume all sampled preferences are equally important.

It is more interesting to deal with the preference-conditioned multiobjective optimization problem. For each preference vector $\boldsymbol{p}_{k}$, suppose we use the rest preference vectors as its reference vectors, the obtained preference-conditioned multiobjective optimization problem would be:

There are $(K-1)$ constraints for each preference vector, hence total $K(K-1)$ constraints for the preference-conditioned multiobjective problem, although many constraints could be inactivated during training. Similar to the single preference case, we can also calculate the valid gradient direction in the form of adaptive linear combination as:

where the adaptive weight $\boldsymbol{\beta}_{i}(t)$ depends on all loss functions $\mathcal{L}_{i}(g(\boldsymbol{p}_{k}|\boldsymbol{\phi}_{t}))$ and activated constraints $\mathcal{G}_{kj}(g(\boldsymbol{p}_{k}|\boldsymbol{\phi}))$.

Appendix B Convergence Analysis

We have shown that our proposed method can find good trade-off curves for different large scale MTL problems via a single model. However, similar to other multiobjective optimization algorithms, our method can not guarantee to find the ground-truth Pareto front for a general problem.

As discussed in the main paper and previous section, the general goal for learning the solution generator is:

It is hard to analyze the proposed method’s convergence behavior, especially when training a deep MTL problem with complicated optimization landscapes and infinite preferences. In this section, we briefly discuss the case with single, multiple, and infinite preferences. We hope this discussion can lead to a better understanding of our proposed algorithm, and could be useful for potential future work on approximating and learning the whole Pareto front.

Single Preference. If we only have a single preference $\boldsymbol{p}_{s}$, the optimization problem will reduce to:

Since the preference $\boldsymbol{p}_{s}$ is fixed, it is the case for finding a single trade-off solution with a single model as in the previous work (Sener & Koltun, 2018; Lin et al., 2019; Ma et al., 2020; Mahapatra & Rajan, 2020). The only difference is our method has an extra hypernetwork structure.

With a valid gradient direction and proper step size at each iteration as discussed in Section A, we can iteratively update $\boldsymbol{\phi}$ to improve all objective values $\mathcal{L}(g(\boldsymbol{p}_{s}|\boldsymbol{\phi}))$. If no valid descent direction can be found, the generated solution $\boldsymbol{\theta}_{\boldsymbol{p}_{s}}$ converges to a local Pareto stationary point (might not be the Pareto optimal point for a general problem). Therefore, it shares the same convergence guarantee with the single model counterparts (Sener & Koltun, 2018; Lin et al., 2019).

Fixed and Finite Multiple Preferences. If we have a set of different but fixed preferences $\{\boldsymbol{p}_{1},\cdots,\boldsymbol{p}_{K}\}$, the optimization problem would be:

where each $\mathcal{L}(g(\boldsymbol{p}_{k}|\boldsymbol{\phi}))$ is a single preference-based multiobjective optimization problem as in the previous case. We can use the batched-preference method in Section A.3 to update the generator for all preferences at each iteration.

Since different preference-based problems are closely related to each other, their optimal solutions should also share some common properties (e.g., in the same $(m-1)$-dimensional manifold as discussed in the main paper). If the hypernetwork has enough learning capacity, it can generate desired solutions for every single preference-based problem at the same time. With proper assumption, we expect all fixed preferences have the same convergence guarantee with the single-preference case.

Infinite Preferences. The general and most challenging case is with infinite preferences as in problem (29). Since we can not optimize the generator for infinite preferences at one step, we sample one or a set of finite preferences to optimize at each iteration $t$, and have:

It is the optimization method we use in this paper.

This method is a Monte Carlo sampling and approximation to optimize the expectation, which is similar to stochastic gradient descent (SGD) or batched SGD against full gradient descent. By iteratively sampling and optimizing for different preferences, the solution generator continually learns and improves its performance for all preferences.

As discussed in the main paper, the set of all valid preference vectors $\boldsymbol{p}\in P_{\boldsymbol{p}}$ is an $(m-1)$-dimensional manifold. Under mild assumptions, the Pareto set is also an $(m-1)$-dimensional manifold, although it is in much higher decision space. With enough learning capacity, we hope the generator can have optimal parameters $\boldsymbol{\phi}^{*}$ for all preferences. However, it is challenging to guarantee the generated solutions for infinite preferences are all Pareto stationary points. Even if this is the case, we can not guarantee the generated manifold would well cover the Pareto set and front (even a local one). Future work in this direction would be crucial for designing more efficient methods to learn the whole Pareto front with the convergence guarantee.

Appendix C More Experimental Results

In the main paper, we have shown that our proposed method can successfully learn the trade-off curves for large scale MTL problems with a single model, and support real-time trade-off control with minimal inference overhead. To our best knowledge, this is the first approach to learn the trade-off curves for large scale MTL problems.

In this section, we compare its performance with two methods on learning the trade-off curves on small scale problems: 1) Continuous Pareto Exploration (Ma et al., 2020): It proposes to use a gradient-based Pareto exploration method to continuously generate and store a dense set of separate Pareto stationary solutions; 2) Pareto HyperNetwork (PHN) (Navon et al., 2021): This is a concurrent work to our method. It also independently proposes using a hypernetwork to learn the Pareto front, but the current method only works on small scale problems.

Problems: We run experiments on the MultiMNIST problem (Sabour et al., 2017) and two variants, namely MultiFashion and MultiFashionMNIST, as used in the above work (Ma et al., 2020; Navon et al., 2021). These problems are to classify two digits, two fashion items, and one digit with one fashion item in a single image. Details can be found in Section D.

Models: We use the open-sourced code for both works to reproduce the results https://github.com/mit-gfx/ContinuousParetoMTLhttps://github.com/AvivNavon/pareto-hypernetworks. For the continuous Pareto exploration method, we modify the MTL model to let it have the same structure as in our method and PHN. We use most default hyperparameters in the code, but carefully fine-tune the exploration step size to let it have a good dense approximation. Follow the setting in their paper, we first generate five seed solutions with linear scalarization (LS) or Pareto multi-task learning (PMTL), and then use Pareto exploration to generate 20 new solutions for each seed solution. Therefore, this method has to train and store $105$ full MTL models in total to approximate the Pareto front.

For the Pareto HyperNetwork method (PHN), we use the default hypernetwork and hyperparameters, which has the same main MTL model structure as in our work. We have tried to fine-tune the hyperparameters and learning strategies, but failed to let it have the same performance with our method. We believe the large hypernetwork size (total $3.2M=3,200K$ parameters) makes it hard to optimize. Recent work on hypernetwork’s initialization method (Chang et al., 2020) and optimization dynamic (Littwin et al., 2020) would be useful to further improve this method’s performance with a large hypernetwork.

Result Analysis: The experiment results are shown in Fig. 12 and Table.2. To compare the quality of approximated Pareto front, we also report the hypervolume indicator (Zitzler & Thiele, 1999) for all methods on each problem. The hypervolume indicator measures the area between a reference point to a set of solutions. Let $S\subset R^{m}$ be a set of solutions in the objective space, and $z^{*}$ be a point dominated by all the points in $S$, the hypervolume $H(S)$ of $S$ is defined as the volume of the set:

With a fixed reference point, a better approximated Pareto front should have a larger hypervolume value. We use a reference point $(0.5,0.5)$ for all experiments.

Our proposed method can generate better or comparable trade-off curves for all problems but with a significantly smaller model size (only $1.1\%$ of the compared methods). The small model size also makes our method suitable for real-time trade-off control. Our model can have a significantly smaller size due to the similarity among different preference-based solutions (e.g., in the same $(m-1)$-dimensional manifold). With the model compression method discussed in the main paper, we can share a large amount of parameters among different preferences without decreasing their performance. This property makes it scale well for large MTL models, which is crucial for real-world applications.

Appendix D Experimental Setting

The synthetic example we use in the main paper is defined as:

We set $n=10$ and use a simple two-layer MLP network with 50 hidden units on each layer to generate the Pareto solutions based on the preference vectors.

MultiMNIST (Sabour et al., 2017) and Two Variants:

In this problem, the goal is to classify two overlapped digits in an image at the same time. The size of the original MNIST image is $28\times 28$. We randomly choose two digits from the MNIST dataset, and move one digit to the upper-left and the other one to the bottom right with up to $4$ pixels. Therefore, the input image is in size $36\times 36$. MultiFashion and MultiFashionMNIST (Lin et al., 2019) are two variants for the MultiMNIST problem, which is to classify two fashion items (Xiao et al., 2017), or one fashion item and one digit in a single image.

Similar to the previous work (Sener & Koltun, 2018; Lin et al., 2019), we use a LeNet-based neural network with two task-specific fully connected layers as the main MTL model. The hypernetwork is a simple MLP. Since the LeNet model is small, we do not use any chunk embedding. We let the hypernetwork-based model have a similar number of parameters with a single MTL model. For all methods, the optimizer is Adam with learning rate lr = $3e^{-4}$, the batch size is $256$, and the number of epochs is $200$.

CityScapes (Cordts et al., 2016):

This dataset has street-view RGB images, and involves two tasks to be solved, which are pixel-wise semantic segmentation and depth estimation. We follow the setting used in (Liu et al., 2019), and resize all images into $128\times 256$. For the semantic segmentation, the model predicts the $7$ coarser labels for each pixel. We use the L1 loss for the depth estimation. We report the experimental results on the Cityscapes validation set.

We use the MTL network proposed in (Liu et al., 2019), which has SegNet (Badrinarayanan et al., 2017) as the shared representation encoder, and two task-specific lightweight convolution layers. In our hypernetwork-based model, the hypernetwork contains three 2-layer MLPs with $100$ hidden units on each layer, and most parameters are stored in the parameter tensors for linear projection. The preference embedding and chunk embedding are all $64$-dimensional vectors. We also let the hypernetwork-based model have a similar number of parameters with the single MTL model. For all experiments, we use Adam with learning rate lr = $3e^{-4}$ as the optimizer, and the batch size is $12$. We train the model from scratch with $200$ epochs.

NYUv2 (Silberman et al., 2012):

This dataset is for indoor scene understanding with two tasks: a $13$-class semantic segmentation and indoor depth estimation. Similar to Liu et al. (2019), we resize all images into $288\times 384$. For the training from scratch experiment, we use a similar MTL network and hyperparameter setting as for the CityScapes problem, except the batch size is $8$ in this problem.

We also test the performance on models with pretrained encoder on the NYUv2 Dataset. We follow the setting in the recent MTL survey paper (Vandenhende et al., 2021), all models have a ResNet-50 backbone (He et al., 2016) pretrained on ImageNet, and two-specific heads with ASPP module (Chen et al., 2018a). In our hypernetwork-based model, the hypernetwork has three different 2-layer MLPs with $200$ hidden unit on each layer. One MLP is for generating shared convolution layers on top of the ResNet backbone, and the other two are for each task-specific head. The shared parameters for backbone are unfrozen, and will be adapted during training. We use $100$-dimensional vectors as the preference and chunking embedding. We use Adam with learning rate lr = $1e^{-4}$ as the optimizer, the batch size is $8$, and the total epoch is $200$.

CIFAR100 with 20 Tasks:

To validate the algorithm performance on MTL problem with many tasks, we split the CIFAR-100 dataset (Krizhevsky & Hinton, 2009) into 20 tasks, where each task is a $5$-class classification problem. Similar setting has been used in the previous work with MTL learning (Rosenbaum et al., 2018) and continual learning (von Oswald et al., 2020).

The MTL neural network has four convolution layers as the shared architecture, and $20$ task-specific FC layers. In our proposed model, we have $1$ MLP as the hypernetwork and the preference and chunking embedding are both $32$-dimensional vectors. The optimizer is Adam with learning rate lr = $3e^{-4}$, the batch size is $128$, and the number of epochs is $200$. We report the test accuracy for all $20$ tasks.