The Conference Paper Assignment Problem: Using Order Weighted Averages to Assign Indivisible Goods

Introduction

Assigning indivisible items to multiple agents is a fundamental problem in many fields including computer science, economics and operations research. Algorithms for matching and assignment are used in a variety of application areas including allocating runways to airplanes, residents to hospitals, kidneys to patients , students to schools , assets to individuals in a divorce, jobs to machines, and tasks to cloud computing nodes . Understanding the properties of the underlying algorithms is an important aspect to ensuring that all participating agents are happy with their allocations and do not attempt to misrepresent their preferences; a key area of study for computational social choice .

An area that is near to many academics’ hearts is the problem of allocating papers to referees for peer review. The results of grant, journal, and conference reviewing can have significant impact on the careers of scientists. Ensuring that papers and proposals are reviewed by the most qualified/interested referees most is part of ensuring that items are treated properly and all participants support the outcome of the processes. Making sure these processes work for both the proposers and the reviewers is important and methods for improving peer review have been proposed and discussed in AI and broadly across the sciences .

There are a number of ways one can improve the quality of peer review . First is to ensure that reviewers are not incentivized to misreport their reviews for personal gain. Along this line there has been significant interest recently in strategyproof mechanisms for peer review . Unfortunately, the method that we discuss in this paper is not strategyproof. Another way is to ensure that reviewers are competent to provide judgements on the papers they are assigned. The Toronto Paper Matching System is designed to improve the process from this paper-centric model. A third alternative, and the one we focus on in this study, is ensuring that reviewers are happy with the papers they are asked to review. This is fundamentally a question about the optimization objectives of the assignment functions used.

Formally, we study the Conference Paper Assignment Problem (CPAP) which is a special of the Multi-Agent Resource Allocation Problem (MARA) , and propose a novel assignment, the $\Sigma$-OWA assignment. In the CPAP setting we have a two-sided market where on one side the agents/reviewers have preferences over the other side, the objects/papers, and both sides have (possibly infinite) upper and lower capacities. A fundamental tension in assignment settings is the tradeoff between maximizing the social welfare, also know as the utilitarian maximal assignment and the Rawlsian fairness concept of maximizing the utility of the worst off agent, known as the egalitarian maximal assignment. These two ideas are incompatible optimization objectives and diverge in a computational sense as well: computing the utilitarian assignment for additive utilities can be done in polynomial time, while computing the egalitarian assignment is NP-complete . This, perhaps, could be the reason that implementers of large conference paper assignment software often opt for utilitarian assignments, as is supposedly the case for EasyChair . This is technically unsubstantiated as when the authors contacted EasyChair to understand the assignment process we were told, “We do not provide information on how paper assignment in EasyChair is implemented. The information in Garg et.al. may be incorrect or out of date - none of the authors worked for EasyChair, they also had no access to the EasyChair code.” However, it is also not clear if an egalitarian assignment is desirable for CPAP.

Contributions. We establish a motivation for using OWA vectors in the assignment setting and define a novel notion of allocation, the $\Sigma$-OWA assignment. We give algorithm to compute an $\Sigma$-OWA maximal assignment in polynomial time and we show that the $\Sigma$-OWA objective generalizes the utilitarian objective. We show that $\Sigma$-OWA assignments satisfy a notion of Pareto optimality w.r.t. the pairwise comparisons of the objects by the agents. We implement an algorithm for $\Sigma$-OWA assignments and perform experiments on real world conference paper assignment data.

Preliminaries

From here we will use the more general notation agents/objects to describe our setting. In assignment settings each agent provides their preference over the objects as a reflexive, complete, and transitive preference relation (weak order) over the set of objects, $\succsim_{i}$. We do not assume that $\succsim_{i}$ is complete; it is possible that some agents may have conflicts of interest or have no preference for a particular object; this assumption is often called “having unacceptable objects” in the literature .

In many real-world CPAP settings there are a fixed number of equivalence classes into which agents are asked to place the objects . We assume that the number of equivalence classes (ranks) of objects are given as input to the problem and agents tell us within which rank each objects belong. Agents also provide a decreasing utility value for each rank We assume that agents can give any utilities as input. However, often the utilities are restricted to be the same, i.e., Borda utilities in conference paper bidding, or come from some fixed budget, i.e., bidding fake currency as in course allocation at Harvard .. Our main result can be extended to the case where the number of equivalence classes is not fixed.

There are two practical constraints that we include in our model, making our model more general than the standard MARA or CPAP problems studied in computer science : upper and lower capacities on both the agents and objects.

Agent Capacity: each agent $i\in N$ has (possibly all equal) upper and lower bound on their capacity, the number of objects they can be allocated, $c^{N}_{min}(i)$ and $c^{N}_{max}(i)$.

Object Capacity: each object $j\in O$ has a (possibly all equal) upper and lower bound on the number of agents assigned to it, $c^{O}_{min}(j)$ and $c^{O}_{max}(j)$, respectively.

We can now define a feasible assignment $A$ for an instance $(N,O,\succ,\textbf{u},\Delta)$. For a given assignment $A$, let $A(i,:)$ denote the set of objects assigned to agent $i$ in $A$, let $A(:,j)$ denote the set of agents assigned to object $j$, and let $|\cdot|$ denote the size (number of elements) of a set or vector. A feasible assignment $A$ must obey:

2 Individual Agent Evaluation

For indivisible (discrete) objects the lexicographic relation can be modeled by the additive utility relation by setting the agent utilities to high enough values. Formally, if the utility for rank $i<j$ is $\textbf{u}(i)>\textbf{u}(j)\cdot m$ then the lexicographic and additive utility relations are the same, i.e., no matter how many additional objects of rank $j$ the agent receives, one additional object of rank $i$ is more preferred. We now define the relations that a referee might consider between assignments A and B.

Lexicographic: An agent $i$ lexicographically prefers $A$ to $B$ if $\boldsymbol{\sigma}_{i}(A)$ comes before $\boldsymbol{\sigma}_{i}(B)$ in the lexicographic order. That is, there is an index $1\geq l\geq\Delta$ such that for all $k>l$ we have $\boldsymbol{\sigma}_{i,k}(A)=\boldsymbol{\sigma}_{i,k}(B)$ and $\boldsymbol{\sigma}_{i,l}(A)>\boldsymbol{\sigma}_{i,l}(B)$; i.e., $i$ receives at least one more paper of a higher rank in $A$ than in $B$. The lexicographic relation over vectors has a long history in the assignment literature .

Additive Utility: An agent $i$ prefers assignment $A$ to $B$ if he has more additive utility for the objects assigned to him in $A$ than in $B$. Formally, (and slightly abusing notation) $\textbf{u}_{i}(A)=\sum_{j\in A(i,:)}\textbf{u}_{i}(r_{i}(j))>\sum_{j\in B(i,:)}\textbf{u}_{i}(r_{i}(j))$, or an alternative formulation using the dot product, $\textbf{u}_{i}(A)=\textbf{u}_{i}\cdot\boldsymbol{\sigma}_{i}(A)>\textbf{u}_{i}\cdot\boldsymbol{\sigma}_{i}(B).$

3 Overall Assignment Evaluation

In the literature there are several optimization objectives defined over an assignment that an implementer may wish to consider. We limit our discussion to the two classical notions below. Additional discussion of objectives, including the imposition of various fairness criteria for the CPAP setting can be found in Garg et al. 2010 and for the MARA setting see e.g., Bouveret and Lemaître 2016.

Utilitarian Social Welfare Maximal Assignment: Often called the utilitarian assignment, we want to maximize the total social welfare over all the agents. An assignment is a utilitarian assignment if it satisfies:

Egalitarian Social Welfare Maximal Assignment: Often called the egalitarian assignment, we want to enforce the Rawlsian notion of fairness by making sure that the worst off referee is as happy as possible, i.e., maximize the utility of the least well off agent. Formally,

In the discrete MARA and CPAP setting where objects are not divisible, the problem of finding an egalitarian assignment is NP-hard while finding a utilitarian assignment can be done in polynomial time .

Background and Related Work

One and two sided matching and assignment problems have been studied in economics and computer science for over 40 years. Matching and assignment have many applications including kidneys exchanges and school choice . Our problem is often called the multi-agent resource allocation (MARA) problem in computer science The papers to referees formulation of this problem has some additional side constraints common in the economics literature, but not as common in computer science . In the economics literature the Workers-Firms problem is the most closely related analogue to our problem, modeling many-many matchings with capacities .

The conference paper assignment has been studied a number of times over the years in computer science , as has defining and refining notions of fairness for the assignment vectors in multi-agent allocation problems . We build off the work of Garg et al. 2010, who extensively study the notion of fair paper assignments, including lexi-min and rank-maximal assignments, within the context of conference paper assignment. Garg et al. 2010 show that for the setting we study, finding an egalitarian optimal assignment and finding a leximin optimal assignment are both NP-hard when there are three or more equivalence classes; and polynomial time computable when there are only two. They also provide an approximation algorithm for leximin optimal assignments. We know that if the capacity constraints are hard values, i.e., each reviewer must review $\leq x$ papers and each paper must receive exactly $y$ reviews, then the resulting version of capacitated assignment is NP-hard . Answer set programming for CPAP was studied by Amendola et al. 2016; they encode the CPAP problem in ASP and show that finding a solution that roughly correspond to the leximin optimal and egalitarian solutions can be done in reasonable time for large settings ($\approx 100$ agents).

CPAP also receives considerable attention in the recommender systems and machine learning communities. Often though, this work takes the approach of attempting to infer a more refined utility or preference model in order to distinguish papers. Fairness and efficiency concerns are secondary. A prime example of this is the Toronto Paper Matching System designed by Charlin and Zemel 2013. This system attempts to increase the accuracy of the matching algorithms by having the papers express preferences over the reviewers themselves; where these preferences are inferred from the contents of the papers.

We make use of Order weighted averages (OWAs), often employed in multi-criteria decision making . OWAs have recently received attention in computational social choice for voting and ranking , finding a collective set of items for a group , and multi-winner voting with proportional representation . The key difference between CPAP and voting using OWAs in the ComSoc literature is that CPAP does not select a set of winners that all agents will share. Instead, all agents are allocated a possibly disjoint set of objects.

Σ\Sigma-OWA Assignments

We now formally define OWAs and their use for defining assignment objectives. We will discuss alternative formulations of $\Sigma$-OWA which have been studied.

An order weighted average (OWA) is a function defined for an integer $K$ as a vector $\boldsymbol{\alpha}^{(K)}=(\alpha_{1},\ldots,\alpha_{K})$ of $K$ non-negative numbers. Let $\textbf{x}=(x_{1},\ldots,x_{K})$ be a vector of $K$ numbers and let $\textbf{x}^{\downarrow}$ be the non-increasing rearrangement of x, i.e. $\textbf{x}^{\downarrow}=\textbf{x}^{\downarrow}_{1}\geq\textbf{x}^{\downarrow}_{2}\geq\ldots\geq\textbf{x}^{\downarrow}_{K}$. Then we say:

In order to apply OWAs to our setting we need to define the weighted rank signature of an assignment. Let $\boldsymbol{\omega}_{i}(A)$ be defined as the sorted vector of utility that a referee gets from an assignment $A$. Formally,

For example, if $A(i,:)$ included two objects with utility $3$, one of utility $1$, and one of utility $0$, we would have

Our inspiration for applying OWAs comes from a multi-winner voting rule known as Proportional Approval Voting (PAV) . In approval voting settings, each agent can approve of as many candidates as they wish. Under the standard approval voting (AV) method, all approvals from each agent assign one point to the candidate for which they are cast. However, this can lead to a number of pathologies described by Aziz et al. 2015b and it intuitively does not seem fair; once a candidate that you like has been selected to the winning set your next candidate selected to the winning set should seemingly count less. Hence in PAV, which is designed to be more fair , a voter’s first approval counts for a full point, the second for $\nicefrac{{1}}{{2}}$, the next for $\nicefrac{{1}}{{3}}$, and on as a harmonically decreasing sequence.

Transitioning this logic to the CPAP setting, we were motivated to find a way to distribute objects to agents that increases the number of agents who receive their top ranked objects. This is the logic of PAV: once you get a candidate into the winning set, you should count less until everyone else has a candidate in the winning set. If we desire to directly get a rank maximal assignment, completely ignoring the utilities, then we know this is polynomial by a result from Garg et al. 2010. However, if we wish to modulate between using the utilities and using only the ranks, perhaps we can use OWAs. We use the sum over all agents of $OWA_{\boldsymbol{\alpha}}(\boldsymbol{\omega})=\boldsymbol{\alpha}\cdot\boldsymbol{\omega}$ as the optimization criteria for the assignment.

In order to cleanly define this we need to place some restrictions on our OWA vectors. Firstly, the length of $\boldsymbol{\alpha}$ needs to be at least as long as the maximum agent capacity, i.e., $|\boldsymbol{\alpha}|\geq\arg\max_{i\in N}(c^{N}_{max}(i))$. Typically the literature on OWAs assumes that $\boldsymbol{\alpha}$ is normalized, i.e., $\sum_{1\leq i\leq K}\alpha_{i}=1$. We do not enforce this convention as we wish to study the PAV setting with $\alpha=(1,\nicefrac{{1}}{{2}},\ldots)$. This is formally a relaxation and we observe that whether or not the OWAs are normalized does not affect our computational results. However, we do require that our OWA vector be non-increasing and that each entry be $\geq 0$, i.e., for any $i,j\in|\boldsymbol{\alpha}|$, $i<j$ we have $\boldsymbol{\alpha}_{i}\geq\boldsymbol{\alpha}_{j}\geq 0$.

In our formulation, the OWA operator is applied to the vector of agent utilities and then we aggregate (or sum) these modified utilities to give the assignment objective. Hence, the $\Sigma$-OWA name. We observe that this formulation strictly generalizes the utilitarian assignment objective; if we set $\boldsymbol{\alpha}=(1)^{n}$ we recover the utilitarian assignment.

One may also wish to consider applying the OWA over the sorted vector of total agent utility for their allocation, which one could call the OWA-$\Sigma$ version of our problem. Indeed, this formulation of the problem has been considered before and proposed in the earliest writings on OWAs for decision making . Taking the OWA-$\Sigma$ formulation allows one to recover both the utilitarian assignment, $\boldsymbol{\alpha}=(\nicefrac{{1}}{{n}},\ldots,\nicefrac{{1}}{{n}})$, as well as the egalitarian assignment, $\boldsymbol{\alpha}=(0,\ldots,0_{n-1},1)$. However, because the OWA-$\Sigma$ formulation is a generalization of the egalitarian assignment, it becomes NP-hard in general .

We think of the $\alpha$ vector as a kind of control knob given to the implementer of the market, allowing them to apply a sub-linear transform to the agent utilities. This ability may be especially useful when agents are free to report their (normalized) utilities for ranks via bidding or other mechanisms . In many settings the utility vector is controlled by the individual agents, while the OWA vector is under the control of the market implementers. Consider the following example.

Consider a setting with four agents $N=\{a_{1},a_{2},a_{3},a_{4}\}$ agents and four objects $O=\{o_{1},o_{2},o_{3},o_{4}\}$. For all agents let $c^{N}_{min}=c^{N}_{max}=2$ and for all objects let $c^{O}_{min}=c^{O}_{max}=2$. For the $\Sigma$-OWA assignment, let $\alpha=(1,\nicefrac{{1}}{{2}})$.

We get the following allocations. Utilitarian: $A(a_{1},:)=\{o_{1},o_{2}\},u_{1}(A)=20$; $A(a_{2},:)=\{o_{1},o_{2}\},u_{2}(A)=16$; $A(a_{3},:)=\{o_{3},o_{4}\},u_{3}(A)=6$; $A(a_{4},:)=\{o_{3},o_{4}\},u_{4}(A)=8$; $\sum_{i}u_{i}(A)=50$.

OWA, $\alpha=(1,\nicefrac{{1}}{{2}})$: $A(a_{1},:)=\{o_{1},o_{2}\},u_{1}(A)=20$, $\alpha\cdot\boldsymbol{\omega}_{1}=15.5$; $A(a_{2},:)=\{o_{2},o_{3}\},u_{2}(A)=10$, $\alpha\cdot\boldsymbol{\omega}_{2}=9.0$; $A(a_{3},:)=\{o_{1},o_{4}\},u_{3}(A)=10$, $\alpha\cdot\boldsymbol{\omega}_{3}=8.5$; $A(a_{4},:)=\{o_{3},o_{4}\},u_{4}(A)=8$, $\alpha\cdot\boldsymbol{\omega}_{4}=5.0$; $\sum_{i}u_{i}(A)=48$.

Egalitarian: $A(a_{1},:)=\{o_{1},o_{4}\},u_{1}(A)=11$; $A(a_{2},:)=\{o_{2},o_{4}\},u_{2}(A)=10$; $A(a_{3},:)=\{o_{2},o_{3}\},u_{3}(A)=10$; $A(a_{4},:)=\{o_{1},o_{3}\},u_{4}(A)=10$; $\sum_{i}u_{i}(A)=41$.

Inspecting the results of Example 1, we observe that in the set of all utilitarian maximal assignments have $a_{1}$ and $a_{2}$ each being assigned to $o_{1}$ and $o_{2}$, in the set of all $\Sigma$-OWA maximal assignments $a_{3}$ is assigned one of $o_{1}$ or $o_{2}$ while $a_{2}$ is assigned one of $o_{3}$ or $o_{4}$, while in the set of all egalitarian maximal assignments each of the agents receives one of either $o_{1}$ or $o_{2}$ along with one of $o_{3}$ or $o_{4}$. Thus we observe the following.

The set of assignments returned by each of the three objective functions, utilitarian, egalitarian, and OWA, can be disjoint.

There are instances where the set of $\Sigma$-OWA assignments is the same as the set of egalitarian assignments, but disjoint from the set of utilitarian assignments. Hence, it is an interesting direction for future work to fully characterize $\Sigma$-OWA assignments and discover OWA vectors with nice properties.

An allocation $S$ is more preferred by a given agent with respect to pairwise comparisons than allocation $T$ if $S$ is a result of replacing an item in $T$ with a strictly more preferred item. Note that the pairwise comparison relation is transitive. An allocation is Pareto optimal with respect to pairwise comparisons if there exists no other allocation that each agent weakly prefers and at least one agent strictly prefers.

Consider an agent $i$ and two allocations $S$ and $T$ of equal size. Then if $S$ is at least as preferred as $T$ by $i$ with respect to pairwise comparison, then $S$ yields at least as much OWA value as $T$ for any OWA vector no matter if it is increasing or decreasing.

Note that $S$ can be viewed as a transformation from $T$ where each item $j$ is replaced by some other item $j^{\prime}$ that is at least as preferred. Hence, the value of the item either stays the same or increases. In either case, the corresponding OWA multiplied with the value is the same. Since the OWA transform is bilinear, the total OWA score of $S$ is at least as much as that of $T$.

The $\Sigma$-OWA maximal assignment is Pareto optimal with respect to pairwise comparison irrespective of the OWA.

Assume for contradiction that a $\Sigma$-OWA maximal assignment $A$ is not Pareto optimal with respect to pairwise comparisons. From Lemma 1, there exists another outcome $A^{\prime}$ that each agent weakly prefers and at least one agent strictly prefers. But this means that in $A^{\prime}$ each agent gets at least as much OWA score and at least one agent gets strictly more. But this contradicts the fact that $A$ is OWA maximal.

An Algorithm for Σ\Sigma-OWA assignments

We give an algorithm for finding $\Sigma$-OWA assignments using flow networks. In this proof we use the most general formulation of our problem by allowing the values of the upper and lower per-agent capacities, $[c^{N}_{min}(i),c^{N}_{max}(i)]$, to vary for each agent; and the upper and lower object capacities, $[c^{O}_{min}(j),c^{O}_{max}(j)]$, to vary for each object.

An $\Sigma$-OWA assignment can be found in polynomial time.

We reduce our problem to the problem of finding a minimum cost feasible flow in a graph with upper and lower capacities on the edges, which is a polynomial time solvable problem. In addition to being polynomial time solvable, we know that the flow is integral as long as all edge capacities are integral, even if we have real valued costs . Figures 1 and 2 provide a high level view of the flow network that we will construct.

In Figure 1 we first build a tripartite graph with two sets of nodes and one set of gadgets per agent: the agent nodes, one for each agent $a_{i}$; the agent gadgets, one (illustrated in Figure 2) for each agent $a_{i}$; and the object nodes, one for each object $o_{j}$. There is an edge from the source node $s$ to each of the agent nodes, each with cost 0, minimum flow capacity $c^{N}_{min}(i)$ and a maximum flow capacity $c^{N}_{max}(i)$. This set of edges and nodes enforces the constraint that each $a_{i}$ has capacity $[c^{N}_{min}(i),c^{N}_{max}(i)]$. We also construct an edge from each object node to the sink $t$. Each of these edges has a cost 0, a minimum capacity $c^{O}_{min}(j)$, and a maximum capacity $c^{O}_{max}(j)$. This set of edges enforces the constraint that each $o_{j}$ has capacity $[c^{O}_{min}(j),c^{O}_{max}(j)]$.

We now turn to the agent gadget depicted in Figure 2 for arbitrary $a_{i}$. The leftmost node and the rightmost set of nodes in Figure 2 correspond to the agent nodes $N$ and object nodes $O$ in Figure 1, respectively. In each agent gadget we create a tripartite sub-graph with the agent node $a_{i}$ serving as the source and the set of object nodes $O$ serving as the sinks.

We create three layers of nodes which we describe in turn from left to right. First, we create a set of decision nodes with labels $\alpha_{1},\ldots,\alpha_{d}$ where $c^{N}_{max}(i)\leq d\leq|\alpha|$. Intuitively, we will be multiplying the OWA value $\alpha_{1}$ by the utility for some object, so we need to keep track of all the values that could result. The arcs from $a_{i}$ to each of the nodes in this set has upper capacity 1, minimum capacity 0, and cost 0. If we have the case that $c^{N}_{max}(i)<d$ then we can set the maximum capacity of the edges to node(s) $\alpha_{j},j>c^{N}_{max}(i)$ to 0. This enforces that each value in the OWA vector can modify at most one utility value.

For each of the decision nodes $\alpha_{1},\ldots,\alpha_{d}$ constructed, we create a set of object/decision nodes for each $o_{j}$ which we denote $o_{j}\alpha_{k}$. From each of the decision nodes $\alpha_{1},\ldots,\alpha_{d}$ we create an edge to each of the object/decision nodes created for this particular decision node $\alpha_{k}$, i.e., $o_{1}\alpha_{1},o_{2}\alpha_{1},\ldots,o_{m}\alpha_{1}$ for $\alpha_{1}$. Each of these edges has maximum capacity 1 and a cost equal to $-1\cdot u_{i}(o_{j})\cdot\alpha_{1}$ for rank 1 and object $o_{j}\in O$. These costs are the (negative) cost that matching agent $a_{i}$ with object $o_{j}$ at weighted rank $d_{k}$ contributes to the OWA objective.

Finally, we create one set of agent/object nodes, one for each $o_{j}$ denoted $a_{i}o_{j}$. From all the object/decision nodes we connect all nodes with a label of $o_{j}$ to the corresponding agent/paper node, i.e., $o_{1}\alpha_{1},o_{1}\alpha_{2},\ldots o_{1}\alpha_{d}$ all connect to $a_{i}o_{1}$ with cost 0 and maximum capacity 1. We then connect the agent/object node to the corresponding object node in the main construction from $O$, i.e., $a_{i}o_{1}$ to $o_{1}$ with cost 0 and maximum capacity 1. This set of nodes and edges enforces that each agent can be assigned each object once.

We can extract an assignment from the minimum cost feasible flow by observing that paper $o_{j}$ is allocated to agent $a_{i}$ if and only if there is a unit of flow passing from the particular agent/object node $a_{i}o_{j}$ to the object node $o_{j}$. We now argue for the correctness of our algorithm in two steps, (1) that all constraints for the $\Sigma$-OWA assignment problem are enforced and (2) that a minimum cost feasible flow in the constructed graph gives an $\Sigma$-OWA assignment. For (1) we note that since the units of flow across the graph represent the assignment and we have explained how the capacity constraints on all edges enforce each of the particular constraints imposed by our definition of a feasible assignment, there is a feasible flow iff the flow satisfies the constraints.

For (2) observe that for each agent, the $\alpha$ nodes fill with flow in order from $\alpha_{1}$ to $\alpha_{d}$ as the OWA vector is non-increasing and the utilities are decreasing, i.e., for each agent, the edge costs monotonically increase from the edges associated with $\alpha_{1}$ to the edges associated with $\alpha_{d}$. Thus, for each agent, the first unit of flow to this agent will use the least cost (most negative) edge must be associated with $\alpha_{1}$; and similarly for $\alpha_{2}$ through $\alpha_{d}$. From the capacity constraints we know there is only one unit of flow that enters each decision node $\alpha_{i}$ and there is only one unit of flow that can leave each agent/paper node $a_{i}o_{j}$. This means that each $\alpha_{i}$ can modify only one $o_{j}$ and each $o_{j}$ selected must be unique for this agent.

As the decision nodes are filled in order and $\alpha_{i}$ can only modify the value for a single object, we know the total cost of the flow across the agent gadget for each $a_{i}$ is equal to $-1\cdot\boldsymbol{\alpha}_{i}\cdot\boldsymbol{\omega}_{i}$. Hence, the price of the min cost flow across all agents is equal to $-1\cdot\sum_{\forall i\in N}\boldsymbol{\alpha}_{i}\cdot\boldsymbol{\omega}_{i}(A)$. Thus, the min cost flow in the graph is an $\Sigma$-OWA assignment.

We observe two possible generalizations of the above construction which allow us to use this constructive proof for more general instances than the CPAP. First, The proof above can be generalized to allow for $\alpha$ to vary for each agent. Specifically, observe that the decision nodes for each agent $a_{i}$ are independent from all other agents. This means that, for each agent (or a class of agents) we could use an OWA vector $\alpha^{a_{i}}$. This ability may be useful, for instance, when a group of agents reports the same extreme utility distribution and the organizer wishes to apply the same transform to these utilities.

The second generalization that we can make to the above construction is to allow each agent to be assigned to each object more than once. While this ability does not make sense in the reviewers/papers setting (unless there are sub reviewers) there could be other capacitated assignment settings where we may wish to assign the agents to objects multiple times e.g., if there are discrete jobs that need to be done a certain number of times but and a single agent can be assigned the same job multiple times.

In order to generalize the capacity constraint from 1 for each agent $i$ for each object $j$ we introduce a capacity upper bound $z_{i,j}$ which encodes the number of times that agent $i$ can be assigned to object $j$. Taking $z_{i,j}=1$ for all $i$ and $j$ gives us the original CPAP setting. In order to enforce this constraint, within each agent gadget (Figure 2) we add a capacity constraint equal to $z_{i,j}$ from each edge $a_{i}o_{j}$ to $o_{j}$. If we want a lower bound for the number of copies of $o_{j}$ assigned to $a_{i}$ we can encode this lower bound on this edge as well.

We can extract an assignment from the minimum cost feasible flow by observing that paper $o_{j}$ is allocated to agent $a_{i}$ $z_{ij}$ times if and only if there are units of flow passing from the particular agent/object node $a_{i}o_{j}$ to the object node $o_{j}$. The argument for correctness follows exactly from the proof of Theorem 5.1 above.

An $\Sigma$-OWA assignment can be found in polynomial time even if each agent $a_{i}$ has a unique OWA vector $\alpha^{a_{i}}$ and each object $o_{j}$ can be assigned to each agent $a_{i}$ any number of times (not just once).

Experiments

We now turn to the question of how good are $\Sigma$-OWA assignments in practice? We answer this question using real world data from three large international conferences (MD-00002-00000001 – 00000003) from www.PrefLib.org . We focus discussion on MD-00002-00000003 which has 146 agents and 175 objects. We implemented the algorithm given in Section 5 using networkX for Python and Lemon for C++. However, we still have a run time $\approx O(V^{4})$, giving runtime $\approx(150^{2}\cdot 3)^{4})=2\times 10^{19}$, which caused our computers to crash even with 16GB of memory. This was quite disappointing as we thought the flow argument could be used to solve this problem on real-world instances.

Not to be deterred, we still wanted to investigate the assignments we get from $\Sigma$-OWA compare to the utilitarian and egalitarian assignments. Consequently, we implemented the model as an MIP in Gurobi 7.0 and it ran in under 1 minute for all instances and settings using 4 cores. Our MIP is similar to the one given by Skowron et al. 2016 and the MARA MIP by Bouveret et al. 2016. However, as we have capacity constraints and individual/variable length OWAs, our MIP is more general than either.

To encode the $\Sigma$-OWA problem we introduce a binary variable $x_{a,o}$ indicating that agent $a$ is assigned object $o$. We introduce a real valued variable $u_{owa,a}$ which is the $\Sigma$-OWA utility for agent $a$. Finally, we introduce $r_{a,o,p}$ for the OWA matrix which notes that agent $a$ is assigned object $o$ at OWA rank $p$. The MIP is given below.

Constraints (1)–(4) enforce the cardinality constraints on the agents, objects, and OWA rank matrix. Constraint (5) links the agent and object assignments to be positions in the OWA rank matrix. Line (6) enforces that the rank matrix fills from the first position to the $c^{N}_{max}$ position for each agent. And finally (7) enforces that the $\Sigma$-OWA value of the assignment positions in the rank matrix must be decreasing. We then maximize the sum over all agents of the OWA objective value.

We found the utilitarian, egalitarian, and $\Sigma$-OWA assignments for each of the real world datasets when each object must receive 3–4 reviews and each agent must review 6–7 objects. In the data, each agent sorts the papers into 4 equivalence classes which we gave utility values $(5,3,1,0)$. We use the PAV inspired decreasing harmonic OWA vector $(1,\nicefrac{{1}}{{2}},\nicefrac{{1}}{{3}},\ldots)$ to compute the $\Sigma$-OWA assignment.

One of the reasons we wanted to use the $\Sigma$-OWA assignment is to allow the market designer to enforce a more equitable distribution of papers with respect to the ranks. Hence, our test statistic is the number of top ranked items that the average agent can expect to receive. Figure 3 shows the agent counts and the cumulative distribution function (CDF) for the number of top ranked items the agents receive.

Looking at the left side of the figure, we see that 71 agents receive 5 top ranked papers under the $\Sigma$-OWA assignment while under the utilitarian assignment only 46 do. Under the utilitarian assignment 35 agents receive more than 5 top ranked papers. Consequently, on average, agents can expect to get 4 top ranked papers in the $\Sigma$-OWA assignment, 3 in the egalitarian assignment, and 4.2 in the utilitarian assignemnt. However under the utilitarian assignment, several agents receive an entire set of top ranked objects, while the egalitarian assignment modulates this so that most agents only receive 3–4 top ranked items. In contrast, the $\Sigma$-OWA assignment is a balance between these with the most agents receiving 5 top ranked items.

Conclusions

We have proposed and provided algorithms for the novel notion of a $\Sigma$-OWA assignment. The $\Sigma$-OWA assignment using decreasing OWA vectors gives the central organizer a “slider” to move from utility maximizing towards a more rank maximal assignment computationally efficient package. An important open question for future work is to find axiomatic characterizations for good OWA vectors. Additionally, the OWA method, and all methods for CPAP that we surveyed, treat objects as having positive utility. It is generally the case that reviewers at a conference want to review fewer, not more, papers. Consequently it would be interesting to study CPAP from the point of view of chores, as they are called in the economics literature.

References