Pose-aware Multi-level Feature Network for Human Object Interaction Detection

Introduction

Visual relations play an essential role in a deeper understanding of visual scenes, which usually requires reasoning beyond merely recognizing individual scene entities . Among different types of visual relations, human object interactions are ubiquitous in our visual environment and hence its inference is critical for many vision tasks, such as activity analysis , video understanding and visual question answering .

The task of human object interaction (HOI) detection aims to localize and classify triplets of human, object and relation from an input image. While deep neural networks have led to significant progresses in object and action recognition , it remains challenging to detect HOIs due to large variations of human-object appearance and spatial configurations, multiple co-existing relations and subtle differences between similar relations .

Most of existing works on HOI detection tackle the problem by reasoning interactions at the visual object level . The dominant approaches typically start from a set of human-object proposals, and extract visual features of human and object instances which are combined with their spatial cues (e.g., masks of proposals) to predict relation classes of those human-object pairs . Despite their encouraging results, such coarse-level reasoning suffers from several drawbacks when handling relatively complex relations. First, it is difficult to determine the relatedness of a human-object pair instance with an object-level representation due to lack of context cues, which can lead to erroneous association. In addition, many relation types are defined in terms of fine-grained actions, which are unlikely to be differentiated based on similar object-level features. For instance, it may require a set of detailed local features to tell the difference between ‘hold’ and ‘catch’ in sport scenes. Furthermore, as these methods largely rely on holistic features, the reasoning process of relations is a blackbox and hard to interpret.

In this work, we propose a new multi-level relation reasoning strategy to address the aforementioned limitations. Our main idea is to utilize estimated human pose to capture global spatial configuration of relations and as a guidance to extract local features at semantic part level for different HOIs. Such augmented representation enables us to incorporate interaction context, human-object and detailed semantic part cues into relation inference, and hence generate robust and fine-grained predictions with interpretable attentions. To this end, we perform relation reasoning at three distinct semantic levels for each human-object proposal: i) interaction, ii) visual objects, and iii) human parts. Fig. 1 illustrates an example of our relation reasoning.

Specifically, at the interaction level of a human-object proposal, we take a union region of the human and object instance, which encodes the context of the relation proposal, to produce an affinity score of the human-object pair. This score indicates how likely there exists a visual relation between the human-object pair and helps us to eliminate background proposals. For the visual object level, we adopt a common object-level representation as in but augmented by human pose, to encode human-object appearance and their relative positions. The main focus of our design is a new representation at the human part level, in which we use the estimated human pose to describe the detailed spatial and appearance cues of the human-object pair. To achieve this, we exploit the correlation between parts and relations to produce a part-level attention, which enable us to focus on sub-regions that are informative to each relation type. In addition, we compute the part locations relative to the object entity to encode a fine-level spatial configuration. Finally, we integrate the HOI cues from all three levels to predict the category of the human-object proposal.

We develop a multi-branch deep neural network to instantiate our multi-level relation reasoning, which consists of four main modules: a backbone module, a holistic module, a zoom-in module and a fusion module. Given an image, the backbone module computes its convolution feature map, and generates human-object proposals and spatial configurations. For each proposal, the holistic module integrates human, object and their union features, as well as an encoding of human pose and object location. The zoom-in module extracts the human part and object features, and produces a part-level attention from the pose layout to enhance relevant part cues. The fusion module combines the holistic and part-level representations to generate final scores for HOI categories. We refer to our model as Pose-aware Multi-level Feature Network (PMFNet). Given human-object proposals and pose estimation, our deep network is trained in an end-to-end fashion.

We conduct extensive evaluations on two public benchmarks V-COCO and HICO-DET, and outperform the current state-of-the-art by a sizable margin. To better understand our method, we also provide detailed ablative study of our deep network on the V-COCO dataset.

We propose a multi-level relation reasoning for human object interaction detection, in which we utilize human pose to capture global configuration and as an attention to extract detailed local appearance cues.

We develop a modularized network architecture for HOI prediction, which produces an interpretable output based on relation affinity and part attention.

Our approach achieves the state-of-the-art performance on both V-COCO and HICO-DET benchmarks.

Related Work

Visual Relationship Detection. Visual Relation Detection (VRD) aims at detecting the objects and describing their interactions simultaneously for a given image, which is a critical task for achieving visual scene understanding. Lu et al. propose to learn language priors from semantic word embedding to fine-tune visual relationships. Zhang et al. design a visual translation network to embed objects into low-dimension relation space for tackling visual relationship detection problem. Besides, Xu et al. model the visual relationship detection in structured scene as a graph, and propagate messages between objects. In our task, we focus on human-centric relationship detection, which aims to detect human-object interactions.

Human-Object Interaction Detection. Human-Object interaction (HOI) detection is essential for understanding human behaviors in a complex scene. In recent years, researchers have developed several human object interaction dataset, like V-COCO and HICO-DET . Early studies mainly focus on tackling HOIs recognition by utilizing multi-stream information, including human, object appearance, spatial information and human poses. In HO-RCNN , Chao et al. propose multi-stream to aggregate human, object and spatial configuration information to resolve HOIs detection tasks. Qi et al. propose graph parsing neural network (GPNN) to model the structured scene into a graph and propagate messages between each human and object node and classify all nodes and edges for their possible object classes and actions.

There have been several attempts that use human pose for recognizing fine-grained human related actions . Fang et al. exploit pair-wise human parts correlation to help to tackle HOIs detection. Li et al. explore interactiveness prior existing in multiple datasets and combine human pose and spatial configuration to form pose configuration map. However, those works only take human pose as a spatial constraint between human parts and object, but do not use it to extract zoom-in feature in each part, which provides more detail information for HOI task. By contrast, we take advantage of this fine-grained feature to capture subtle differences between similar interactions.

Attention Models. Attention mechanism has been proved very effective in various vision tasks, including image captioning , fine grained classification , pose estimation and action recognition. Attention mechanism can help highlight informative regions or parts and suppress some irrelevant global information. Xu et al. firstly utilize attention mechanism in image captioning and automatically attend some informative region in image relevant to generated sentences. Sharma et al. apply attention models realized by LSTM into action recognition task to learn important parts of video frames. Yu et al. propose a stacked semantic guided attention to focus on informative birds’ parts and suppress irrelevant global information. In our work, we focus on pose-aware attention for HOIs detection in human parts.

Method

We now introduce our multi-level relation reasoning strategy for human-object interaction detection. Our goal is to localize and recognize human-object interaction instances in an image. To this end, we augment object-level cues with human pose information and propose an expressive relation representation that captures the relation context, human-object and detailed local parts. We develop a multi-branch deep neural network, referred to as PMFNet, to learn such an HOI representation and predict categories of HOI instances. Below we first present an overview of our problem setup and method pipeline in Sec. 3.1, followed by the detailed description of our model architecture in Sec. 3.2. Finally, Sec. 3.3 outlines the model training procedure.

We adopt a hypothesize-and-classify strategy in which we first generate a set of human-object proposals and then predict their relation classes. In the proposal generation stage, we apply an object detector (e.g., Faster R-CNN ) to the input image and obtain a set of human proposals with detection scores $\{\langle\mathbf{x}_{h},s_{h}\rangle\}$, and object proposals with their categories and detection scores $\{\langle\mathbf{x}_{o},c_{o},s_{o}\rangle\}$. Our HOI proposals are generated by pairing up all the human and object proposals. In the relation classification stage, we first estimate a relation score $s_{h,o}^{a}$ for each interaction $a$ and a given $\langle\mathbf{x}_{h},\mathbf{x}_{o}\rangle$ pair. The relation score is then combined with the detection scores of relation entities (human and object) to produce the final HOI score $R_{h,o}^{a}$ for the tuple $\{\langle\mathbf{x}_{h},\mathbf{x}_{o},c_{o},a_{h,o}\rangle\}$ as follows,

where we adopt a soft score fusion by incorporating human score $s_{h}$ and object score $s_{o}$ at the same time, which represent detection quality of each proposal.

where the network $\mathcal{F}_{a}$ composes of four modules: a backbone module, a holistic module, a zoom-in module and a fusion module. Below we will describe the details of our model architecture.

2 Model Architecture

Our deep network, PMFNet, instantiates a multi-level relation reasoning with the following four modules: a) a backbone module computes image feature map and generates human-object proposals plus spatial configurations; b) a holistic module extracts object-level and context features of the proposals; c) a zoom-in module focuses on mining part-level features and interaction patterns between human parts and object; and d) a fusion module combines both object-level and part-level features to predict the interaction scores. An overview of our model is shown in Fig. 2.

We adopt ResNet-50-FPN as our convolutional network to generate feature map $\bm{\Gamma}$ with channel dimension of $D$. For proposal generation, we use Faster R-CNN as object detector to produce relation proposal pairs $\{\langle\mathbf{x}_{h},\mathbf{x}_{o}\rangle\}$. As mentioned earlier, we also compute human pose vector $\mathbf{p}_{h}$ for each human proposal $\mathbf{x}_{h}$ and take it as one of the inputs to our network.

In addition to the conv features, we also extract a set of geometric features to encode the spatial configuration of each human-object instance. We start with two binary masks of human and object proposal in their union space to capture object-level spatial configuration as in . Moreover, in order to capture fine-level spatial information of human parts and object, we add an additional pose map with predicted poses following . Specifically, we represent the estimated human pose as a line-graph in which all the joints are connected according to the skeleton configuration of COCO dataset. We rasterize the line-graph using a width of $w=3$ pixels and a set of intensity values ranging from 0.05 to 0.95 in a uniform interval to indicate different human parts. Finally, the binary masks and pose map in union space are rescaled to $M\times M$ and concatenated in channel-wise to generate a spatial configuration map.

2.2 Holistic Module

In order to capture object-level and relation context information, the holistic module is composed of four basic branches: human branch, object branch, union branch and spatial branch, illustrated in Fig. 3 (left). The input features of human, object and union branches are cropped from convolution feature map $\bm{\Gamma}$ by applying RoI-Align according to human proposal $\mathbf{x}_{h}$, object proposal $\mathbf{x}_{o}$ and their union proposal $\mathbf{x}_{u}$. $\mathbf{x}_{u}$ is defined as the minimum box in spatial region that contains both $\mathbf{x}_{h}$ and $\mathbf{x}_{o}$. Then human features, object features and union features are rescaled to $R_{h}\times R_{h}$ resolution. The input of spatial branch directly comes from spatial configuration map generated in Sec. 3.2.1. For each branch, two fully connected layers are adopted to embed the features to an output feature representation. We denote the output features of human, object, union and spatial feature as $f_{h}$, $f_{o}$, $f_{u},f_{s}$, and all the features are concatenated to obtain the final holistic feature $\Gamma_{hol}$:

where $\oplus$ denotes concatenation operation.

2.3 Zoom-in Module

While the holistic features provide coarse-level information for interactions, many interaction types are defined at a fine-grained level which require detailed local information of human part or object. Hence we design a zoom-in (ZI) module to zoom into human parts to extract part-level features. The overall zoom-in module can be viewed as a network that takes human pose, object proposal and convolution feature map as inputs and extract a set of local interaction features for the HOI relations:

Our zoom-in module, illustrated in Fig. 3 (right), consists of three components: i) A part-crop component that aims to extract fine-grained human parts features; ii) A spatial align component that assigns spatial information to human parts features; iii) A semantic attention component that enhances the human part features relevant to interaction and suppress irrelevant ones.

Our zoom-in module aims to extract fine-level features of local part regions and model the interaction patterns between human parts and objects. Many interactions have strong correlations with spatial configuration of human parts and object, which can be encoded by the relative locations between different human part and target object. For example, if the target object is close to ‘hand’, the interaction are more likely to be ‘hold’ or ‘carry’, and less likely to be ‘kick’ or ‘jump’. Based on this observation, we introduce the spatial offset of $x,y$ coordinates relative to object center as an additional spatial feature for each part.

In particular, we generate a coordinate map $\bm{\alpha}$ with the same spatial size as the convolution feature map $\bm{\Gamma}$. The map $\bm{\alpha}$ consists of two channels, indicating the $x$ and $y$ coordinates for each pixel in $\bm{\Gamma}$, and normalized by the object center. Then we apply RoI-Align for each human part $\mathbf{x}_{p_{k}}$ and object proposal $\mathbf{x}_{o}$ on $\bm{\alpha}$ and get the spatial map $\alpha_{k}$ for part $k$ and $\alpha_{o}$ for object. We concatenate the spatial map with the part-crop features so that for a $R_{p}\times R_{p}$ cropped part region, we align relative spatial offset to each pixel, which augments part features with a fine-grained spatial cues. The final $k$-th human part feature and object feature are :

As the pose representation also encodes the semantic class of human parts, which typically have strong correlations with interaction types (e.g., ‘eyes’ are important for ‘read’ a book). We thus predict a semantic attention using the same spatial configuration map from Sec. 3.2.1.

where $\beta_{k}\in$ is the $k$-th value of $\bm{\beta}$, $\odot$ indicates element-wise multiplication.

Finally, we concatenate human part features and object feature to obtain the attended part-level features $f_{att}$ and feed it to multiple fully-connected layers ($\mathcal{FC}$) to extract final local feature $\Gamma_{loc}$:

2.4 Fusion Module

In order to compute the score $s_{h,o}^{a}$ of pair $\langle\mathbf{x}_{h},\mathbf{x}_{o}\rangle$ for each interaction $a$, we employ a fusion module to fuse relation reasoning from different levels. Our fusion module aims to achieve the following two different goals. First, it uses the coarse-level features as a context cue to determine whether any relation exists for a human-object proposal. This allows us to suppress many background pairs and improve the detection precision. Concretely, we take the holistic feature $\Gamma_{hol}$ and feed it into a network branch consisting of a two-layer fully-connected network followed by a sigmoid function $\sigma$, which generates an interaction affinity score $s_{G}$:

Second, the fusion module use the object-level and part-level features to determine the relation score based on the fine-grained representation. Using a similar network branch, we compute a local relation score $s_{L}$ from all the relation features:

Finally, we fuse those two scores defined above to obtain the relation score for a human-object proposal $\langle\mathbf{x}_{h},\mathbf{x}_{o}\rangle$:

3 Model Learning

In training stage, we freeze ResNet-50 in our backbone module, and train FPN and other components in Sec. 3.2 in an end-to-end manner. Note that the object detector (Faster R-CNN ) and the pose estimator (CPN ) are external modules and thus do not participate in learning process.

Assume we have a training set of size $N$ with relation labels set $Y=\{\textbf{y}^{i}\}$ and interaction affinity label set $Z=\{z^{i}\}$, where $\textbf{y}^{i}=(y^{1,i},...,y^{A,i})\in\mathds{1}^{A}$ indicates ground truth relation label for $i$-th sample, and $z^{i}\in\{0,1\}$ indicates the relatedness of this sample, $i\in\{1,...,N\}$. We define $z^{i}=1$ if $\exists a\in\mathcal{A},y^{a,i}=1$, else $z^{i}=0$.

Suppose our predicted local relation scores are $S_{L}=\{\textbf{s}_{L}^{i}\}$ and affinity scores are $S_{G}=\{s_{G}^{i}\}$ for those samples, where $\textbf{s}_{L}^{i}=(s_{L}^{1,i},...,s_{L}^{A,i})$ indicates the predicted local scores of all interactions and $s_{G}^{i}$ is predicted interaction affinity score for $i$-th sample. As our classification task is actually a multi-label classification problem, we adopt a binary cross entropy loss for each relation class and interaction affinity. Let $L_{CrossEntropy}(a,b)=a\log(b)+(1-a)\log(1-b)$, the overall objective function for our training $L$ is defined as:

where $\mu$ is a hyperparameter to balance the relative importance of multi-label interaction prediction and binary interaction affinity prediction.

Experiments

In this section, we first describe the experimental setting and implementation details. We then evaluate our models with quantitative comparisons to the state-of-the-art approaches, followed by ablation studies to validate the components in our framework. Finally, we show several qualitative results to demonstrate the efficacy of our method.

We evaluate our method on two HOI benchmarks: V-COCO and HICO-DET . V-COCO is a subset of MS-COCO , including 10,346 images (2,533 for training, 2,867 for validation and 4,946 for test) and 16,199 human instances. Each person is annotated with binary labels for 26 action categories. HICO-DET consists of 47,776 images with more than 150K human-object pairs (38,118 images in training set and 9,658 in test set). It has 600 HOI categories over 80 object categories (as in MS-COCO ) and 117 unique verbs.

We follow the standard evaluation setting in and use mean average precision to measure the HOI detection performance. We consider an HOI detection as true positive when its predicted bounding boxes of both human and object overlap with the ground-truth bounding boxes with IOUs greater than $0.5$, and the HOI class prediction is correct.

2 Implementation Details

We use Faster R-CNN as object detector and CPN as pose estimator, which are pre-trained on the COCO train2017 split. Each human pose has a total of $K=17$ keypoints as in COCO dataset.

Our backbone module uses ResNet-50-FPN as feature extractor, and we crop RoI features from the highest resolution feature map in FPN . The size of our spatial configuration map $M$ is set to 64. The RoI-Align in holistic module has a resolution $R_{h}=7$, while in zoom-in module, the size of human parts is $\gamma=0.1$ of human box height and all the features are rescaled to $R_{p}=5$.

We freeze ResNet-50 backbone and train the parameters of FPN component. We use SGD optimizer for training with initial learning rate 4e-2, weight decay 1e-4, and momentum 0.9. The ratio of positive and negative samples is 1:3. For V-COCO , we reduce the learning rate to 4e-3 at iteration 24k, and stop training at iteration 48k. For HICO-DET , we reduce the learning rate to 4e-3 at iteration 250k and stop training at iteration 300k. During testing, we use object proposals from for fair comparison. See Suppl. Material for more details.

3 Quantitative Results

We compare our proposed framework with several existing approaches for evaluation. We take only human, object and union branches in holistic module as our baseline, while our final model integrates all the modules in Sec. 3.2.

For V-COCO dataset, we evaluate $AP_{role}$ of 24 actions with roles as in . As shown in Tab. 1, our baseline method achieves 48.6 mAP, outperforming all the existing approaches . Compared to those methods, our baseline adds a union region feature to capture context information, which turns out to be very effective for predicting interaction patterns in a small dataset like V-COCO. Moreover, our overall model achieves 52.0 mAP, which outperforms all the current state-of-the-art methods by a sizable margin, and further improves our baseline by 3.4 mAP.

For HICO-DET, we choose six current state-of-the-art methods for comparison. As shown in Tab. 2, our baseline still performs well and surpasses most existing works except . One potential reason is that HICO-DET dataset has a more fine-grained labeling of interactions (117 categories) than V-COCO (24 categories), and hence the object-level cue is insufficient to distinguish the subtle difference between similar interactions. In contrast, our full model achieves the-state-of-art performance with 17.46 mAP and 20.34 mAP on Default and Know Object categories respectively, outperforming all the existing works. In addition, it further improves our baseline by 2.54 mAP and 1.51 mAP on Default and Know Object modes respectively.

Furthermore, we divide the 600 HOI categories of the HICO-DET benchmark into two groups as in : Interactiveness (520 non-trivial HOI classes) and No-interaction (80 no-interaction classes for human and each of 80 object categories). We show the performance of our full model on those two groups compared with our baseline in Tab. 3. It is evident that our method achieves larger improvement on the Interactiveness group. As the No-interaction group consists of background classes only, this indicates that our pose-aware dynamic attention is more effective on the challenging task of fine-grained interaction classification.

4 Ablation Study

In this section, we perform several experiments to evaluate the effectiveness of our model components on V-COCO dataset (Tab. 4).

As in , we augment human and object binary masks with an additional human pose configuration map, which provides more detailed spatial information on human parts. This enriched SCM enables the network to filter out non-interactive human-object instances more effectively. As shown in Tab. 4, the SCM improves our baseline by 0.7 mAP.

Part-crop component zooms into semantic human parts and provides fine-grained feature representation of human body. Experiments in Tab. 4 show the effectiveness of zoom-in parts feature, which improves mAP from 49.9 to 51.0. We note that the following spatial align and semantic attention component are built on top of the part-crop component.

Spatial align component computes relative locations of all the parts w.r.t. the object and integrates them into the part features, which captures a ‘normalized’ local context. We observe a significant improvement from 51.0 to 52.4 in Tab. 4.

The semantic attention focuses on informative human parts and suppress other irrelevant ones. Its part-wise attention scores provides an interpretable feature for our predictions. As shown in Tab. 4, SeAtten slightly improves the preformance by 0.3 mAP.

Similar to , the interaction affinity indicates whether a human-object pair have interaction, and can reduce false positives by lowering their interaction scores. We can observe from Tab. 4 that IA improves performance by 0.3 mAP.

We further perform a drop-one-out ablation study which all independent components are removed individually, shown in tab. 4. The results demonstrate that each independent component indeed contribute to our final performance.

5 Qualitative Visualization Results

Fig. 4 shows our HOIs detection results compared with the baseline approach. We can see that our framework is capable of detecting difficult HOIs where the target objects are very small and generates a more confident score. This suggests that part-level features provide more informative visual cues for difficult human-object interaction pairs.

Fig. 5 visualizes semantic attention on a variety of HOI cases, each of which provides an interpretable outcome for our predictions. The highlighted joint regions indicate that our Semantic Attention (SeAtten) component generates an attention score higher than 0.7 for the related keypoint. In Fig. 5(a), for the same person interacting with various target objects, our SeAtten component is capable of automatically focusing on different human parts that are strongly related to interaction action. As the two images in up-left show, when the child interacting with chair, SeAtten will concentrate on the full body joints; while he interacting with an instrument, SeAtten will focus on his hands. To validate the generalization capacity of the SeAtten component, we also visualize several other HOI examples in Fig.5(b). For different persons with various interactions, our SeAtten component can always produce meaningful highlight on human parts which are relevant to each interaction type.

Conclusion

In this paper, we have developed an effective multi-level reasoning approach to human-object interaction detection. Our method is capable of incorporating interaction level, visual object level and human parts level features under the guidance of human pose information. As a result, it is able to recognize visual relations with subtle differences. We present a multi-branch deep neural network to instantiate our core idea of multi-level reasoning. Moreover, we introduce a semantic part-based attention mechanism at the part level to automatically extract relevant human parts for each interaction instance. The visualization of our attention map produces an interpretable output for the human-object relation detection task. Finally, we achieve the state-of-the-art performances on both V-COCO and HICO-DET benchmarks, and outperform other approaches by a large margin on V-COCO dataset.

Acknowledgement

This work was supported by Shanghai NSF Grant (No. 18ZR1425100) and NSFC Grant (No. 61703195).

References