ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors

Introduction

Data-driven virtual 3D character animation has recently witnessed remarkable progress, and several neural methods modelling human motion have been proposed . The realism of virtual characters is a core contributing factor to the quality of computer animations and user experience in immersive applications like games, movies, and VR/AR. In such settings, a virtual character often interacts with different assets in the environment (e.g., pieces of furniture) and elaborate interactions usually require manual work of professional users. However, automatically synthesising natural animations with scene interaction constraints (as shown in Fig. 1) remains unsolved and challenging, especially when the virtual assets in the scene significantly differ from those captured during the data acquisition phase.

Approaches available in the literature enable human motion and performance capture , scene-agnostic animation , fine-grained grasping and manipulation synthesis ), interaction with dynamic objects , and inverse problems such as inferring object states from human interactions . In our context, several existing works on scene-aware character animation focus on specific interaction types such as sitting on a chair . Despite producing fairly natural animation for known objects, such methods struggle to generalise to unseen objects of the same category. Creating a generalisable motion model, which is capable of synthesising object-aware and realistic motions remains very challenging and unsolved. This is a major shortcoming due to the exponential number of possible asset-motion combinations. Naïvely training a generalisable neural network to synthesise sitting and lying motion would require collecting and annotating a large dataset of object-interaction motions with a large variety of chairs and couches. This is expensive and laborious. We believe all these challenges necessitate a fundamentally new approach to character-environment interaction modelling.

Hence, we propose ROAM, i.e., an alternative solution that works around the requirement of large and diverse datasets while also providing a complete framework for virtual character animation with object interaction; see Fig. 2 for an overview. ROAM is a robust and object-aware virtual character animation framework to synthesise human-object interactions; it generalises to unseen objects of the same category without relying on a large dataset of human-object animations. To this end, we follow a divide-and-conquer strategy to dataset collection, which utilises existing large-scale object-only dataset for goal pose generation and thus reduces the data demand for paired human-object interaction sequences. This affords us generalisability across instances of an object category while requiring character animation data with as few as one exemplar for an object category. We divide the problem into two subtasks: goal pose estimation and goal-driven motion synthesis.

Goal pose estimation is framed as a descriptor matching problem, which only requires a library of object shapes (such as ShapeNet or ObjaVerse ) and a desired reference pose. By designing $\mathsf{SE(3)}$-invariant neural descriptors , we encode the geometric relations for human-object interaction. This allows us to optimise the reference pose for an unseen object by minimising the descriptor distance between the reference human-object pair and the pose for the target object.

Once the goal pose for a previously unseen target object is optimised, our proposed l-NSM auto-regressively animates the character’s motion from a standing pose to the sitting and lying goal pose. Remarkably, l-NSM is trained using motion capture data with only a single exemplar object per category. This separation of tasks makes ROAM robust to variations in unseen object styles and scales while also avoiding expensive data capture overheads.

In summary, our technical contributions are as follows:

ROAM, a new approach for synthesising virtual character-object animations in 3D, which generalises to unseen objects of the same type while avoiding the capture of large motion-capture datasets;

A divide-and-conquer strategy simplifying the learning problem while also increasing the robustness of the method with respect to unseen object instances;

A novel, $\mathsf{SE(3)}$-equivariant neural descriptor objective to generate the goal pose and its integration into l-NSM, producing realistic motions towards the goal pose (Sec. 3.1).

To evaluate our design choices, we record a dataset of sitting motion on a reference chair and a reference sofa as well as a lying down motion on a reference sofa, which we will make publicly available for future research. Through a comprehensive set of experiments and perceptual study, we show the effectiveness of our method in terms of motion plausibility and robustness to unseen chair and sofa types.

Related Work

We now discuss two closely-related method classes, i.e., methods that generate static poses conditioned on the scene (Sec. 2.1) and methods supporting dynamic and scene-aware animations (Sec. 2.2). Tab. 1 provides a conceptual comparison of our method to previous works.

This setting requires generating a plausible 3D pose of a character, given the scene geometry. In this regard, Zhang et al. use a conditional Variational Auto-Encoder (cVAE) to estimate a pose conditioned on the latent space of the scene and further optimise it to achieve physical plausibility. Li et al. build a large-scale dataset with scene affordance and use it to train a 3D generative model, which plausibly places humans in a scene. POSA encodes contact probabilities between body mesh vertices and scene semantics. COIN synthesises compositional human-object interaction given object semantics and intended action while requiring extensive labelling of objects and interactions on the PROX dataset . In contrast to existing methods, we introduce a different paradigm for object-aware human pose generation. We also differ in that we are interested in synthesising dynamic motion sequences which can be controlled by a user.

2 Scene-aware Human Motion Generation

Dynamic, scene-aware motion generation requires a motion synthesis pipeline such that the characters perform a desired interaction with the scene. Towards this goal, Neural State Machine (NSM) synthesises periodic and non-periodic human motion with scene interactions. It encodes the environment and the objects using a coarse volumetric representation which can often lack robustness for unseen objects. However, it is not $\mathsf{SE(3)}$-equivariant and therefore requires the user to manually label the goal position and orientation for the character to interact with the object. SAMP improves upon NSM by predicting the goal location. However, it also uses the voxel-based scene representation, which limits the method’s robustness and generalisability to novel instances of the same object category. COUCH studies human–chair interaction by first sampling hand contact points on the chair using a VAE and then synthesising motion for the hands to reach the regressed target locations. This approach only allows partial variability and controllability as it provides no constraints to joints that are not in contact with the object. Moreover, it relies on a large dataset with labelled contact points and lacks generalisability to unseen objects. Wang et al. tackles long-duration motion synthesis by first generating intermediate static poses and then synthesising short in-between motions. Scene constraints such as collision avoidance and affordance are incorporated during post-processing, prohibiting real-time performance.

We argue that previous methods on 3D human pose and motion generation either tend to be bound to object instances close to the ones seen during training or rely on large and expensive data collections . In contrast, our method does not rely on extensive motion capture sequences with diverse objects while still achieving robust motion synthesis.

Method

Our goal is to synthesise 3D motion sequences of a character approaching and interacting with chairs and couches of various geometries, scales, and orientations. To learn such motions, we restrict ourselves to using motion-capture data with only one reference exemplar object per category; chairs and couches are considered separate categories. The overall schema of our method is shown in Fig. 2. We propose a divide-and-conquer strategy consisting of two main stages: First, our Goal Pose Synthesis (GPS) module takes a reference pose and an unseen target object and then generates an adapted goal pose, which fits the shape of the target object. We do so using our novel Skeletal Descriptor Fields (Sec. 3.1) which help the optimiser discover correspondences between the reference and the target object. With the goal pose synthesised, we need to seamlessly integrate it into our motion synthesis pipeline for low-level control. This is facilitated by our second component, lightweight Neural State Machine(l-NSM), that synthesises a natural walking motion approaching the object and interacting with it from the starting pose towards the final goal pose (Sec. 3.2). Finally, we propose an improved bidirectional blending scheme that seamlessly integrates the l-NSM motions with the goal pose.

The key motivation behind this two-stage design is that each task in separation can be trained with less data. More precisely, the goal pose synthesis solely requires an object dataset and l-NSM only requires motion-capture data paired with as few as one object per category.

The main question we seek to address in this section is the following: Given a person’s sitting pose on a chair of known geometry, how should this pose be adapted to a target chair with a previously unknown and different geometry? The target geometry could differ in several aspects, such as the height, type of armrest, or back support. Given a reference pose, $P$, of a character sitting or lying on the reference chair or sofa, $\mathcal{O}$, our goal is to optimise the target pose, $P{{}^{\prime}}$, for a previously unseen target object $\mathcal{O}^{\prime}$. We formulate this as a descriptor-matching problem on 3D points around the object, which allows us to reduce the dataset size significantly. Specifically, we train a neural network to estimate an $\mathsf{SE(3)}$-equivariant descriptor field around a given 3D object . $\mathsf{SE(3)}$-equivariance allows us to find a plausible orientation of the goal pose that does not penetrate the back of the chair. Once trained, we can construct a descriptor, $\mathbf{Z}(\mathcal{O},{P})$, for the skeleton by aggregating the learned point descriptors on the 3D keypoint locations of the reference pose. These descriptors encode the spatial relationship between the character pose and object geometry. This is used to finally optimise for $P^{\prime}$ by minimising the discrepancy between $\mathbf{Z}(\mathcal{O},P)$ and $\mathbf{Z}(\mathcal{O}^{\prime},P^{\prime})$. It is worth noting that such an approach does not require any manual labelling at inference but only a goal reference pose which can be conveniently sampled from the dataset or selected by the user, in contrast to prior work .

As shown in Fig. 3, $\Psi$ consists of a PointNet encoder $\Phi$ and an occupancy network $\Omega$. $\Phi$ encodes the object $\mathcal{O}$ into a compact latent code that informs the occupancy network of the object geometry. $\Omega$ is conditioned on the object’s latent code and a query point, $\mathbf{x}$, which allow it to predict the occupancy flag at the query point:

Since NDF’s architecture is based on VectorNeurons , the descriptor is by design $\mathsf{SE(3)}$-equivariant. As the $\Omega$ network is trained to estimate the occupancy values, the intermediate features implicitly model object-centric boundary information and spatial relationships between the points. Further, since $\Psi$ is a coordinate-based MLP, it allows us to extract these object-related features at an arbitrary point in space. To obtain the descriptor for a point $\mathbf{x}$, the activations from $L$ layers of $\Omega$ are concatenated so that it contains hierarchical features:

By matching the point descriptors in two different objects, one can find correspondences between them.

However, this descriptor, in the described form, only enables to rigidly match a coordinate frame at $\mathbf{x}$ from a source object to a target one, whereas we are interested in transferring the skeletal pose interacting with a source object to a target one. Therefore, we introduce our skeletal pose descriptor (SPD) in the following.

1.2 Skeletal Pose Descriptor (SPD)

Our core idea is to aggregate a neural descriptor $\Psi$ by querying all kinematic joints, which can be defined as

where $\mathcal{F}(\cdot)$ is the differentiable forward kinematics function that outputs the set of 3D joint positions for a given pose. However, constructing $\mathbf{Z}$ by querying $\Psi$ at only the 3D joint positions makes it sensitive to the descriptor’s quality at that specific point. To make it robust to such outliers, we also sample the query points from the neighbourhood of each joint position. Specifically, the points are sampled from a Gaussian distribution with standard deviation $\sigma$ centred at the joint positions $\mathcal{F}(P)_{j}$:

Note that our descriptor $\mathbf{Z}$ is now a function of the skeletal pose $P$. Moreover, as each joint descriptor $\mathbf{z}_{\mathbf{x}}$ is $\mathsf{SE(3)}$–equivariant, it follows that our pose descriptor $\mathbf{Z}$ is $\mathsf{SE(3)}$–equivariant. In practice, this means that our method is robust to any rigid transform applied to the object, which is in contrast to previous work that assumes orientation and translation of the object are known in advance.

1.3 Goal Pose Optimisation

So far, we constructed a descriptor that is robust to rigid transforms and shape variations. However, ultimately we are interested in matching a pose $P$ from a source object $\mathcal{O}$ to an unseen target object $\mathcal{O}^{\prime}$. For this, we use the 3D skeleton pose descriptors $\mathbf{Z}$ defined above to find the optimal pose $P^{*}$ for $\mathcal{O}^{\prime}$. Recall that our forward kinematics function $\mathcal{F}(P)$ in (5) is differentiable, thereby allowing any gradients from the descriptor to also back-propagate into $P$.

We, thus, formulate pose optimisation as an energy minimisation problem with the underlying postulate that the descriptors encode the geometric relationship between the pose $P$ and the object point cloud $\mathcal{O}$:

$\omega_{j}$ is a weight assigned to each joint descriptor. For more details, please refer to Sec. C in the supplementary material.

Although the descriptor matching term, $E_{d}$, acts as the main data term driving the optimisation, it is limited in that it does not encode the human-body specific priors in it. After all, the descriptor is based on an occupancy network trained on a dataset of chairs and couches. Therefore, we introduce additional regularisation terms to keep $P^{*}$ from drifting away from the manifold of plausible poses. Concretely, we introduce an angle-limit constraint ($E_{a}$), a foot-floor penetration regulariser ($E_{f}$), and a pose regulariser ($E_{p}$). Our overall objective function is defined as:

Angle Constraints. The SPD term, $E_{d}$ can result in unnatural twists between joints when source and target geometries vary significantly. Similar to Shimada et al. , we penalise joint angles that are beyond a fixed bound:

$P^{\prime}_{j}$ is the joint rotation (Euler angles) and $A_{j,max},A_{j,min}$ denote the upper and lower bounds of a joint rotation.

Floor Penetration. The floor penetration term penalises the penetration distance from the ground plane to all joints that are below the ground and is formulated as:

$\mathcal{F}(P)_{j}.y$ denotes the y-magnitude of joint $j$ and $\mathcal{G}_{y}$ is the height of the ground.

Pose Regularisation. Note that for symmetric objects, there can be more than one location with similar descriptor features. For example, consider a left-right symmetric chair. The features on the right leg of the chair can be very similar to the features on the left leg due to similar geometric patterns and spatial relationship with the rest of the chair. Without further constraints, using an L1-loss in isolation would give us non-unique goal poses, which fall inside the local minima of the energy landscape. Additionally, we observe the optimisation may converge to an implausible sitting pose if the reference pose is very challenging (see Fig. 6-(c)). Thus, we introduce a pose regularisation term $E_{r}$ below that ensures that the target pose does not deviate drastically from the reference pose in the canonical space:

2 Interactive Motion Generation

After having obtained $P^{*}$, we synthesise transitions from a start (standing) pose to the goal pose. To that end, we use a motion generation network which autoregressively synthesises idling, walking, sitting and transitioning motion between the three actions. Inspired by Starke et al. , we propose a lightweight Neural State Machine (l-NSM). While NSM defines interaction goal as a 6D vector encoding the goal position and orientation of the character’s root, such an approach does not scale to our setting, which requires a full-body, 75DoF goal pose. Furthermore, we handle goal pose conditional object interaction (sitting down or lying down) and non-interaction locomotion without goal pose (walking or idling) in a single framework. Finally, since our goal pose is already object-aware, we do not use the environment and interaction sensors, thereby making the l-NSM lighter. As a result, our l-NSM consists of the frame encoder and the goal encoder.

Recall that our goal pose, $P^{*}$ is automatically computed by the GPS module. This requires us to define a per-joint goal coordinate system where the origin lies at the joint’s position, which we call the goal-joint coordinate system during object interaction. The orientation is computed by recovering the normal of the torso-plane of $\mathbf{P}^{*}$. As a result, the proposed Bidirectional Pose Blending elegantly generates seamless object interaction and locomotion. It is worth noting that we do so with minimal overheads to already large input and output dimensions of l-NSM. Once the goal pose is computed, l-NSM synthesises motion in real time and allows both low-level interactive control and high-level goal-conditioned motion synthesis.

Experiments

We compare our ROAM to its most related approaches, i.e, NSM , SAMP and COUCH and evaluate it in terms of the animation realism and the plausibility of the synthesised sitting poses. Due to different skeleton definitions, types of interaction actions and number of reference objects, we retrain the models on our dataset. Please refer to the supplementary material Sec. A for more details. NSM needs to be provided with a point of contact labels and the object orientations; COUCH assumes a predefined chair orientation), while we do not make any of such assumptions. All the methods are provided a character’s starting pose and orientation along with the point cloud of the chair in the scene. All of them receive the same initial state of the character and the environment. The evaluation settings and their corresponding metrics are discussed next. For detailed visual comparisons with the state-of-the-art methods, the reader is referred to the supplementary video.

Recall that NSM, COUCH and SAMP animations are conditioned on the points of contact alone and often do not adapt well to object geometries. Such cases are hard to numerically account for, and hence, necessitate a perceptual user study to evaluate the quality of motion synthesis. The participants are asked to compare $12$ randomly generated animation sequences in a forced-choice manner, i.e, the participant must choose the best motion sequence out of the presented sequences. We ask the participants the following two questions for each set of motions: Which motion do you prefer in terms of the naturalness of the motion? (Realism) and Which motion do you prefer in terms of the agreement of the sitting motion with the chair geometry? (Semantics)

Results: We received responses from $50$ participants, leading to $600$ valid comparisons in total ($200$ per competing method); Fig 4 summarises the results. Our method is preferred over NSM, COUCH and SAMP by a significant margin. We calculate $p$-values (binomial test) for our comparison with each method and observe statistically significant results with $p{<}0.001$ for all three methods. Interestingly, NSM’s motion generation performs better than the other two more recent methods in terms of user preference. It is worth noting that NSM chairs have been hand-labelled for the goal position whereas our motion synthesis relies on the automatically generated goal pose.

2 Quantitative Results

We next perform quantitative analysis to evaluate our design choices. However, establishing a firm evaluation metric for our setting is challenging as no single one would account for all aspects.

Goal Distance Error. We evaluate whether the motion synthesis module reaches the intended target by measuring the Position Error (PE) to the goal. The comparison is done by using 100 unseen test objects with the character starting position randomly sampled between two to four meters to the goal object with a starting angle between -30°and 30°. We record the minimum goal error reached within 300 frames in Tab. 2. For comparison with NSM, we consider the manually labelled target position on the chair or couch as the goal position and compute the distance of the hip joint to it (Hip PE in Tab. 2). For SAMP, we use their generated hip positions and directions. Next, since COUCH facilitates reaching a goal pose based on the hand contacts, we use the generated hand contact positions and measure their distance from the hand positions in the synthesised sitting motion (Hand PE in Tab. 2). In the above setting, we choose the ‘hip’ joint and the ‘hand’ joints of ${P}^{*}$ as our target positions, respectively. We also present an ablation of our method without bidirectional blending, which proves to be inferior. Finally, we evaluate our full-body per-joint position error between the goal pose ${P}^{*}$ and the l-NSM output (Avg PE in Tab. 2).

3 Qualitative Results

In Fig. 5, we show how GPS adapts the reference pose to the target geometry for sitting on a chair and sofa and lying on a sofa.

Another interesting analysis is to measure the difference between the optimised novel pose and the reference pose as the target chair becomes more distant from the reference chair in terms of the Chamfer distance. To perform this analysis, we align the position and orientation of $100$ chairs in the ShapeNet dataset and use a single reference pose to generate goal poses for each chair. We then compute the difference between the reference and the target pose in terms of their root translation, root orientation and root-aligned 3D joint positions. The scatter plot in Fig. 7 follows the expected trend and we observe that the goal pose has to adjust significantly as the target chair’s geometry deviates from that of the reference chair.

4 Ablation

In Fig. 6, we present ablation visualisations of three design choices for two poses of different difficulty. In the first ablation, we discard the proposed sequential three-stage optimisation routine (see Sec. C in the supplementary material) and instead jointly optimise for the root orientation, root translation and root-relative articulations. We notice that doing so often leads to sub-optimal results because in the beginning of the optimisation, the reference pose can be in a significantly different orientation and the corresponding gradients are expected to be uninformative for optimising joint articulations. Next, we evaluate the effect of adding angle constraints ${E}_{a}$ in our loss function in Eq. 6. ${E}_{a}$ penalises implausible joint configuration and leads to more natural results, especially for challenging poses and out-of-distribution chairs. We also evaluate the choice of including the pose regulariser ${E}_{r}$ on the joint positions in the canonical space. This regulariser becomes important to ensure that the optimisation does not stuck in local minima and often leads to more natural poses. Finally, the rightmost column of Fig. 6 shows that the proposed design choices lead to improved results.

Discussion and Conclusion

We presented ROAM, a framework for robust synthesis of human-object interactions. Focusing on chairs and couches as an important instance arising in many applications, we demonstrated that the proposed two-step strategy of ROAM allows robust generalisation to a variety of unseen objects. Furthermore, we showed that the proposed bidirectional pose blending not only allows the motion synthesis pipeline to smoothly transition to the goal pose but also generates natural motion that the users found to be better than existing scene interaction methods. Crucially, we could achieve this by training with motion capture data involving only a single chair/couch exemplar. However, our method leads to artefacts when the target geometry differs significantly from the source geometry. We believe these artefacts can be removed by leveraging character mesh instead of the skeleton representation with physical plausibility constraints (see supplementary material for more discussion on future work). With the ever-growing demand for realistic 3D characters in virtual worlds, we envision our principled and practical approach to contribute to the creation of scalable immersive visual experiences.

Acknowledgement: This work was carried out as part of a dissertation at Saarland University. This project was also supported by Saarbrücken Research Center for Visual Computing, Interaction and AI. Christian Theobalt was supported by ERC Consolidator Grant 4DReply (770784). We thank Janis Sprenger for helpful discussions on experiment design and visualisation.

References

Appendix A Dataset

To train our approach, we recorded a new dataset of two subjects performing idling, standing and walking, and one subject interacting with objects: eg. sitting on a chair and a sofa and lying down on a sofa in a multi-view green screen studio. Notably, we only record motions for a single object per action category (one chair and one sofa for sitting, and one sofa for lying down). In total, we recorded around 90 minutes of motion at a frame rate of 25fps using 30 calibrated and synchronised RGB cameras. To recover the skeletal motion, we leverage a markerless motion capture software . Foot contact and phase labels are automatically annotated based on the altitude of the foot joints. We will release our dataset for future research.

Appendix B Robustness to Orientation Initialisations

We show the robustness of our goal pose synthesis (GPS) module to perturbations in initial orientations in Fig 8. As shown on the top left, our initial pose for optimisation is the same as the reference pose, with the root position shifted to the nearest chair point. Our method is robust even with poor initialisation (e.g., the initial pose is rotated 40°, 80°, 120°and 160°respectively). The SPD optimisation routine not only correctly optimises the pose orientation, but also adjusts the pose according to the height of the target chair.

Appendix C Implementation Details

Appendix D Quantitative Ablation

We now discuss the analysis of our method and compare it with its ablated versions. For quantitative evaluations, we perform the following ablations. Joint Sampling is our baseline where each query point cloud is centred at the exact joint positions. Bone Sampling refers to adding eight additional query point clouds from the midpoint of the limbs. Note that we do not add more midpoints along the spine since they are relatively denser than limb joints. Perturbation Sampling refers to adding noise to each joint query point cloud to lift the optimisation out of potential local minima. We first compute the pose plausibility by measuring the percentage of non-penetrating goal poses. We do it in the PyBullet physics simulator and plot the AUC curve of the percentage of non-penetrating poses against collision thresholds (Fig. 9(left)). The contact collision threshold defines the degree of allowed penetration in order to account for the inaccuracies of the body and object modelling in the simulator. We also evaluate if the non-penetrating sitting poses lead to a stable sitting posture by running physics simulations on a humanoid robot with the same DoF as in our skeleton. We let the humanoid simulate for $t=1s$ at a simulation rate of $240$Hz and measure the drift of the character before and after the simulation. Again, we plot an AUC curve of the per-joint drift for several thresholds on the maximum allowed drift (Fig. 9(right)).

We observe that sampling points along the bone leads to lesser penetration, but the success rate remains unaffected by the type of ablation. This is expected as sampling along the bone provides a larger coverage of the occupancy field around the chair, unlike the case where we sample around the 3D joints.

Appendix E Limitations and Future Work

While we have highlighted experimentally the generalisability and data efficiency of our approach, it is not free from limitations.

First, ROAM reasons on the level of a kinematic skeleton. Consequently, neither the human body shape nor the clothing is accounted for and we observe occasional character-object penetrations. Thus, a promising avenue for future work is to explicitly model the dense geometry of the virtual character and its deformable surface, from which both the goal pose estimation and the motion synthesis would benefit. Moreover, additional physics-based constraints could further boost the realism of our results (e.g., by mitigating foot sliding, enforcing contacts and penalising implausible poses as shown in (Fig. 10)). In addition, NDF for estimating the goal pose inherently exhibit only low-frequency variations. Therefore, only rather coarse-grained pose estimations can be performed (e.g., preventing hand articulations, which are, therefore, not part of our model). Furthermore, the quality and variability of our pose estimator are tightly linked to the NDF training dataset. Substantially out-of-distribution objects can, therefore, lead to unpredictable results. Note that we inherit these limitations from the original NDF formulation; any respective advances will directly benefit our method. In terms of motion generation, l-NSM is scene-agnostic and cannot avoid obstacles along the way. When an out-of-distribution goal pose is provided, the bidirectional pose blending scheme could lead to artefacts. Finally, we have illustrated the feasibility of our approach using only two actions (sitting and lying) on two categories (chairs and sofas). Yet, no component of our method is specifically designed for this particular combination and it could be trained for other combinations once other datasets are available in future.