MotionCapture with iPad und iPhone

Design of avatars for an app for motion-capturing and face-capturing of movements via iPad and iPhone.

In the context of the application, the body movement and facial expressions of a person were to be transferred to virtual 3D figures (avatars). In this system, an iPad captures the body movement (motion capture) and an iPhone attached in front of the head captures the facial expressions (face tracking). The data was further processed in an external Apple computer with ARKit and transferred to the 3D figures (avatars) in real time. In order to get as close as possible to the two people used in the client’s key visual, we used image material from a photo shoot to design the figures.
In the next step, the “skeleton” of the figure responsible for the body movement had to be created according to special specifications defined by Apple. For the facial expressions, about 50-60 different “facial expressions” (actually individual movements of parts of the face such as left eye closed, upper lip raised, left corner of the mouth pulled down, etc.) had to be created. ARKit interpreted the videos recorded by the iPhone using an AI (actually a program part created by machine learning or deep learning) and recreated the facial expressions on the virtual character’s face. Since the system mixed dozens of the blendshapes together even for quite simple facial expressions, the challenge was to design them in such a way that the final result did not produce distorted or completely exaggerated poses.

Project details: 

Content: 2 characters with bones system and about 55 blendshapes for the face

Utilization: App for use at congresses

Specifications: Realization with Apple ARKit and iPhone and iPad

Client:  -nda-

The rights of use for the illustrations shown here lie with the client; use is not permitted. The images are protected with watermarks.

Screencapture IOS female avatar
Screencapture IOS male Avatar
Motion capture facial expressions iPhone
Motion capture of facial expressions with an iPhone
t-pose male character motioncapturing ios
Male “Avatar” in T-Pose
motioncapturing ipad head male avatar
Male “Avatar” head
female character t-pose arkit mocap
Female “Avatar” in T-Pose
mocap ipad head female character
Female “Avatar” Kopf
screenshot bone structure character
Bone structure character
motioncapture face blendshapes
Character with Apple-specified blendshapes/shapekeys/morphagets to control facial expression

More projects: