Unity3D + Kinect, or animations on the knee

Introduction
Hello, dear reader. Before starting, I would like to talk a little about the purpose and format of my article. Typically, with such headings, the articles are for informational purposes, they describe the technical aspects of the implementation of a solution or provide instructions for use, etc. The reader makes conclusions independently. Before writing this article, I did a little research work on the subject of analogues. And I found several articles both on the hub, and a lot of video materials on YouTube. Therefore, I decided not to write another review of the technology, but to share my impressions, talk about the rake that we stepped on, and bring a little "kitchen" analytics.
The purpose of this article is to help you decide whether or not to use this method. First of all, it is designed for technical specialists, although it is not full of technical details. But it will also be useful to everyone else - art specialists, project managers, and simply to everyone who, in one way or another, is involved in game development. Perhaps the link to this article will help to knock out the budget from the authorities and speed up the development / improve the quality of your future game. So let's get started.

Background

Animation is often the "problem spot" of many 3D projects. Of course, it all depends on the specifics of the project, but often this specificity is determined precisely by the complexity and high cost of producing high-quality animation. Models, textures, sketches, effects, and even the interface are easier to find / order than a package of specific animations for the character of your game. Sometimes the game design is limited to the framework that the animation sets. This is most pronounced in indie projects and projects of small teams. Alas, sometimes we observe problems with animation with large developers. For example, a new game comes out in which everything seems to be fine: the gameplay is on top, the models are beautiful, the style is sustained, and even the effects seem to be nothing ... But the game looks dull, because the characters run as if they were seated or all the abilities in the game have three animations. Repeatedly, I witnessed the death of interesting projects that were never born, because the developers could not find resources for animation. As a result, there is an opinion among developers that high-quality and varied animation is expensive in terms of resources. And for money, and for time. Especially if we want realistic animation of humanoid characters. It was with the problem of animation that we encountered in one of our projects and want to share experience in solving it, which, perhaps, will be useful to you, dear readers. So, we had two problems.
Firstly, the conflict of desires and opportunities. I wanted a lot, quality and varied. Budget and resources were limited, and time was running out.
Secondly, experimental gameplay. In the process of development, some game concepts have changed, respectively, and the visualization has changed. I wanted to be able to quickly respond to possible changes or adding features. After a little research of options and possibilities, we thought about using Motion Capture technology, as it met all our needs. Quality, variety, the ability to change animations on the fly, etc. But the price of the technology was too high. On average, renting a Motion Capture studio in Moscow costs $ 800-1000. per day, cleaning the animation after recording 1–2 cu second. We began to look for alternatives ... and found - using Kinect + iPi Motion Capture Software.

Impressions and solutions

I will not describe the process of recording animation and go into technical details, as it is described in detail on the website of software developers (iPi Motion Capture Software) and in this article . Impressions of using this technology are more than positive. With minimal resources, we got a quality result. The whole process - from recording movements to importing results into Unity3D - takes an average of 1-3 hours depending on the duration of the recording (most of the time is an automatic process of recognition and post-processing). At the output, you get a ready-made fbx file that you simply import into Unity3D, and even without cleaning, you can get a quite acceptable result. And the participation of a professional animator is not required. There is no need for a special room and workwear. The toolkit provides the ability to both automatically "clean" the animation and manually fine-tune the process. The biggest plus is that you can export your own skeleton and associate it with the skeleton from the record. This allows you to see the future result on the fly and adjust it as necessary, and of course, export the final result, tailored to your needs.

Naturally, in the process of work, we filled up cones and made conclusions:
1. Knowledge is power - do not rush to record and export animation. Carefully read the examples and manuals, watch the video on YouTube. A complete understanding of the process will save you a lot of time and nerves.

2. From simple to complex - do not rush to record long and complex animations right away. Start with the simple ones. Thus, you will gradually get acquainted with the technology and gain experience in its use.

3. Calibration - when calibrating 2 Kinect-s, we did not immediately figure out how to properly configure the cameras, and got bad results for a couple of days. It is very important that after automatic calibration using a special camera recording in the software editor, they match the spatial arrangement that you used when recording the calibration video as closely as possible. This can be achieved by moving the cameras in the editor after calibration. Usually you have to do several iterations until the match is exact.

4. Prepare a place - although a special place is not required for recording, it is still worth preparing in advance. Please note that you need 4x4 meters of free space. It is also worth marking the position for the feet on the floor, for example with tape. This will help to maintain the correct position between motion recordings. Put a large mirror outside the camera visibility so that you can see yourself with it: this way you can control the accuracy of the movements.

5. Training and doubles - rehearse and memorize the movements that you plan to record in advance and write a few takes of each movement to select the best later.

6. Post-processing - the software provides good post-processing tools. Even if any bones were lost during recognition, you can move them and configure them directly in the editor. There are also tools to smooth out small jitter, etc. After several entries, we came to the conclusion that the revision is more convenient to do in the editor, since after export it’s not very convenient to set up animations in another editor. Although this item is rather subjective.

Summary

Using Kinect to record animations in practice shows itself as a quick, easy to use and inexpensive way to get high-quality animations for your games and more. The average price of the issue is 12–40 thousand rubles, depending on the chosen ipiSoft license and the number of Kinect s, which is not a big financial barrier not only for studios, but also for indie developers. Moreover,
if you consider how much time and resources will be saved when working with outsourcers or full-time animators, this approach turns out to be very profitable. In addition, you can try the technology absolutely free. A 30-day trial applies to all necessary software, and getting 1-2 Kinect is not a problem at all. Of course, Kinect is not a panacea for all problems. Of course, you cannot record animations
non-humanoid characters. And complex animation with sharp movements can’t be recorded. But technologies are developing, for example, the same Kinect - Demonstration of Kinect 2.0 capabilities - with such recognition, the quality of animations will be just great. The benefits of using Kinect for MotionCapture are obvious. Just have a try. I will be happy to answer any questions related to the topic of the article in the comments.

Links
Since laziness is an engine of progress, I give a list of useful links for those who are too lazy to google themselves:
1. ipiSoft Motion Capture - the software site itself, which has everything you need. Lessons, examples, software distributions, etc.

2. Exporting MoCap data to Unity3d is a useful article describing the process of calibrating, exporting animations, and post-processing.

3. A selection of videos on YouTube - many examples of how, what, where and why.

4. Demonstration of Kinect 2.0 features - demonstration of the second version of the controller. The prospects are very interesting.