After seeing the amazing work done by Angjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik on the End-to-end Recovery of Human Shape and Pose paper. I wanted to have a look if I could use OpenPose and HMR to implement a one click video to mocap solution. The script takes a video and outputs a separate maya file for each person in the video containing and animated HIK skeleton.

The HMR repository was slightly adjusted to work with python3 in a windows environment and matrices to be exported over multiple frames to support video. The number of joints exported by the HMR solution is insufficient to create a working HIK skeleton, a center hip and spine joint had to be estimated. The OpenPose util was updated to recognize the same person over multiple frames, and omit people if they are under a certain presence threshold.

For this demo the animation curves exported have been simplified to reduce initial jitter. As you can see it will not be a render ready solution but potentially a great tool to get an initial starting point or simply to be used as 3D reference.

Index

Installation

Clone repository:

git clone https://github.com/robertjoosten/video2mocap.git

In order for this tool to work a couple of 3rd party application will have to be installed. To make this an easy process bat files are located in the 3rdparty directory and can be used to download/extract and install the necessary applications.

  • Run 3rdparty/getFFmpeg.bat
    • Download and extract FFmpeg ( 20180630-9f0077c ).
  • Run 3rdparty/getOpenPose.bat
    • Download and extract OpenPose ( 1.3.0 ).
    • Download models for OpenPose
  • Run 3rdparty/getHMR.bat ( Requires python3 + pip3 accessible in PATH )
    • Pull the latest custom fork of the HMR repository.
    • Download models for HMR
    • Pip install requirements.txt

Usage

From the command line:

cd video2mocap/
python video2mocap.py --video_path <VIDEO> --output_dir <OUTPUT>

Available Arguments:

  • –video_path: Path to video
  • –output_dir: Directory to output the maya files too
  • –keep_temp: Keep temp files for debugging ( False by default )
  • –mayapy_exe: Overwrite mayapy.exe ( Default requires accessible in PATH )
  • –python2_exe: Overwrite python.exe ( Default requires accessible in PATH )
  • –python3_exe: Overwrite python3.exe ( Default requires accessible in PATH )

The exe files can be overwritten in case the python interpreters are not accessible through the PATH variable and a relative path cannot be provided.

Limitations

  • No camera tracking ( static camera advised )
  • No partial body ( full body in view each frame advised )
  • No ankle/wrist and head rotation
  • Limited depth adjustment

Logs

As loads of things are running in process it is quite simple for something to go wrong. For this reason a log file is implemented that gets saved into the output_dir. If the desired result is unexpected these logs can be investigated to find out what and where something went wrong.

Keypoint matching example:

2018/07/05/ 12:31:40 | INFO | ---- Match Keypoints Over Multiple Frames ----
2018/07/05/ 12:31:40 | DEBUG | New Person:            Frame 276
2018/07/05/ 12:31:40 | DEBUG | Omit Person:           Presence Percentage 0.01

Credits

Versions

Citations