TensorFlow.js & PoseNet

Who said machine learning couldn’t be fun and interactive? With TensorFlow’s PoseNet, you can build all sorts of cool interactive applications that function based on real-live human interaction and body language. What is PoseNet and how do you use it? Let’s take a look!

TensorFlow

TensorFlow is an open-sourced machine learning library developed by Google, released in 2015. TensorFlow has a set of APIs that provide a number of machine learning solutions and features a faster compilation time than other similar libraries, like Keras or Touch. The tool was originally created to run large numerical computations and accepts input data in multidimensional arrays. Formally known as tensors! TensorFlow can handle a large set of data, making it super useful when estimating on large datasets.

TensorFlow is not a traditional program, in that all data is represented as a Flow Graph they are prepared as nodes. All nodes in the graph represent a mathematical operation and every edge a tensor. Finally, the graph is executed and the data is processed. Thanks to this process, it makes it easy to train your models quickly and efficiently.

Pose Estimation

Before we dive into how PoseNet works and what it is, we should have an understanding of what pose estimation is and how it works. It's simple! Pose estimation refers to any computer vision technology that can detect and estimate the human form in pictures, video, or a live stream. This type of software could estimate the points of your shoulders, arms, wrist, head, and more as you move through the provided media. It’s not estimating who is in the image, but where their body is placed, and the different points on the body.

Pose estimation has many applications in the world around us, whether you realize it or not! One example is exercise! If you’re a runner, it’s possible you’ve been to a running shoe store and been offered a test that examines how you run and offers shoes based on this estimation. How does this work? Using machine learning models and pose estimation, the computer and treadmill examine your pose as you run during the test and get information on how your foot rises and falls as you run. In the end, the data from the pose estimation is crunched and the machine estimates based on the result what type of shoe best suits your running style!

Exciting right? That’s only the beginning! There are so many other applications for pose estimation, and TensorFlow’s PoseNet API opens the door for any developer to try their hand at building their own pose estimation solutions.

What Is PoseNet?

As mentioned previously, PoseNet is a pose estimation API that’s apart of Google’s TensorFlow library. The system can be used to estimate a single pose or even multiple. Since PoseNet can detect either one or multiples poses, that means it has two separate algorithms that you can possibly work with. At the beginning of the pose detection, an image is run through the convolutional neural network, then either the single pose or multi-pose algorithm is ran to detect the poses.

PoseNet will then spit out a few different metrics. First, PoseNet will provide a pose object that features key points and a confidence level for each person detected in the photo and their pose estimation. PoseNet will also provide a pose confidence score. The score is set on a scale of 0.1–1.0 and can be used to filter out any responses that the system isn’t very confident about. You'll also be provided with Keypoints. Which are the coordinates and positions estimated for the 17 points on the body that PoseNet can detect, which include eyes, ears, nose, knees, elbows, and more. Along with the coordinates of these keypoints, you’ll receive a confidence score, also within the scale of 0.1–1.0.

There’s a lot you can do with all of this information in your Javascript application. Let’s take a look at how to get started today!

Getting Started

Ready to get started? First, run the following command in your terminal:

npm install @tensorflow-models/posenet

Then, you’ll want to import it into your program:

import * as posenet from '@tensorflow-models/posenet';
const net = await posenet.load();

You can also set it up via page bundle, like so:

<html>
  <body>
    <!-- Load TensorFlow.js -->
    <script src="https://unpkg.com/@tensorflow/tfjs"></script>
    <!-- Load Posenet -->
    <script src="https://unpkg.com/@tensorflow-models/posenet">
    </script>
    <script type="text/javascript">
      posenet.load().then(function(net) {
        // posenet model loaded
      });
    </script>
  </body>
</html>

Now, you’re all set! Have fun trying out PoseNet and be sure to share any cool projects you work on. Happy coding :)

Pose estimation | TensorFlow Lite

TensorFlow