Automated Scoring for Axe Throwing

Summary

I helped the Lumberjaxes axe-throwing company in Pittsburgh, PA develop an automated scoring system for axe throws with mentorship from Dr. George Kantor. In axe throwing, there are three rings that make up a bullseye, and the deepest ring that an axe lies in determines the score. Working from inside to outer ring, scores are 5, 3, 1, and 0 where 0 score is given if the axe does not lie in any ring. Having an automated scoring system for a static bullseye is not useful since people will need to pry their axe off a target and can calculate the score while doing so with no extra effort. However, this project is a foundation for more interesting games with moving targets and multiple throws where score-keeping would grow complicated.

Data Collection + Detection

My first step was to implement axe detection (not scoring) and set up a system to take a picture whenever an axe appears on the target. This serves the dual purpose of collecting images for the dataset as well as detecting on the real system. Axe detection was accomplished by using the SSIM algorithm, which essentially compares two different images based on luminance, contrast, and structure. Then, if similarity is within some bound (different enough to not be identical images, but similar enough to not be caused by a person moving into view), the image is saved. With all these images, I manually parsed through all the positives and negatives and hand-labeled the bounding box of the axe tip for every positive image. This makes up the axe dataset. Future work could include negative hard mining, a CNN-based axe detector, and the inclusion of edge-case images where the axe is still visible, but part of the image is occluded by a person:

Ring Detection

The next component is ring detection, which involves classical copmuter vision. Each ring has a distinct color, so I tuned basic RGB color filters to isolate pixels of each ring. Since filtering isn't perfect, I then implemented RANSAC to extract the best fit circle for each ring. As long as one ring is detected, the other rings can be approximated using hard-coded offsets for radii, which works since our camera is assumed to a fixed distance from the target. Relevant code can be found in this directory.

Axe Scoring

I tried multiple methods for axe tip detection (Blob detection + PCA, Fully Connected Network (FCN)) before finalizing on MaskRCNN, a pixel-level segmentation CNN. The Blob detection would often fail due to shadows and noise in the image, while precisely detecting the axe tip location with PCA did not work. The FCN was a naive modification of a previous homework assignment since I had no in-depth knowledge of Deep Learning at the time. MaskRCNN served well to handle the complexities in lighting, shadows, and size of the axes.

Results

The above figure compares performance between (top) MaskRCNN and (bottom) classical Blob Detection + PCA. A confusion matrix plots true scores vs predicted scores from the model/algorithm. Ideally, all counts or probability should lie on the diagonal from top left to bottom right. This means the true score matches the predicted score. The left plot is simply the normalized version of the right plot, showing percentages instead of counts, where each row should sum to 1. As a refresher, here are the definitions of precision and recall:

• Precision = True Positives / Predicted Positives
• Recall = True Positives / Actual Positives

The above only has mentioned software. For hardware, I used a Raspberry Pi 4 Model B since this is the only model that supports Tensorflow for MaskRCNN. Average runtime for axe detection was 1.2s, while axe scoring was 11s. This long runtime for scoring is unacceptable since we want scoring done in a matter of 2 to 3 seconds, but is left as future work. Choice of macro lens and camera can be found here.

Mechanical Design

I designed a custom mount system with adjustable angles for the camera. The below pictures show different angles of the final design and orientation of the camera with respect to the bullseye target. More pictures can be found here. Please contact Dr. George Kantor for access to the CAD files.

The red lines above represent a rubber band that create tension force and lock the teeth into place.