Calorie-Expending Machine
As part of the Build18 Hackathon hosted annually at CMU, my team and I designed and implemented a vending machine that accepts calories burned as a currency! We used tensorflow's Movenet to estimate the number of calories burned by a customer, who is asked to perform any physical movement in order to reduce the price of their snack. The overarching goal was to encourage physical exercise through a monetary and tasty reward and hopefully improve the health and lifestyle choices of customers who interact with the machine.
As perception lead, I developed the pose estimation and calorie estimation software that would be used to calculate the discount for the customer's snack. To do so, I initially tried to use CMU's own OpenPose package to estimate pose for each frame. However, inference proved too computationally expensive to compute locally on a Raspberry Pi in real-time. Instead, I switched to using a tensorflowlite Movenet Lightning model, which improved response time significantly and made local inference possible (though still resulting in a choppy live-vide feed of around 3-4 fps).
For calorie estimation, I initially hoped to use an LSTM to interpret a sequence of frames in a video feed and estimate calories based on some ground truth gathered from biometric sensors (e.g. smart watch calorie tracking). However, This would prove difficult, especially in gathering enough labeled data to train the model on any arbitrary human movement. Instead, I opted for a geometric approach that computed the kinetic energy of the customer based on the velocity of each tracked point on their body and an estimate of the mass of each point. This resulted in a reasonably accurate measure of energy consumption (given a still camera), which was then converted to some discount using an arbitrary scaling factor to vary the duration of exercise needed to reach the maximum discount.
Combined with the mechanical, electrical, and UI work of my team, we were able to build a fully functioning prototype with the choice of two snacks, a platform that lowers corresponding to the number of calories burned, and an easy to use UI. This UI had simple yet clear indicator for discount, calories burned, and position change prompts, and was well integrated with the pose estimation code to provide a seamless and fun user experience.
After a week's worth of work, our team won the Officer's Choice Award at the CMU Build18 Hackathon!
Check out our GitHub here.
