Google Research: Cardiometabolic Risk from Smartphone Photos (PhotoScan)
The Daily Diff by Premchand Chidipoti
Episode notes
A health-AI piece with a clever engineering core. Jordan and Riley cover PhotoScan, a deep-learning framework that estimates 3D body-composition metrics — body fat %, android-to-gynoid (apple vs. pear) fat ratio, and visceral-to-subcutaneous fat ratio — from ordinary 2D smartphone photos, to flag insulin resistance (which precedes type 2 diabetes by years and is poorly captured by BMI). The standout trick solves a data problem: pre-train a ResNet-50 (ImageNet-init) on UK Biobank (N=35,323) using 2D projections rendered from 3D MRI with DXA as ground truth, fuse image features with sex/height/weight/BMI, and output probability density functions (uncertainty, not point guesses); then fine-tune on real smartphone photos (PhotoBIA, N=677, with landmark detection picking best frames from 360-degree video) and validate on an independent cohort ...