Role: Developer, 3D Artist, Founding ML Team
Tools Used: Unreal Engine, C++, OpenCV, 3ds Max, JSON, Synthetic Data Pipelines
Focus: Digital Twin Simulation, Synthetic Dataset Generation, Real-Time AI Training Support
Project Overview
This project aimed to build a retail-focused AI system capable of detecting shopper product selections using in-store camera footage. The end goal was to train and validate deep learning models that could recognize which items a customer picks up and places in their cart—all without invasive tracking or sensors.
To accomplish this, the team developed a digital twin—a fully simulated retail environment—where shopper behavior could be replicated, synthetic data could be generated, and models could be trained before ever being deployed in the real world. This simulation-first approach allowed for fast iteration, controlled testing, and reduced real-world costs.
My Role & Contributions
I was a founding member of the machine learning development team, collaborating closely with ML engineers to build the entire synthetic data pipeline. My responsibilities included:
-
Digital Twin Development in Unreal Engine:
Constructed the retail simulation space in Unreal Engine, including layout, shopper logic, camera placements, and procedural item placement. -
3D Asset Creation:
Modeled and textured a wide range of store assets and products in 3ds Max, ensuring realism and consistency for synthetic dataset integrity. -
Lighting & Render Pipeline:
Designed performant, accurate lighting for both realism and consistency across renders to support high-quality AI training data. -
Synthetic Data Generation Pipeline:
Wrote C++ systems for automated labeling and synthetic dataset creation, generating structured JSON metadata compatible with downstream ML pipelines. -
OpenCV Integration:
Integrated OpenCV tools into the simulation to emulate the perspective of in-store cameras and align generated data with real-world conditions.
Problem Solving with Synthetic Data
During initial testing, object detection models failed to recognize items once placed inside shopping carts. Using the digital twin, I rapidly adjusted the simulation to include cart-based product placements and expanded the synthetic dataset accordingly. This retraining cycle allowed the team to fix the problem without needing new real-world data collection—saving significant time and cost.
Outcome
The updated model exceeded performance benchmarks and was validated in real-world testing. The digital twin approach proved critical in surfacing edge cases early and making fast corrections using controlled, synthetic data. This project highlighted the power of simulation-driven machine learning and showcased a scalable model for deploying AI in real retail environments.