Apple ML Research: Training a model on Synthetic Data
During my internship at Apple, I was partnered with researchers at Carnegie Mellon University to help train a neural network to detect different features and attributes of iPhone.
View PaperThe Problem
One of the biggest issues I faced when training a neural network for my main project during my internship at Apple was getting the right amount of data to classify different iPhone features. Thus, I created a script in Blender's Python environment to render randomized images of different iPhone models for training.
Randomizing Attributes
To create a programmatic way of generating these renders, I created a Python script using Blender's API to randomize different environment features and product colors. This randomizes the product color, the model, and environmental placement on a custom background (typically a conveyor belt background). It can also adjust things like lighting, grain, noise, and other features that classification models have trouble with.
I used Blender's API, bpy, to first generate the scene, and worked on loading in specific models of iPhone provided by Apple. A simple script that scans through a specific directory was all that was needed, then once a phone was imported correctly, I selected the entire object and ensured that it was set at xyz(0, 0, 0).
Since object selection, especially for the mesh that was provided, is so difficult to keep track of, I worked on randomizing the position and tilt direction of the camera and lighting, to simulate different positions the phone is on. Randomizing the attributes to properly simulate all camera angles that will point at iPhones ended up being a challenge, especially with the field of view that we were working with, but with the hardest part finished, all I needed to do was take all my data and train a ML Model off of it.
Model Training
I ended up training a smaller model on my 2019 Macbook Pro 16 inch, provided by Apple, and it took almost a full day of training for only a single Epoch. I noticed that 2 epochs weren't nearly enough, so I ended up going through Apple to get a VM spun up for model training, and started out with 5 epochs, as a preliminary test. I got some real world training data, and the split between real and synthetic data being trained on was for every 1 real image of an iPhone, there would be 5 syntheticly generated iPhone images.
This ended up being the issue, as not only did we end up getting a pretty good model split, but it was extremely accurate. However, my 7 month internship was coming to an end, and I sadly had to wrap everything up, and move on.
Project Outcomes
The program I wrote ultimately aided CMU and Apple researchers for developing classification models quicker, which helps fulfill a need for a finer-tune and more accurate classifier when it comes to recycling Apple products. Training classifiers on products that look fairly similar already requires a lot of training data, and with this, it became a whole lot easier to automate and generate realistic synthetic data quickly.