Dataset creation tied to one developer,
now a pipeline that researchers run
I work as a research assistant on my Kyushu Institute of Technology lab's joint research with Hitachi. I rewrote from scratch the code that turns machine-vision camera captures into AI training data, and deployed it in a lab in the Netherlands in September and October 2026.
Architecture
The stages from the cameras to the training dataset. The project's code and data cannot be published, so only the stages are drawn.
- 01On site
-
- Others
- The hardware in the lab in the Netherlands was built by a junior researcher who took over the design.
-
- What I did
- Worked directly with the camera maker's Spinnaker SDK (PySpin) to connect several cameras, and made their settings adjustable in real time from the PyQt GUI while capturing.
- Equipment
- Several industrial machine-vision cameras (USB3)
- Stack
- Spinnaker SDK
- PySpin
- PyQt
-
- Others
- Preparing the samples and the reference labels was done by the project partner's researchers.
- Note
- What the samples are and what is measured are confidential to the joint research, so they are not described.
-
- 02Control and capture
-
- What I did
- Built a PyQt GUI with which non-developer researchers capture images and create datasets themselves, without a development environment, and shipped it as a PyInstaller executable. Camera settings are adjusted live from it, and every step runs from this one GUI.
- Stack
- PyQt
- PyInstaller
-
- What I did
- Wrote from scratch the module that controls several machine-vision cameras, matches their capture settings and saves the images.
- Stack
- Python
- PySpin
- Spinnaker SDK
-
- 03Measurement and analysis
-
- What I did
- A program that extracts values from the captures and analyzes them. On site, this program produced the AI training data.
- Stack
- Python
-
- 04Preprocessing
-
- What I did
- Cleaned the captured images and shaped them for training. I replaced the old code stage by stage, checking that the numbers matched on the same inputs.
- Stack
- Python
-
- 05Output
-
- Contents
- An AI training dataset of sample images and reference labels.
- Note
- The research topic and what is measured are confidential to the joint research and are not disclosed.
-
Context
The project produces AI training data from images taken with industrial machine-vision cameras. The code I inherited in April 2025 could not be maintained. At first I fixed only what broke, but every change in capture conditions broke something else, and every dataset needed a developer sitting next to it.
I concluded the problem was the structure, not the symptoms. The core of it: creating a dataset was tied to one developer.
Key decisions
Why I chose to rewrite, and how I kept the results from changing.
Stop patching and rewrite the whole thing
- Why
- Each fix broke something else, and every patch tied dataset creation more tightly to a developer.
- How
- I rewrote everything from camera control and capture to dataset preprocessing as one Python pipeline.
Replace it step by step against the old results
- Why
- A full rewrite risked changing the results without anyone noticing.
- How
- On the same inputs, I checked that the new code's numbers matched the old code's, and replaced it stage by stage.
A GUI that non-developers run themselves
- Why
- Researchers are the ones who make datasets, but only a developer could run the code.
- How
- I built a PyQt GUI and shipped it as a PyInstaller executable, so researchers create datasets themselves without a development environment.
Hand over the design intent before the code
- Why
- The project had to keep going without me.
- How
- I walked two juniors through why each stage is split the way it is, and a junior built the hardware on site.
Scope
The engineering that was mine, and the parts others handled on site.
What I built
- Rewrote the Python pipeline that runs from camera control and capture to preprocessing the training dataset
- Control of several machine-vision cameras and standardized capture settings (Spinnaker SDK, PySpin)
- The measurement and analysis program
- A PyQt GUI for non-developer researchers, shipped as a PyInstaller executable
- Design handover to two juniors, on-site deployment and the demo
What others did
- A junior researcher · building the hardware in the lab on site
- The project partner's researchers · preparing samples and reference labels
- Hitachi staff · using the program themselves on site
On-site deployment in the Netherlands (Sep–Oct 2026)
- Apr 2025Inherited the existing codeIt could not be maintained. After getting by with partial fixes, I decided on a full rewrite.
- Pipeline and GUI rewrittenReplaced stage by stage, checking the numbers against the old results. Design handed over to two juniors.
- Sep–Oct 2026Preparing the lab on siteOur lab and Hitachi staff were given a lab on site and worked there together. A junior built the hardware.
- Producing real dataThe project partner's researchers prepared the samples and reference labels, and the measurement and analysis program turned the captures into AI training data. Hitachi staff used it themselves.
- Demo to the project partnerRated well both by the people doing the work and by the project partner. The work on site was in Japanese and English.
The research is confidential, so the topic, what is measured, the code and the data are not described. This page covers only the engineering I was responsible for.
- Python
- Spinnaker SDK
- PySpin
- PyQt
- PyInstaller