Built and ran the data layer
of a public R&D platform
On the “AI-based public R&D value-creation platform” (later SMART K2C), commissioned by KISTI (Korea Institute of Science and Technology Information) and designed by KISTI researchers, I handled collection, loading and serving as the data engineer on Vision Inside's side. My part ran from the collection setup for shopping and procurement data to turning analysis models into FastAPI services and admin screens.
Architecture
The path data takes from the sources to the users. Click a node to highlight the path through it and see the work at that stage and who did it.
- 01Sources
-
- What I did
- Selenium crawlers collected it subcategory by subcategory, and the products were de-duplicated by product ID (ASIN).
- Later
- Moved to the Rainforest API.
- Stack
- Selenium
- Rainforest API
-
- What I did
- Selenium crawlers collected it category by category.
- Stack
- Selenium
- BeautifulSoup
-
- What I did
- Collected data from the Public Procurement Service API on the public data portal and crawled the national procurement site.
- Stack
- REST API
- Python
-
- 02Collection
-
- What I did
- Built and ran a collection setup that split the crawlers across several laptops and loaded the results into one remote database.
- Why
- The crawlers were spread out to avoid being blocked by the sites they collected from.
- Stack
- Python
- Selenium
- BeautifulSoup
- MySQL
-
- What I did
- Built the collector for the Public Procurement Service API on the public data portal. Moved Amazon collection to the Rainforest API.
- Stack
- Python
- REST API
-
- 03Storage
-
- What I did
- Loaded the procurement, Danawa and Amazon data into a central MySQL database.
- Stack
- MySQL
- SQLAlchemy
- pandas
-
- 04Processing
-
- What I did
- Model 3: keyword matching between NTIS projects and companies (SQL). Model 4: price-band K-means clustering (number of clusters chosen by the Dunn index). Model 5: a daily batch that mines required-quality keywords from reviews (KeyBERT, Sentence-Transformers, Mecab). SQL statistics on procurement prices, attributes and deliveries.
- Stack
- SQL
- scikit-learn
- KeyBERT
- Sentence-Transformers
- Mecab
-
- Others
- Some of the platform's models were designed by university research teams (Sungkyunkwan University, Kookmin University). Vision Inside owned the data and serving layers those models use.
-
- 05Serving
-
- What I did
- Served the analysis models as FastAPI services and built the admin screens (filters, sequential batch runs, unified search, Excel export).
- Stack
- FastAPI
- Jinja2
- Tabulator
-
- What I did
- Owned the DB schema, backend integration and frontend wiring of the Java Spring prototype web app. Moved it, with a partner's models 1–2, to company servers and set up its Docker runtime.
- Worked with
- Built together with Vision Inside's in-house Java team; models 1–2 came from a partner company.
- Stack
- Java
- Spring
- MySQL
- Docker
-
- 06Users
-
- Others
- KISTI commissioned the project, and KISTI researchers designed it.
- What I did
- Presented results in regular technical meetings with KISTI researchers and handled data and code requests directly. Wrote the audit deliverables (operator manual, initial data-build report).
-
Key decisions
The rule used to clean the data.
De-duplication by product ID (ASIN)
- Problem
- The same product appeared in several categories at once, so collecting category by category produced overlaps.
- Result
- Products with the same ASIN were merged into one, removing the overlaps between categories.
Scope
Not the whole project: the parts that were mine, listed separately.
What I built and ran
- Selenium crawlers for Amazon and Danawa, a collector for the Public Procurement Service API on the public data portal, and crawling of the national procurement site
- A distributed crawling farm of laptops feeding one remote database, with de-duplication
- Loading into central MySQL (procurement, Danawa and Amazon data)
- Models 3–5, built by me: project–company keyword matching (SQL), price-band K-means (Dunn index), a daily KeyBERT review-keyword batch. Procurement statistics in SQL
- FastAPI services and Jinja2 and Tabulator admin screens (filters, sequential runs, unified search, Excel export)
- Moving Amazon collection to the Rainforest API
- Owned the DB schema, backend integration and frontend wiring of the KISTI prototype web app in Java Spring (with the in-house Java team); moved it, with a partner's models 1–2, to company servers and set up its Docker runtime
- Audit deliverables (operator manual, initial data-build report), presenting results in technical meetings, and handling data and code requests
What others did
- KISTI researchers · commissioning and designing the project
- University research teams (Sungkyunkwan University, Kookmin University) · algorithm design for some of the platform's models
- A partner company · models 1–2
- The in-house Java team · built the Java Spring prototype web app with me
- Another colleague · meeting minutes and reporting
Progress log
The order the work went in; only the start and end are dated.
- Aug 2022Kickoff · regular technical meetings with KISTI researchersPresented results and handled data and code requests directly.
- Amazon product collectionCollected by subcategory and de-duplicated by ASIN.
- Audit deliverablesOperator manual and initial data-build report.
- Distributed crawling farm in operationSeveral laptops loading into one remote database.
- KISTI prototype web app · moved in-houseOwned the DB schema, backend integration and frontend wiring of the Java Spring KISTI prototype web app, with the in-house Java team; moved it with the partner's models 1–2 to company servers on Docker.
- Rainforest API switch builtThe Amazon collection path replaced.
- Procurement statistics in SQLPrices, attributes and deliveries.
- Jun 2024Contract endedThe project led to a follow-on main contract.
What is public
The code and data belong to the company and the client, so they are not public.
- Python
- SQL
- Selenium
- BeautifulSoup
- pandas
- SQLAlchemy
- MySQL
- scikit-learn
- KeyBERT
- Sentence-Transformers
- Mecab
- FastAPI
- Jinja2
- Tabulator
- Java · Spring
- Docker
- Linux