KISTI-commissioned project · an AI-based platform for creating value from public R&D (later SMART K2C)
Project design KISTI researchers · Some model design commissioned university research · My part built and ran the data layer on Vision Inside's side (collection, loading, serving), and turned analysis models into FastAPI services and admin screens
- 01Sources
-
- What I did
- Selenium crawlers collected it subcategory by subcategory, and the products were de-duplicated by product ID (ASIN).
- Later
- Moved to the Rainforest API.
- Stack
- Selenium
- Rainforest API
-
- What I did
- Selenium crawlers collected it category by category.
- Stack
- Selenium
- BeautifulSoup
-
- What I did
- Collected data from the Public Procurement Service API on the public data portal and crawled the national procurement site.
- Stack
- REST API
- Python
-
- 02Collection
-
- What I did
- Built and ran a collection setup that split the crawlers across several laptops and loaded the results into one remote database.
- Why
- The crawlers were spread out to avoid being blocked by the sites they collected from.
- Stack
- Python
- Selenium
- BeautifulSoup
- MySQL
-
- What I did
- Built the collector for the Public Procurement Service API on the public data portal. Moved Amazon collection to the Rainforest API.
- Stack
- Python
- REST API
-
- 03Storage
-
- What I did
- Loaded the procurement, Danawa and Amazon data into a central MySQL database.
- Stack
- MySQL
- SQLAlchemy
- pandas
-
- 04Processing
-
- What I did
- Model 3: keyword matching between NTIS projects and companies (SQL). Model 4: price-band K-means clustering (number of clusters chosen by the Dunn index). Model 5: a daily batch that mines required-quality keywords from reviews (KeyBERT, Sentence-Transformers, Mecab). SQL statistics on procurement prices, attributes and deliveries.
- Stack
- SQL
- scikit-learn
- KeyBERT
- Sentence-Transformers
- Mecab
-
- Others
- Some of the platform's models were designed by university research teams (Sungkyunkwan University, Kookmin University). Vision Inside owned the data and serving layers those models use.
-
- 05Serving
-
- What I did
- Served the analysis models as FastAPI services and built the admin screens (filters, sequential batch runs, unified search, Excel export).
- Stack
- FastAPI
- Jinja2
- Tabulator
-
- What I did
- Owned the DB schema, backend integration and frontend wiring of the Java Spring prototype web app. Moved it, with a partner's models 1–2, to company servers and set up its Docker runtime.
- Worked with
- Built together with Vision Inside's in-house Java team; models 1–2 came from a partner company.
- Stack
- Java
- Spring
- MySQL
- Docker
-
- 06Users
-
- Others
- KISTI commissioned the project, and KISTI researchers designed it.
- What I did
- Presented results in regular technical meetings with KISTI researchers and handled data and code requests directly. Wrote the audit deliverables (operator manual, initial data-build report).
-
