Hyeongju Lee 이형주

Built and ran the data layer
of a public R&D platform

On the “AI-based public R&D value-creation platform” (later SMART K2C), commissioned by KISTI (Korea Institute of Science and Technology Information) and designed by KISTI researchers, I handled collection, loading and serving as the data engineer on Vision Inside's side. My part ran from the collection setup for shopping and procurement data to turning analysis models into FastAPI services and admin screens.

Period
Aug 2022 – Jun 2024At Vision Inside · outsourced contract
Role
Data engineerCollection, loading and serving were mine
Design
KISTI researchersSome models by university research teams
Outcome
Led to a follow-on main contractIncluding the audit deliverables

Architecture

The path data takes from the sources to the users. Click a node to highlight the path through it and see the work at that stage and who did it.

Built by me Built together Others External sources and users
  1. 01Sources
    • What I did
      Selenium crawlers collected it subcategory by subcategory, and the products were de-duplicated by product ID (ASIN).
      Later
      Moved to the Rainforest API.
      Stack
      • Selenium
      • Rainforest API
    • What I did
      Selenium crawlers collected it category by category.
      Stack
      • Selenium
      • BeautifulSoup
    • What I did
      Collected data from the Public Procurement Service API on the public data portal and crawled the national procurement site.
      Stack
      • REST API
      • Python
  2. 02Collection
    • What I did
      Built and ran a collection setup that split the crawlers across several laptops and loaded the results into one remote database.
      Why
      The crawlers were spread out to avoid being blocked by the sites they collected from.
      Stack
      • Python
      • Selenium
      • BeautifulSoup
      • MySQL
    • What I did
      Built the collector for the Public Procurement Service API on the public data portal. Moved Amazon collection to the Rainforest API.
      Stack
      • Python
      • REST API
  3. 03Storage
    • What I did
      Loaded the procurement, Danawa and Amazon data into a central MySQL database.
      Stack
      • MySQL
      • SQLAlchemy
      • pandas
  4. 04Processing
    • What I did
      Model 3: keyword matching between NTIS projects and companies (SQL). Model 4: price-band K-means clustering (number of clusters chosen by the Dunn index). Model 5: a daily batch that mines required-quality keywords from reviews (KeyBERT, Sentence-Transformers, Mecab). SQL statistics on procurement prices, attributes and deliveries.
      Stack
      • SQL
      • scikit-learn
      • KeyBERT
      • Sentence-Transformers
      • Mecab
    • Others
      Some of the platform's models were designed by university research teams (Sungkyunkwan University, Kookmin University). Vision Inside owned the data and serving layers those models use.
  5. 05Serving
    • What I did
      Served the analysis models as FastAPI services and built the admin screens (filters, sequential batch runs, unified search, Excel export).
      Stack
      • FastAPI
      • Jinja2
      • Tabulator
    • What I did
      Owned the DB schema, backend integration and frontend wiring of the Java Spring prototype web app. Moved it, with a partner's models 1–2, to company servers and set up its Docker runtime.
      Worked with
      Built together with Vision Inside's in-house Java team; models 1–2 came from a partner company.
      Stack
      • Java
      • Spring
      • MySQL
      • Docker
  6. 06Users
    • Others
      KISTI commissioned the project, and KISTI researchers designed it.
      What I did
      Presented results in regular technical meetings with KISTI researchers and handled data and code requests directly. Wrote the audit deliverables (operator manual, initial data-build report).
ResultThe project led to a follow-on main contractThe company's code cannot be published, so only the structure is drawn

Key decisions

The rule used to clean the data.

  1. De-duplication by product ID (ASIN)

    Problem
    The same product appeared in several categories at once, so collecting category by category produced overlaps.
    Result
    Products with the same ASIN were merged into one, removing the overlaps between categories.

Scope

Not the whole project: the parts that were mine, listed separately.

What I built and ran

  • Selenium crawlers for Amazon and Danawa, a collector for the Public Procurement Service API on the public data portal, and crawling of the national procurement site
  • A distributed crawling farm of laptops feeding one remote database, with de-duplication
  • Loading into central MySQL (procurement, Danawa and Amazon data)
  • Models 3–5, built by me: project–company keyword matching (SQL), price-band K-means (Dunn index), a daily KeyBERT review-keyword batch. Procurement statistics in SQL
  • FastAPI services and Jinja2 and Tabulator admin screens (filters, sequential runs, unified search, Excel export)
  • Moving Amazon collection to the Rainforest API
  • Owned the DB schema, backend integration and frontend wiring of the KISTI prototype web app in Java Spring (with the in-house Java team); moved it, with a partner's models 1–2, to company servers and set up its Docker runtime
  • Audit deliverables (operator manual, initial data-build report), presenting results in technical meetings, and handling data and code requests

What others did

  • KISTI researchers · commissioning and designing the project
  • University research teams (Sungkyunkwan University, Kookmin University) · algorithm design for some of the platform's models
  • A partner company · models 1–2
  • The in-house Java team · built the Java Spring prototype web app with me
  • Another colleague · meeting minutes and reporting

Progress log

The order the work went in; only the start and end are dated.

  1. Aug 2022
    Kickoff · regular technical meetings with KISTI researchersPresented results and handled data and code requests directly.
  2. Amazon product collectionCollected by subcategory and de-duplicated by ASIN.
  3. Audit deliverablesOperator manual and initial data-build report.
  4. Distributed crawling farm in operationSeveral laptops loading into one remote database.
  5. KISTI prototype web app · moved in-houseOwned the DB schema, backend integration and frontend wiring of the Java Spring KISTI prototype web app, with the in-house Java team; moved it with the partner's models 1–2 to company servers on Docker.
  6. Rainforest API switch builtThe Amazon collection path replaced.
  7. Procurement statistics in SQLPrices, attributes and deliveries.
  8. Jun 2024
    Contract endedThe project led to a follow-on main contract.

What is public

The code and data belong to the company and the client, so they are not public.

  • Python
  • SQL
  • Selenium
  • BeautifulSoup
  • pandas
  • SQLAlchemy
  • MySQL
  • scikit-learn
  • KeyBERT
  • Sentence-Transformers
  • Mecab
  • FastAPI
  • Jinja2
  • Tabulator
  • Java · Spring
  • Docker
  • Linux
Contact

For roles and collaboration, please write by email.

I am looking for roles in data engineering, backend development and vision data pipelines.

Finishing my master's in March 2027