# Project tagging — spec

Deterministic topic / language / framework tags for the D2 project topics in
`forms/extracted/`. Machine implementation: `tag_projects.py`; outputs are
committed under `forms/extracted/`.

The tagger does **not** call a model. Every tag comes from a keyword rule or a
normalized stack value, and every assignment carries its evidence
(`project_tags.json`), so results are reproducible and reviewable.

## 1. Tag axes

| Axis | Column | Answers | Vocabulary |
|---|---|---|---|
| Topic | `topics` | What is the project *about*? | closed, curated (25 slugs, §3) |
| Language | `lang_tags` | What human-facing programming language / markup / query / shell is used? | aliases over the source `languages` values (§4) |
| Framework | `framework_tags` | What frameworks, libraries, runtimes, platforms, databases, tools, protocols, standards are used? | aliases + passthrough over the source `frameworks` values (§5) |

Semantics:

- **Topics are aspects, not exclusive classes.** A project may carry several
  (`ai` + `computer-vision` + `healthcare`). `ai` is an umbrella; vision/NLP
  projects get the umbrella *and* the specific topic.
- **A canonical tech token lands on exactly one axis**, decided by its class,
  not by which source column mentioned it. `Node.js` listed under *languages*
  is a `runtime` → it appears only in `framework_tags`. `ReactJS` listed under
  *languages* → `framework_tags`. `HTML`/`CSS`/`SQL`/`YAML` stay on the
  language axis with classes `markup`/`query`/`dsl`.
- **Tags are per project, not per company**: every D2 project row is tagged
  from its own title, timeline and stack cells.

## 2. Identity, sources, duplicates

- Source of truth for taggable content: `forms/extracted/d2.json`
  (`fields.projects[]`). `d3.json` is an acceptance letter — no projects.
- `projects.csv` has one row per (form file × project slot); the same project
  often appears twice (`.docx` and signed `.pdf` copy of one form). Rows are
  joined back with `(source_file, no)`.
- `project_key = sha1(company_id + "\x1f" + normalized-title)[:12]` collapses
  the copies into one canonical project in `project_tags.json` (581 vs 959
  rows). Normalized title = lowercase, whitespace-collapsed, trailing `.`
  stripped. Empty titles fall back to their company+slot.
- Copies occasionally disagree (OCR variants like `tập`/`tệp`, reworded
  titles across revisions). Tags are computed per copy; near-duplicate
  revisions therefore can differ in a tag. This is visible, not silent.

## 3. Topic vocabulary (25)

| slug | meaning / dominant triggers |
|---|---|
| `web` | web apps, sites, portals, APIs, front/back-end (`web`, `website`, `portal`, `api`, React/Node/… stack) |
| `mobile` | Android/iOS/Flutter/React Native apps, phone features |
| `game` | game dev/design/QA/live ops, Unity, Unreal, Roblox, `trò chơi`, `gameplay` |
| `ai` | umbrella: ML, LLM, agents, chatbots, RAG, prediction, classification, training |
| `computer-vision` | vision/image/OCR/camera: `thị giác máy`, `xử lý ảnh`, OCR, detection, segmentation, YOLO/OpenCV |
| `nlp` | language/speech: `ngôn ngữ tự nhiên`, text, voice/TTS/ASR, translation, chatbot text |
| `data` | data engineering/analytics/BI: pipelines, ETL, dashboards, warehouses, SQL/Spark/BI stack |
| `cloud-devops` | cloud, infra, Docker/K8s, CI/CD, servers, monitoring, IaC |
| `security` | AppSec/pentest/SOC/forensics/malware/crypto: `bảo mật`, `an toàn thông tin`, OWASP, MITRE |
| `networking` | networks, 5G/LTE, Wi-Fi, mesh/BLE, routing, protocols, ICT solutions |
| `embedded` | embedded/firmware/MCU/RTOS/HMI nhúng, STM32/ESP32, drivers, boards |
| `iot` | IoT/IIoT/AIoT, smart devices, sensors, remote monitoring, MQTT |
| `chip-design` | IC/RTL/Verilog/ASIC/FPGA/SoC, physical design, STA, EDA flow, verification |
| `robotics` | robots, UAV/drone, AMR, SLAM, autonomous navigation, ROS/Gazebo |
| `automotive` | automotive software: ECU/ADAS/CAN/AUTOSAR, in-vehicle systems |
| `healthcare` | medical/EMR/clinical: `y tế`, `sức khỏe`, `bệnh nhân`, HL7/FHIR, diagnosis* |
| `education` | education/e-learning/LMS/courses/lectures, university systems |
| `ecommerce` | e-commerce/retail/POS/orders/delivery/storefront |
| `fintech` | banking/payments/insurance/finance/securities/accounting/loans |
| `enterprise-systems` | ERP/CRM/HRM, internal tools, workflow, HR, recruitment, ticketing, customer care |
| `testing-qa` | QA/test automation/coverage/verification as the *project subject* |
| `blockchain` | blockchain/smart contracts/web3/traceability |
| `ar-vr` | AR/VR, virtual/augmented reality, digital twin, 3D simulation |
| `gis` | GIS/geospatial/maps/coordinates, ArcGIS/postgis |
| `industrial` | factories, production lines, SCADA/PLC/CNC, power/water plants, machinery |

\* `chẩn đoán` requires medical context (`bệnh`/`y tế`/…) so it does not fire
on software debug/diagnosis.

## 4. Normalization rules

### Stack values (languages + frameworks)

1. **Alias table** (`STACK_VOCAB` in `tag_projects.py`): ~230 entries of
   `slug → class → [regex patterns]`, e.g. `nodejs ← Node.js|NodeJS|node js`,
   `postgresql ← postgres(ql)?|psql`, `dotnet ← .NET|ASP.NET|.NET Core|…`.
2. Patterns are matched **longest-first with span masking**, so
   `react native` beats `react`, `sql server` beats `sql`, `verilog hdl`
   is consumed before `verilog`. One pattern may emit several slugs
   (`django rest framework → django + django-rest-framework`).
3. **Word guards** are applied automatically to alphanumeric-edged patterns,
   so `unity` cannot match inside `community`, `excel` inside `excellent`,
   `c` inside `C#`/`C++`. Intentional prefix matches are written explicitly
   (`unity\w*`, `stm32\w*`, `sqlite\d?`).
4. Composites split naturally: `C/C++ → c, cpp`; `Java/Kotlin → java, kotlin`;
   prose values are scanned too (`"…hiểu biết cơ bản về C#, Javascript/Typescript…" → csharp, javascript, typescript`).
5. Unmatched **language** values are surfaced verbatim in
   `unmapped_languages` (closed vocabulary — nothing is invented). Unmatched
   **framework** values pass through as a slug with class `unclassified`
   (open vocabulary — the long tail is company-specific tooling).

### Topic text

- Lowercased regex over `title` and the 8-week `timeline` tasks.
- **Strength tiers per topic**:
  - `any` — domain words reliable in title *or* timeline (`iot`, `blockchain`, `trò chơi`, `vi điều khiển`);
  - `title_any` — words too generic for timelines, which are boilerplate
    ("phân tích yêu cầu, code, **kiểm thử**, **báo cáo**, deploy") — e.g. every
    `testing-qa` keyword, `dashboard`, `báo cáo`, `bảo mật`, `đào tạo`;
  - `all` — compound requirements (`\bsoc\b` **and** chip vocabulary for
    `chip-design`, so Security-Operations-Center prose is not a SoC;
    `chẩn đoán` **and** medical vocabulary for `healthcare`);
  - `stack` — normalized slugs evidence a topic (`pytorch` → `ai`,
    `playwright` → `testing-qa`, `arcgis` → `gis`).
- Evidence records the first title match, else the first text match, else the
  stack slug, with `n_matches`.

## 5. Output contract

All files in `forms/extracted/`:

| file | content |
|---|---|
| `projects_tagged.csv` | `projects.csv` columns + `project_key`, `topics`, `lang_tags`, `framework_tags`, `unmapped_languages`; 1:1 joinable with `projects.csv` by `source_file` + `no`; list separator `"; "`, sorted |
| `project_tags.json` | one record per `project_key`: titles, companies, source files, all tag axes, unmapped values, per-topic evidence |
| `tag_vocabulary.json` | every emitted slug with `class`, usage `count`, up to 12 source values that produced it |
| `tag_report.json` | coverage, per-topic counts, unmapped language values, top framework passthrough slugs |

## 6. Quality gates & limitations

Precision fixes already applied (regression-checked): `stem` inside
"sy**stem**", `unity` inside "Comm**unity**", `mAP` (object detection) inside
map/GIS rules, `SOC` (security) inside chip rules, `invest` inside
"investigate", `health` inside "API health check", `opc` inside "ZenoPCB".

Known limitations:

- Source noise propagates: OCR errors, truncated titles ("Phát triển và duy trì các ứng"),
  program-level forms listing several tracks (e.g. *Viettel Digital Talent*
  forms mention all disciplines → several topics).
- Near-duplicate titles across form revisions are kept as separate canonical
  projects (no fuzzy merge) — they receive identical tags in practice.
- 10 projects have no title in the source; they stay untagged (`topics` empty).
- `unclassified` passthrough slugs preserve the long tail verbatim; promote a
  value to a real slug by adding an alias in `STACK_VOCAB`.

## 7. Regenerate / extend

```bash
python3 tag_projects.py            # rebuild all four outputs (~6 s, deterministic)
python3 tag_projects.py --report   # + per-topic counts
```

- Add a tech alias: extend `STACK_VOCAB` (`slug, class, patterns`), rerun.
- Add/adjust a topic rule: extend `TOPICS`, document the slug here and in the
  topic table, rerun; check `tag_report.json` coverage.
- Removing a rule: prefer `title_any` for words that timelines over-trigger.