About SITI

What SITI Does

SITI operates across five integrated modules, each designed to answer a specific question a minister or policy analyst would ask.

Data Sources

SITI does not receive any data from the Ministry. All data is collected from publicly accessible sources. No authenticated APIs, no credentials, no privileged access.

Collection Schedule

All times are Malaysia/Kuching (MYT, UTC+8). The scraper runs as a daemon with an in-process scheduler (APScheduler). Every run writes a log row recording status, duration, item count, and any errors.

AI Pipeline Schedule

How It Works

Three-Service Architecture

SITI is built as three independent services that share a single PostgreSQL database and Redis instance. This separation means each service can be restarted, scaled, or replaced independently.

Infrastructure

The database and Redis instance run on a Synology NAS (nas1) with PostgreSQL 16, pgvector for embedding search, and Apache AGE for graph-based cross-pillar correlation. The user-facing edge is a Hostinger VPS running Traefik for SSL termination and reverse proxy. The VPS connects to nas1 over an SSH reverse tunnel — no database ports are exposed to the internet.

Briefing Generation Pipeline

1. User picks scope in the UI (event, attraction, date range, or custom)
2. bundler.py assembles a context bundle: all relevant reviews, news articles, hotel signals, social signals, and forecasts for the scope
3. DeepSeek Reasoner receives the bundle with a system prompt establishing it as a senior tourism intelligence analyst writing for the Sarawak Minister of Tourism
4. The model streams response tokens back to the browser over SSE (Server-Sent Events) in real time
5. citations.py resolves every inline citation token against the reference map; orphaned citations cause rejection
6. linter.py runs six deterministic quality rules. Any violation marks the run as FAILED and prevents persistence
7. The briefing is persisted transactionally with its data references and generation run metadata

Technology Stack

Data Collection Ethics

• Public data only. SITI only accesses publicly available web pages. No authenticated APIs, no credential harvesting, no privileged access.
• robots.txt respected. Sites with robots.txt restrictions (Google, Meta) are only accessed where permitted.
• No PII beyond public display. Reviewer names and profile information are only stored as they appear publicly on review platforms.
• Rate limited. Maximum 1 request per 3 seconds per domain by default, tuned per source to avoid server load.
• Residential proxies. Bright Data residential IP pool used for review sites to respect geographic rate limits. Direct connections used where permitted.
• User-agent transparency. Crawlee's built-in fingerprint rotation identifies the scraper honestly; no impersonation of real browsers.
• No silent data loss. Partial scrapes are flagged as incomplete. Three consecutive failures on any source trigger an alert. Nothing is silently dropped.
• Audit trail. Every scrape run and every AI generation is logged with status, duration, and item count for full traceability.

Project Context

SITI is a demonstration system built by Aidju Digital to win a paid engagement with MTCP Sarawak. The Ministry does not provide data, so SITI scrapes public sources, runs AI analysis, and produces decision-grade intelligence the Ministry cannot get from standard Sarawak Tourism Board reports.

This is a pre-engagement demo. It is not a finished product. The demo's job is to demonstrate capability, not to ship at production scale. All modules render with real data or clearly marked stubs. Paid engagement scope (multi-tenant, Bahasa Melayu, real-time dashboards, public API, neural forecasting) is documented and awaits ministerial sign-off.

Last updated: 24 May 2026. Status: Demo build, feature-complete.