ClosedBit
Guide Index Previous Part Next Part Download Markdown

#Part 12 Section 1. Native Browser Boundary

ClosedBit Browser is authored in .cbp and uses ClosedBit drawing, input, HTTP, storage, parser, and web-rendering paths. It does not embed Chromium, WebView, CEF, Qt, or another application framework. Its native engine currently handles a deliberately bounded subset of HTML/CSS/image behavior and sandboxed ClosedBit Web packages.

That boundary must be described honestly. Passing a hand-picked page is not full web compatibility. Compatibility work requires parser conformance, URL behavior, Unicode, layout, fonts, accessibility, cookies/storage policy, TLS behavior, navigation security, sandboxing, crash isolation, and large automated test suites.

#Part 12 Section 2. Search Contract

When the address bar receives words rather than an address, the browser calls the ClosedBit Search endpoint. Results use a compact tab-separated contract:

title<TAB>https://public.example/page<TAB>snippet

The UI accepts at most ten rows, strips line breaks from fields, and opens only the selected result. Search authentication tokens are not required for public queries, and ClosedBit ID tokens are never sent to the search endpoint.

#Part 12 Section 3. Owned Web Crawler

ClosedBit Search uses its own crawl frontier and full-text index instead of forwarding each query to another search engine. The service consists of:

seed API -> D1 crawl frontier -> Cloudflare Queue -> bounded fetch/parser
                                              -> D1 documents + FTS5
browser search -> FTS5 ranking -> ten TSV results

The crawler accepts public HTTPS only, rejects local/private targets, enables Cloudflare's strict public-Internet fetch routing, follows a bounded number of redirects, identifies itself as ClosedBitCrawler/1.0, fetches /robots.txt, honors applicable allow/disallow rules, honors X-Robots-Tag and meta noindex/nofollow, limits document bytes and crawl depth, strips common tracking parameters, and initially discovers same-host links only. It does not send cookies or authentication credentials.

#Part 12 Section 4. Crawler Operations

The relay project is in D:\S3cur1tyF1rst\relay. Local validation is safe and does not create remote infrastructure:

npm install
npm run types
npm run db:migrate:local
npm run check

The first production-shaped deployment now runs at https://s3cur1tyf1rst-relay.i-do-nerdy-stuff-3eb.workers.dev on Cloudflare's free tier. Its closedbit-search D1 database, crawl queue, dead-letter queue, migration, 15-minute scheduled trigger, and authenticated seed route are active. The initial ClosedBit site and documentation seeds have produced searchable FTS5 documents. This proves the end-to-end service path; it does not imply web-wide coverage.

For a new deployment environment, create a D1 database, a crawl queue, and a dead-letter queue in the intended Cloudflare account; set its D1 ID in wrangler.jsonc; set RELAY_TOKEN and optionally a least-privilege CRAWLER_ADMIN_TOKEN as secrets; apply migrations remotely; then deploy intentionally. When the dedicated crawler token is absent, the private relay token authorizes seed administration. Do not put either secret in source control.

Seed a public HTTPS URL with an authenticated request to POST /v1/crawl/seed containing {"url":"https://example.com/"}. GET /v1/crawl/stats reports document and frontier counts. A scheduled trigger requeues due pages. Failed queue messages back off and eventually reach the dead-letter queue for inspection.

#Part 12 Section 5. Ranking And Quality

The first ranking layer uses FTS5 BM25 weighting for title, description, and body text, then uses fetch recency as a stable tie-breaker. Search quality work should add duplicate clustering, canonical consolidation, language detection, spam resistance, link signals, host diversity, freshness policies by content class, safe-search classification, and evaluation sets. Never claim web-wide coverage from a small seed set.