2026-07-03 · Journal

Public Content To CDN: Make App Content Discoverable

public-content seo sitemap json-ld cdn crawlable-pages write-time-read-time 410-gone

App content was visible to humans, invisible to crawlers — SPA shells with no title, body, or structured data. This entry records the new public content surface: an app declares an alias and publishes items into a registry; the platform serves crawlable pages, JSON-LD, canonical/OG metadata, and a per-alias sitemap.xml — runtime-updated, no rebuilds — and the site's CDN maps a clean prefix onto it. Plus the field notes: stable lastmod, rewrite-never-redirect, the singleton rule, seeding, and 410, not 404.

This journal entry records how KDCube apps went from "visible to humans, invisible to crawlers" to a first-class public content surface: crawlable HTML pages, JSON-LD, canonical/OG metadata, a per-alias sitemap.xml, and 410 Gone after retraction — published from app data at runtime, no rebuilds. The worked case is our own Reading Corner: the journal and blog articles you are reading through it.

The starting problem was blunt. App surfaces are client-rendered: an iframe widget serves an SPA shell, and a shell is what a crawler sees — no title, no body, no structured data. The runtime's /sitemap.xml was the SPA shell too. Content the apps publish — articles, docs, reports — simply did not exist for search engines or AI crawlers.

The design decision that shaped everything:

The app owns WRITE time      which items are public, their metadata,
                             publish / update / retract.
The platform owns READ time  every discoverability artifact is rendered by
                             the platform from a registry — no app code runs
                             on the serving path.
The site owns its root       robots.txt and the top-level sitemap index stay
                             with whoever owns the domain.
PUBLISH ON ONE SIDE, CRAWL ON THE OTHER — THE APP OWNS WRITE TIME, THE PLATFORM OWNS READ TIME WRITE TIME · THE APP app admin actions publish · update · retract hooks “Publish to Web” idempotent seed enabled-gated on public_content.<alias> content registry durable · object storage items/<slug>.json generation.json counter, not clocks rebuildable, once per fleet hot · shared app storage index.json → sitemap items/<slug>.json → pages READ TIME · THE PLATFORM crawler / shared link site.example/articles/<slug> site CDN /articles/* behavior + URI rewrite never a redirect a URL that answers 3xx can't be canonical …/__content__/… runtime · proc reserved __content__ handler reads ONLY the hot tier no app code on the serving path (app is a singleton — gate check only) read what a crawler gets crawlable page · 200 title/meta/body, no JS · rel=canonical OG/Twitter · JSON-LD + BreadcrumbList sitemap.xml published items only · accurate lastmod retracted · 410 Gone record kept · noindex body · fast de-index federation robots.txt + the site's sitemap index reference the app sitemap — the runtime host never appears in the index. Publish flows down the left; crawls flow down the right; the hot tier is where they meet.
Publish flows down the left; crawls flow down the right; the hot tier is where they meet.

The surface

An app declares one method per content alias and enables the surface in its config. That is the whole app-side contract:

@public_content(alias="articles", schema_type="Article")
async def public_articles(self) -> list[PublicContentItem]:
    """Full-sync source: the app's current public items."""
    ...
public_content:
  articles:
    enabled: true                                # explicit exposure — off by default
    canonical_base: "https://example.com/articles"
    sitemap: true
    og_defaults: { site_name: "Example" }

Day-to-day, the app mirrors its own publish/delete actions into a registry:

registry.publish(item)    page + sitemap entry live immediately
registry.update(item)     same, with a lastmod bump
registry.retract(slug)    record kept; the URL answers 410 Gone from now on

And the platform serves, with no further app code, under the app's existing public route namespace:

GET …/public/__content__                        sitemap descriptor list (host federation)
GET …/public/__content__/articles/sitemap.xml   the per-alias sitemap
GET …/public/__content__/articles/<slug…>       crawlable page · 410 · 404

Each page carries a real <title>/description/body (verifiable with plain curl, no JavaScript), rel=canonical, Open Graph + Twitter cards, and two JSON-LD blocks: the item document (Article, BlogPosting, …) plus a BreadcrumbList. The widget URL stays a widget shell — an iframe is never the SEO surface.

The registry: two tiers, a counter, no clocks

Where does the sitemap actually come from? A registry with two tiers — no Postgres, no Redis:

DURABLE (source of truth)   object storage (S3 in cloud, local FS in dev)
  public_content/<alias>/items/<slug>.json     full records
  public_content/<alias>/generation.json       monotonic mutation counter

HOT (what serving reads)    shared app storage (EFS in cloud, disk in dev)
  _public_content/<alias>/index.json           the per-alias index -> sitemap
  _public_content/<alias>/items/<slug>.json    mirrored records -> pages
  _public_content/<alias>/.index.signature     generation stamp

Crawler traffic reads only the hot tier; the durable backend is never on the request path. The hot tier is derived and rebuildable: wipe it, and the next app load rebuilds it from durable records — one fleet-wide owner builds while every other worker observes the signature and skips. Publishes serialize under a shared-storage lock; readers never take it (hot files are replaced atomically).

Consistency never compares dates. It compares the generation counter — a publish bumps it durably, and any tier whose stamp disagrees rebuilds. No clock skew, no timezone traps. Dates (published_at, lastmod) are carried metadata for crawlers, nothing more.

Field notes

Things a real implementation got wrong first, and the rules that came out:

  • lastmod from "now" is a trap. The first mapping defaulted a missing update time to the current time — so an idempotent full re-seed re-stamped every sitemap <lastmod>, telling crawlers everything changed. Rule: lastmod derives from the item's own update time, falling back to its publish date — never to "now".
  • A canonical cannot redirect. Our edge already had a vanity mechanism — a CloudFront function answering 308 redirects. Tempting, wrong: a URL that answers 3xx cannot be the canonical, and the equity flows to the redirect target. Clean content URLs need the other primitive: rewrite + forward (a path behavior whose function rewrites the URI to the serving route and proxies through). The browser URL never changes.
  • Content providers must be singletons. No app logic runs per crawler request, but the route plumbing does resolve the app instance to check the declaration and the config gate. A non-singleton app would construct a fresh entrypoint per crawled page. singleton: true is now part of the contract.
  • Existing content predates the registry. The lifecycle hooks cover new publishes; everything published before the surface existed is invisible until seeded. The pattern: an idempotent admin operation that pushes the full current set — surfaced as a "Publish to Web" button in the app's admin widget. Safe to press repeatedly (stable lastmod makes it truly idempotent), also the recovery move after any bulk archive change.
  • Retraction is 410, not 404. A retracted item keeps its record, and its URL answers 410 Gone with a noindex body — the strongest "permanently removed" signal, which de-indexes much faster than a 404's "maybe temporary".
  • Two environments, one durable store. Reads propagate pull-on-load (a running environment serves its own hot copy until its next reload — serving never checks the durable backend per request). Writes must come from one environment: the mutation lock lives on each environment's own shared filesystem.

To the CDN: the split-origin schema

Production shape: the marketing site on one domain, the KDCube runtime on another. Public content is anonymous — no cookies, no session — so unlike embedded widgets (which need same-origin auth cookies), crawlable pages can be fronted across origins by pure CDN plumbing:

crawler
  → site CDN (https://site.example)
      static site              /, /docs/*, sitemap.xml, robots.txt
      behavior /articles/*  →  custom origin = the runtime host
        + viewer-request function rewriting the URI:
          /articles/<rest> → /api/…/<app>/public/__content__/articles/<rest>
              → runtime CDN → proxy (public location) → proc → hot tier

Three config pieces, three owners:

canonical_base            runtime descriptor      https://site.example/articles
path behavior + rewrite   the SITE's CDN          maps the clean prefix (never a redirect)
robots + sitemap index    the website build       one Sitemap: line + one <sitemap> entry

The site's sitemap index federates the app's runtime-generated sitemap exactly like its build-time doc/blog sitemaps — the only novelty is that this sub-sitemap updates on every publish, with no site rebuild. A crawler walks: robots.txt → site index → the app's sitemap (fresh lastmod) → clean item URLs, each canonical to itself. The runtime host never appears in the index.

Local development composes the same shape on one origin — any reverse proxy performing the same rewrite. The raw runtime URLs need zero infra changes (the public proxy locations and gateway patterns already pass them), so staging verification is a plain curl before any CDN work:

curl …/public/__content__/articles/sitemap.xml      urlset + lastmod
curl …/public/__content__/articles/<slug>           200, canonical, OG, JSON-LD
(retract) → same URL                                410

What this is not (yet)

The publish path today ends at the registry + sitemap. The designed next stage attaches to the same public_content.changed notification and the durable registry: IndexNow push, RSS/Atom + WebSub, Search Console sitemap registration, CDN invalidation on publish. The notification is deliberately not a delivery log — the registry stays authoritative, and any worker that misses messages resyncs from durable records.

Documentation on GitHub

The live docs behind this entry:

KDCube Journal · 03.07.2026