Palace Ingest, Capture, And Source Workflows

Source basis: Palace of Truth backend/app/main.py, backend/app/api/ingest.py, backend/app/api/capture.py, backend/app/api/web_saves.py, backend/app/api/feeds.py, backend/app/api/source_subscriptions.py, backend/app/api/jobs.py, backend/app/api/palace.py, backend/app/schemas/ingest.py, backend/app/schemas/feed.py, backend/app/schemas/source_subscription.py, backend/app/services/palace.py, backend/app/pipelines/webpage.py, backend/app/pipelines/youtube.py, docs/source-subscriptions.md, and extension/README.md, last reviewed for this landing surface on July 9, 2026.

This guide separates current public workflows from compatibility aliases, operator-only surfaces, and deferred ideas. Use generated OpenAPI at /api/openapi.json for exact schemas in a running Palace environment.

Current Workflow Map

Workflow	Current public route family	Status	Notes
Notes	`POST /api/v1/ingest/note`	Supported	Accepts a title, content, optional tags, optional webhook URL, and optional model override.
Webpages	`POST /api/v1/ingest/webpage`	Supported	Accepts an HTTP(S) URL and queues webpage processing. Duplicate active URLs return a conflict instead of creating another item.
Media links	`POST /api/v1/ingest/media`	Supported	Queues audio/video URL processing through the media pipeline. YouTube URLs are media ingest, not a separate current workflow. Operators can choose OpenAI, AssemblyAI, or local WhisperX transcription providers.
Documents and PDFs	`POST /api/v1/ingest/doc`	Supported	Uploads `.pdf`, `.docx`, `.xlsx`, `.md`, or `.txt`; extracts text before queueing work so extraction failures do not leave orphaned jobs.
Images	`POST /api/v1/ingest/image`	Supported	Uploads `.jpg`, `.jpeg`, `.png`, `.gif`, or `.webp`; analyzes image content before queueing the ingest job.
Batch ingest	`POST /api/v1/ingest/batch`	Supported for notes, webpages, media, and legacy `youtube` item type	Accepts up to 100 entries. It does not upload binary docs or images.
Browser capture	`POST /api/v1/capture/browser`	Supported	The extension sends one capture contract; Palace resolves it to note, media, social post, or webpage capture.
Saved web lookup	`GET /api/v1/web-saves` and `PATCH /api/v1/web-saves/{web_save_id}`	Supported	Used to list active saved URLs, identify duplicate saves, filter by capture kind or tag, and archive or restore a saved-web record.
RSS feeds	`/api/v1/feeds`	Supported	Create, list, edit, poll, enable, disable, soft-delete, restore, list items, and import OPML.
OPML import	`POST /api/v1/feeds/import_opml`	Supported	Uploads an OPML file and creates missing RSS feed records; newly created feeds are queued for immediate polling.
YouTube channel subscriptions	`/api/v1/source-subscriptions`	Supported v1 provider	Only `youtube_channel` subscriptions are supported in v1. Preview before create, manually sync, list entries, and retry failed entries.
Job visibility and retry	`/api/v1/jobs`	Supported	List, read, cancel, and retry ingest jobs that are visible to the authenticated tenant or capture token.
Palace sync sources	`/api/v1/palace/sync-sources` and `/api/v1/palace/sync-runs`	Operator/API-key surface	Current sync-source support covers local folder, repository, and S3-compatible source kinds under the Palace control-plane API. Treat it as an operator workflow, not browser capture.

All route families above are mounted under /api/v1 by the backend. Auth is tenant-scoped unless a route explicitly belongs to an operator-only or capture-token path.

Direct Ingest

Direct ingest is the right path when a client already knows the content type.

Use POST /api/v1/ingest/note for short text notes or manually composed memory material.
Use POST /api/v1/ingest/webpage for ordinary public HTTP(S) pages.
Use POST /api/v1/ingest/media for YouTube, podcast, video, or audio URLs handled by the media pipeline.
Use POST /api/v1/ingest/doc for uploaded documents and PDFs.
Use POST /api/v1/ingest/image for image files that should be described and indexed.
Use POST /api/v1/ingest/batch for mixed note, webpage, media, and compatibility youtube URL entries when one client needs to enqueue many items together.

The async response gives a job_id with status queued. Use /api/v1/jobs/{job_id} or /api/v1/jobs to inspect progress before claiming the item is searchable.

Webpage extraction can use the built-in local scraper or an operator-configured Firecrawl Cloud/self-hosted provider. That provider choice is deployment configuration, not a different public route.

Media transcription provider choice is also deployment configuration. The public route remains /api/v1/ingest/media; local WhisperX is available for deployments that intentionally select it.

Browser Capture And Saved Web

The browser extension submits to POST /api/v1/capture/browser. The extension sends page URL, page title, optional selection text, user tags, optional detected kind, and extension metadata.

Palace resolves that request as follows:

Selected text becomes a selection_note and routes through note ingest.
YouTube, Shorts, youtu.be, and direct audio/video URLs route through media ingest.
Social post URLs preserve capture_kind=social_post while using webpage processing.
Other HTTP(S) URLs route through webpage ingest.

The extension checks /api/v1/web-saves before save so exact active URL matches can show an already-saved state. Saved-web records can also be filtered by query, capture kind, tag, sort, and active-only state. Archiving a saved-web record is separate from deleting production knowledge.

RSS Feeds And OPML

RSS feeds use /api/v1/feeds.

POST /api/v1/feeds creates a feed and queues an initial poll.
POST /api/v1/feeds/{feed_id}/poll force-polls an existing feed.
POST /api/v1/feeds/import_opml imports RSS outlines from OPML, skips existing tenant-scoped feed URLs, and queues polls for newly created feeds.
POST /api/v1/feeds/{feed_id}/disable, /enable, /restore, and DELETE /api/v1/feeds/{feed_id} manage feed lifecycle. Delete is a soft delete of the feed record.

Feed polling writes item provenance through feed metadata. It is distinct from YouTube channel source subscriptions.

YouTube Channel Source Subscriptions

Source subscriptions use /api/v1/source-subscriptions. The current v1 provider is youtube_channel.

Recommended flow:

Preview the channel with POST /api/v1/source-subscriptions/preview.
Create the subscription with POST /api/v1/source-subscriptions.
Let the worker poll on its configured interval, or request a bounded manual sync with POST /api/v1/source-subscriptions/{subscription_id}/sync.
Inspect discovered, queued, captured, skipped, or failed entries with GET /api/v1/source-subscriptions/entries or GET /api/v1/source-subscriptions/{subscription_id}/entries.
Retry failed entries with POST /api/v1/source-subscriptions/entries/{entry_id}/retry.

V1 supports public YouTube channel uploads. It does not claim support for unbounded historical backfill, Shorts, live streams, upcoming streams, comments, playlists as independent sources, or private/member-only videos. Backfill must be bounded with backfill_limit, backfill_published_after, or both.

Palace Sync Sources

Palace sync sources use /api/v1/palace/sync-sources and /api/v1/palace/sync-runs. They are operator/API-key workflows for curated corpora, not browser capture flows.

Supported source kinds:

local for mounted folders allowed by deployment policy.
repo for curated GitHub repository sources, with public/no-credential, deployment-managed token, stored PAT, or stored SSH-key credential modes depending on operator configuration.
s3 for MinIO, R2, or another S3-compatible object store.

Create, edit, soft-delete, restore, and manually sync these sources through the Palace control-plane API. Keep secret material in deployment-managed secrets or encrypted stored credentials. Public docs should describe the shape, not include real repository tokens, object-store keys, or private corpus paths.

Compatibility And Deprecated Surfaces

POST /api/v1/ingest/youtube is a deprecated alias for /api/v1/ingest/media and is hidden from the OpenAPI schema.
POST /api/v1/ingest/pdf is a deprecated PDF-only alias for /api/v1/ingest/doc and is hidden from the OpenAPI schema.
Older MVP and brainstorm documents describe historical plans. Treat them as provenance, not current public workflow documentation.
Internal admin routes and operator-only recovery paths should not appear in public smoke instructions for runtime agents.
Production data cleanup or deletion remains a human-only operation. This guide documents supported paths but does not authorize data deletion.

Auth And Secret Boundaries

REST ingest uses tenant runtime credentials. Examples should use placeholders such as PALACEOFTRUTH_API_KEY, never raw secrets.
Browser capture uses a revocable capture token exchanged from a Palace API key. The pairing key is not stored by the extension.
Admin secrets are operator-only and must not be placed in public docs examples, agent runtime config, MCP config, browser storage, or landing-site code.
Remote MCP bearer or OAuth credentials belong in deployed/shared-runtime configuration, not checked-in docs.

Verification Checklist

Before publishing workflow copy, verify:

Endpoint names match the route families mounted under /api/v1.
/ingest/youtube and /ingest/pdf are clearly marked deprecated aliases.
YouTube channel subscriptions are described as the only v1 source-subscription provider.
OPML import is documented under feeds, not source subscriptions.
Browser capture and saved-web lookup are separate but connected workflows.
Palace sync sources are described as operator workflows with local, repo, and S3 source kinds.
No example contains a raw tenant key, bearer token, admin secret, cookie, or user content.

Source Inventory

palaceoftruth/backend/app/main.py
palaceoftruth/backend/app/api/ingest.py
palaceoftruth/backend/app/api/capture.py
palaceoftruth/backend/app/api/web_saves.py
palaceoftruth/backend/app/api/feeds.py
palaceoftruth/backend/app/api/source_subscriptions.py
palaceoftruth/backend/app/api/jobs.py
palaceoftruth/backend/app/api/palace.py
palaceoftruth/backend/app/schemas/ingest.py
palaceoftruth/backend/app/schemas/feed.py
palaceoftruth/backend/app/schemas/source_subscription.py
palaceoftruth/backend/app/services/palace.py
palaceoftruth/backend/app/pipelines/webpage.py
palaceoftruth/backend/app/pipelines/youtube.py
palaceoftruth/docs/source-subscriptions.md
palaceoftruth/extension/README.md