# Knowledge Source Sync

## Objective

Keep the AI's knowledge current as documents, records, and external sources change.

An ingestion pipeline that syncs approved sources on a schedule, tracks versions, and removes content whose source disappears.

## Before You Begin

This feature is being added to an application that already exists and already
works. Do not scaffold a new project, and do not assume a blank slate.

Inspect the codebase first and establish:

- The existing application structure and where code of this kind already lives.
- The framework and version in use.
- The existing design system — colours, spacing, typography, and component conventions.
- Existing UI components you can reuse instead of writing new ones.
- The existing database structure, if this feature needs to persist anything.
- The existing authentication and authorization system, if this feature is user-scoped.
- Dependencies already installed, so you don't add a library that duplicates one.
- The existing test setup and conventions.

Only start writing code once you understand the above. If the application
already implements part of this feature, extend it rather than replacing it.

## Implementation Instructions

1. Give an operator an explicit approval step for every source, and default to nothing being indexed. Automatic inclusion of whatever the connected account can see is how private material ends up answerable.
2. Store a content fingerprint or version for each document alongside the time of the last successful sync, and re-ingest only what has actually changed since then.
3. Reuse the app's existing background-job system and make each sync resumable at the document level, so an interruption partway through a large source restarts from the last completed document rather than from the beginning.
4. Remove indexed content in the same run in which its source is deleted, revoked, or made private. Content that outlives its source keeps answering questions the user is no longer permitted to ask.
5. Query-time retrieval and answer generation belong to Retrieval-Augmented Generation, and the operator's view of index freshness belongs to Vector Index Health. This entry owns ingestion and deletion only.

## UI and UX Requirements

Match the application's existing design system exactly. Reuse its components,
spacing, and typography. This feature should look like it was always there.

## Responsive Requirements

Works on mobile, tablet, and desktop. Touch targets are large enough to hit on a
phone, and nothing overflows horizontally at 320px.

## Accessibility Requirements

- Fully keyboard navigable.
- Correct semantic elements and ARIA roles.
- Visible focus states.
- Meets WCAG AA contrast.
- Dynamic changes are announced to screen readers.
- Respects prefers-reduced-motion.

## Edge Cases

- Every source needs a recorded version and a last-successful-sync time, and a run that fails must leave the previous successful state intact rather than marking the source as current.
- When access to an external source is revoked, or a document is deleted or restricted at the source, the corresponding indexed content and its vectors must be removed promptly, not left until the next full crawl.
- Large syncs get interrupted by deploys, timeouts, and rate limits. Progress must be checkpointed so a resumed run continues rather than reprocessing everything and re-billing the embedding work.
- Drafts, archived items, trashed pages, private sections, and comment threads are easy to sweep up by accident. Decide explicitly which states are eligible and exclude the rest by default.
- A run where most documents succeeded and a handful failed is the normal case. Report failures per source and per document with the reason, rather than marking the whole sync green or red.
- Access to a third-party source expires. Refresh credentials ahead of expiry, and when a refresh fails, pause the source, tell the operator which connection needs reauthorizing, and leave the existing index in place.
- Third-party sources throttle. Honour the retry signalling they send, back off progressively, and spread scheduled syncs so every account does not start on the same minute.
- The same document reachable through two connected sources must not be indexed twice, or retrieval will return it twice and crowd out other evidence.

## Testing

Exercise the feature end to end in the running application. Cover every edge case
above, then run the existing test suite and confirm nothing regressed.

## Acceptance Criteria

- [ ] No source is indexed until an operator approves it.
- [ ] Each document carries a version fingerprint and each source records its last successful sync time.
- [ ] An interrupted sync resumes from the last completed document instead of restarting.
- [ ] Deleting, restricting, or revoking access to a source removes its indexed content and vectors.
- [ ] Drafts, archived, and private items are excluded unless explicitly included.
- [ ] Partial failures are reported per source and per document with a reason and a retry.
- [ ] Expired third-party credentials pause the source with a reauthorization prompt rather than silently ending the sync.
- [ ] The feature matches the existing design system.
- [ ] No existing functionality is broken.

## Adaptation Rules

- Match the existing design system. Do not introduce a new colour palette,
  spacing scale, or component library.
- Reuse existing components and utilities wherever they fit.
- Follow the naming, file layout, and code style already present.
- Do not upgrade, replace, or remove existing dependencies to make this
  feature fit. Adapt the feature to the app, not the app to the feature.
- Do not break existing functionality. If a change is genuinely required in
  existing code, make the smallest one that works and say so.
- If something in these instructions conflicts with how the application is
  built, follow the application and explain the deviation.

## Final Verification

Before you report the work as done:

1. Re-read the acceptance criteria above and check each one against what you
   actually built.
2. Run the application and exercise the feature end to end.
3. Run the existing test suite and confirm you have broken nothing.
4. Check the feature on mobile, tablet, and desktop widths.
5. Check keyboard navigation and focus handling.
6. Summarize what changed: files added, files modified, and anything you
   deliberately did differently because of how this application is built.

If any acceptance criterion is unmet, fix it before reporting completion.