Skip to content

Page collection reads and restrict federation export to inspection - #194

Open
nirmalhk7 wants to merge 5 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/collection-pagination
Open

nirmalhk7 wants to merge 5 commits into
NTARI-RAND:mainfrom
nirmalhk7:nirmal/collection-pagination

Conversation

@nirmalhk7

Copy link
Copy Markdown
Contributor

What the change is

Adds bounded pages to conversation lists, the user directory, and read-only federation exports. JSON response shapes remain compatible; encrypted continuation tokens travel in response headers, and the bundled chat client follows them.

Dependency: stacked on #191 at eae4e15. The current combined diff includes that ownership PR; this PR’s own change is commit 4c58c3240377ac7faaa039733dfdc3a17c0cd901. Review #191 first and deploy private chat with #192.

Why it is needed

Single-page DynamoDB scans silently lose later records, while collecting every page in one response makes reads unbounded. Following more pages also increases the impact of existing privacy gaps: directory responses expose account fields, and federation export sends raw Users records and local dialog notes.

How the change works

Each request evaluates up to 50 records per table by default, with a maximum of 100. Cursors expire after one hour and bind the account, endpoint, limit and filters. Empty filtered pages retain their continuation. Conversation ownership applies on every page; directory responses allowlist public fields and include coarse location only with the existing sharing opt-in.

Federation export requires a distinct read-only key and omits account records, dialog narrative and scalar ratings. Outbound inspection uses explicitly configured peer read keys, refuses redirects, and requires declared pagination. Scheduled paths inspect all pages and report counts; automatic importing is disabled and lastSyncAt is not advanced. The existing importer cannot safely merge redacted summaries without source-aware conflict handling. Manual import and other account/admin hardening remain documented follow-up work.

Scans provide no snapshot or stable sort under concurrent changes. External clients must follow the pagination headers. Export-key configuration, field omissions, compatibility changes, and deployment dependencies are detailed in the guide below.

Where it is involved

Backend collection routes, conversation model, shared cursor/scanning helper, federation export/inspection paths, and the chat account client. API, privacy and rollout guide. CONFORMANCE.md records the application boundary and remaining federation work.

Verified locally: all 37 Node tests, 82 Jest tests, 13 chat-client tests, and the production UI build passed. New HTTP and client regressions failed before implementation. A real DynamoDB Local check passed continuation across the 1 MiB scan boundary, empty filtered pages, owner isolation, private projections, account-client traversal, read-only export counts, and preservation of local credentials and narrative. The same storage check is included in CI with a digest-pinned database image. Syntax, hardcoded-URL and diff whitespace checks passed. The commit carries a DCO sign-off.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Bound conversation, directory, and federation scans with encrypted continuation tokens. Keep owner and sharing filters on every page, and make bundled clients follow continuation explicitly.

Use a separate read-only export credential and explicit peer configuration. Exclude local accounts and narrative, and disable automatic merging of these summaries until source-aware reconciliation is defined. Document rollout dependencies and compatibility changes.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
@nirmalhk7
nirmalhk7 marked this pull request as ready for review October 28, 2026 20:19
Comment thread backend/routes/userRoutes.js Outdated
TableName: USER_TABLE_NAME
};
// GET /users - bounded directory, never account or private location records
router.get('/', account, async (req, res) => {
};
const app = express();
app.use(express.json());
app.use(require('../middleware/authMiddleware'));
Comment thread backend/server.js Outdated
// Middleware
app.use(authMiddleware);
app.use('/uploads', express.static(path.join(__dirname, 'uploads')));
app.use('/uploads', chatAuth, asyncHandler(require('./middleware/attachmentOwner')), express.static(path.join(__dirname, 'uploads'), {
Apply the 1,000-per-minute limits before authentication and after
account verification. Adding an API key must not reset the account's
allowance. Bound the local storage-check server as well.

Keep pagination and private attachment checks intact. Merge current
main's retry-safe messages, retain both message-model exports and CORS
headers, and test retries against owned conversations. API-key-only
callers remain unable to send or replay private messages.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Move input checks and saved-receipt replay into focused helpers. Extract
verification database setup and worker cleanup without changing request
results, receipt keys or lifecycle ordering.

Add a regression for a missing stored assistant reply: replay must return
410 without repeating delivery or returning a partial success.

Signed-off-by: Nirmal Khedkar <nirmalhk7@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants