[themarolt]

Summary

findOrCreateOrganization rewrote every existing organization it found: the identity and attribute upserts ran on every call, and markOrgAttributeDefault updated every attribute row of its name without checking whether the flag changed. Most organizations arrive unchanged, so almost all of these were no-op writes. data-sink-worker runs without transactions, so each of them was its own commit.

This PR works out which writes would change anything and skips the rest.

Changes

• For an existing organization, findOrCreateOrganization reads its attributes and identities in one query, a single-round-trip combination of what findOrgAttributes and fetchOrgIdentities return. It then checks whether any write would change something: the organization update, the identity upsert, the attribute upsert or the default flags. If none would, it returns before taking the per-organization lock. prepareOrganizationData rewrites the default flags of the attributes it's given, so the check runs on a copy and compares with what is stored.
• If anything would change, every write runs as before: the organization update when columns differ, the identity and attribute upserts, and the default marking. Callers without the per-organization lock can see the organization change after the first read, so these writes don't rely on decisions taken from it.
• The members enrichment worker and the public API use the same function, so they also skip organizations that are unchanged.

Local end-to-end runs against main (b440cf5), on 92.5k real github/git results and 39,659 real meetings activities seeded read-only from prod:

	main	this PR
github/git, orgAttributes updates	113,057	18
github/git, s per batch	0.185–0.227	0.104
meetings, orgAttributes updates (pass 1)	285,022	2,455
meetings, s per batch (first pass / re-send)	0.227–0.259 / 0.222–0.249	0.192 / 0.177

Inserts and the writes to organizations and organizationIdentities, the tables that are replicated, are the same as on main. Every table has the same rows and columns as main, including the default flags, and so do the Kafka activity messages, search-sync messages and Redis keys. The only differences are generated ids, timestamps, and one organization whose payload order varies between runs of main as well (linuxfoundation/lfx-cdp-insights-ops#972).

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

Related issue

Closes linuxfoundation/lfx-cdp-insights-ops#969

[cursor]

PR Summary

Medium Risk
Changes the write path for org upserts used by data-sink and API; incorrect skip logic could leave organizations stale, though checks are tied to existing upsert conflict keys and default-flag rules.

Overview
findOrCreateOrganization now loads an existing org’s attributes and identities in one query, runs prepareOrganizationData on a copy of stored attributes, and returns immediately (without the per-org lock) when nothing would change: no org column updates, identities already match (including verified when provided), attributes already exist under the same upsert key, and default flags already match markOrgAttributeDefault semantics.

When a change is needed, behavior stays the same: optional lock, re-read + re-plan after lock, then the existing update/upsert/default writes. New helpers hasAllOrgAttributes, isOrgAttributeDefault, and hasAllOrgIdentities (plus shared findStoredOrgIdentity) centralize the “already stored?” checks used by that plan.

^{Reviewed by Cursor Bugbot for commit bcf5717. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🟡 Changes recommended

Concurrent updates without organization locking can leave multiple attributes marked as default or skip requested identity-verification changes based on stale state.

2 open issues

• Concurrent default changes can leave multiple attributes as default · New
• Precomputed identity changes can go stale while waiting for the update · New

What changed in this PR

This PR reduces unnecessary organization writes in the shared data-access layer, primarily improving data-sink member updates while also affecting enrichment and public API callers.

Changes:

• Reads stored attributes and identities together to plan updates and skip unchanged organizations before locking.
• Rechecks state after acquiring the optional organization lock and executes only planned writes.
• Adds identity and attribute comparison helpers and filters default-flag updates to changed rows.

File	Description
services/​libs/​data-access-layer/​src/​organizations/​identities.ts	Adds identity comparison helpers and reuses identity matching in upserts.
services/​libs/​data-access-layer/​src/​organizations/​base.ts	Plans organization updates and skips unnecessary locks and writes.
services/​libs/​data-access-layer/​src/​organizations/​attributes.ts	Adds attribute comparisons and limits default-flag updates to changed rows.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are recorded for the optimization that skips writes when organization data is unchanged.

No open issues 🎉

2 resolved since last review

• Precomputed identity changes can go stale while waiting for the update
• Concurrent default changes can leave multiple attributes as default

_{Review effort: Balanced}