Knowledge Base

The Knowledge Base captures practical operational knowledge that doesn't fit neatly into architecture docs or user guides. This is the home for hard-won lessons, known issues, vendor quirks, debugging guides, and standard procedures accumulated while building and operating the SONAN DIGITAL CRM.

---

Purpose

Architecture Decision Records explain what was decided and why. User guides explain how to use features. The Knowledge Base answers the questions that fall between those two:

• Why did that deployment break?
• What's the correct column name — amount_cents or subtotal_cents?
• How do I safely commit to a FUSE-mounted repo?
• What's the Resend free tier limit?
• What do I do when a client can't log into the portal?

These are the things you learn by doing, and the KB exists so you only learn them once.

---

Contents

Document	What It Covers
Lessons Learned	Detailed post-mortems on technical mistakes: root cause, fix, and how to prevent recurrence
Known Issues	Tracked bugs, limitations, and accepted risks — with severity, workarounds, and target fix versions
Vendor Notes	Operational quirks, limits, and gotchas for Supabase, Vercel, Stripe, and Resend
Troubleshooting	Step-by-step diagnosis for the most common errors and failures
Common Fixes	Quick-reference tables for auth issues, RLS issues, build errors, and the FUSE-safe commit pattern
Production Incidents	Log of all P1/P2 production incidents with timeline, root cause, fix, and prevention actions
SOPs	Standard Operating Procedures: deploy, migrate, rotate secrets, monitor crons, onboard/offboard
AI Prompt Library	Ready-to-use prompts for Claude and other AI assistants to accelerate CRM development tasks

---

How to Use This KB

Starting a new feature? Check Known Issues for any deferred limitations that might affect your feature, and Vendor Notes for any constraints from the services you'll use.

Something broke in production? Start with Troubleshooting for the most common errors. If it's a new issue, open an incident record in Production Incidents immediately, even if it's not resolved yet.

Writing a migration or new table? Review Lessons Learned — specifically the RLS policy gaps section — before shipping.

Making a git commit to the FUSE-mounted repo? Use the quick reference in Common Fixes → FUSE-Safe Commit Steps, or read the full pattern in CLAUDE.md.

Adding a new vendor or rotating secrets? See the relevant vendor section in Vendor Notes and follow SOP-003: Rotate a Secret.

---

Contributing

When you fix a bug, hit a vendor limit, or resolve an incident:

1. Add a lessons-learned entry or update the known issue status
2. If it was a production incident (user-impacting), add it to production-incidents.md
3. If there's now a repeatable fix, add it to troubleshooting.md or common-fixes.md
4. If you wrote a useful AI prompt during the work, add it to ai-prompt-library.md

The KB is only useful if it stays current. A two-sentence entry written immediately after fixing something is worth more than a detailed write-up that never gets written.

---

Lessons Learned

This document records significant technical mistakes made during development and operation of the SONAN DIGITAL CRM. Each entry has a root cause, the fix that was applied, and how to prevent recurrence. The goal is that each mistake only costs us once.

Entries are ordered by impact, highest first.

---

LL-001: FUSE Blob Corruption (HIGH IMPACT)

Impact: Corrupted files pushed to GitHub. Vercel build failed. Required manual recovery via git plumbing.

What Happened

Large TypeScript files were written to the FUSE-mounted workspace path using the Edit and Write tools (and direct bash writes to the mount path). The files appeared to write successfully — no error was returned. The on-disk file showed a line count that matched what was expected.

When the git blob was hashed and checked, the blob also showed the same truncated line count. Because both the disk file and the blob agreed on the (truncated) count, the verification step passed — and a corrupted blob was committed and pushed to GitHub.

Vercel pulled the commit and attempted to build. The build failed because the TypeScript file was incomplete (truncated mid-function). The corruption was only discovered at build time, not at commit time.

Root Cause

FUSE (Filesystem in Userspace) mounts have write size limitations. When a file larger than the internal FUSE buffer is written in a single operation, the write is silently truncated — no error is raised, and the partial write is accepted as complete. This is a known FUSE behavior, not a bug in any specific tool.

The standard verification approach (compare line count of disk file to blob) failed here because both the disk write and the git hash-object read from the same FUSE path. Both saw the same truncated content, so they agreed — giving a false pass.

Fix Applied

All file writes must go through /tmp (non-FUSE), not the FUSE-mounted workspace path. The correct pattern:

1. Write file content to /tmp/filename.tsx using Python open() + f.write()
2. Hash the blob from /tmp using Python subprocess reading the file into stdin
3. Verify blob line count against the /tmp file (not the FUSE path)
4. Build tree with the new blob SHA
5. Commit and push

# Step 1: Write to /tmp (NEVER to the FUSE-mounted workspace path)
content = r"""...full file content..."""
with open('/tmp/filename.tsx', 'w') as f:
    f.write(content)

# Step 2: Hash from /tmp
import subprocess
with open("/tmp/filename.tsx", "rb") as f:
    data = f.read()
result = subprocess.run(
    ["git", "hash-object", "-w", "--stdin"],
    input=data,
    capture_output=True
)
sha = result.stdout.decode().strip()

# Step 3: Verify (compare /tmp line count to blob line count)
# Both must match EXACTLY

Prevention

• The FUSE-safe commit pattern is now documented in CLAUDE.md and is mandatory for every commit
• Never use Edit or Write tools on paths under the FUSE mount
• Never use bash echo / cat / tee to write files to the FUSE mount path
• The warning "Write/Edit tools on FUSE path truncate large files silently" is tracked in CLAUDE.md § 6

---

LL-002: Supabase Nested FK Array Typing

Impact: TypeScript build errors in production. Runtime crashes when accessing .name on an array value.

What Happened

A Supabase query joined time_logs with users and tasks using the FK hint syntax:

const { data } = await supabase
  .from('time_logs')
  .select(`
    id, hours, logged_date,
    tasks!task_id ( id, title, projects!project_id ( id, name ) ),
    users!employee_id ( id, full_name, email )
  `)

The TypeScript interface was written as if the joins returned single objects:

// WRONG — this is what was written
interface TimeLogRow {
  id: string
  hours: number
  logged_date: string
  tasks: { id: string; title: string; projects: { id: string; name: string } | null } | null
  users: { id: string; full_name: string; email: string } | null
}

This compiled without error because TypeScript was satisfied with the interface. But at runtime, tasks and users were arrays, not objects. Accessing row.tasks?.title returned undefined instead of the task title, and row.users?.full_name crashed with "Cannot read properties of undefined" on some code paths.

Root Cause

Supabase always returns FK joins as arrays, even for many-to-one relationships. This is a fundamental aspect of the Supabase JS SDK — the type inference from supabase.from().select() does reflect this, but when you define your own interface (which is common for clarity), it's easy to define it as a single object and miss the array shape.

Fix Applied

Define interfaces to match the actual array shape returned by Supabase. Access all nested FK data with [0] indexing:

// CORRECT — matches actual Supabase return shape
interface TimeLogRow {
  id: string
  hours: number
  logged_date: string
  tasks: Array<{
    id: string
    title: string
    projects: Array<{ id: string; name: string }>
  }>
  users: Array<{ id: string; full_name: string; email: string }>
}

// CORRECT — access with [0]
const taskTitle = row.tasks?.[0]?.title
const projectName = row.tasks?.[0]?.projects?.[0]?.name
const employeeName = row.users?.[0]?.full_name

Prevention

• CLAUDE.md § 4 documents this pattern explicitly
• All new Supabase queries with FK joins must use the array interface + [0] pattern
• Code review checklist: check any Supabase join interface for single-object definitions

---

LL-003: Wrong Column Names

Impact: Runtime errors and broken queries. Required DB schema lookup and code fix after the fact.

What Happened

Two separate column name mistakes were made on different occasions:

Mistake A: amount_cents vs subtotal_cents

Invoices and proposals use subtotal_cents as the column name for the monetary total. The name amount_cents was used in several queries and API responses, which caused Supabase to return null for those fields (Supabase returns null for non-existent selected columns without erroring). The invoices list showed $0.00 for all amounts until the bug was traced to the query.

Mistake B: content_approvals.decision vs .action

The content approvals table stores the reviewer's decision in a column called action. The column name decision was used in a filter and in an INSERT statement. The INSERT silently created rows with decision: null (the column didn't exist and the insert allowed it), and the filter never matched anything.

Root Cause

Column names were assumed from semantic reasoning ("an approval has a decision", "an amount is in cents so it's amount_cents") rather than verified against the actual database schema.

Fix Applied

Corrected all occurrences of amount_cents → subtotal_cents and decision → action. Added correct column names to CLAUDE.md § 6 as a permanent reference.

Prevention

• Always check the DB schema (supabase/migrations/ or Supabase dashboard table editor) before writing queries against a table you haven't worked with recently
• Both column name corrections are documented in CLAUDE.md § 6 (Common Past Mistakes table)
• When in doubt: select * from table_name limit 1 in Supabase SQL Editor to see actual column names

---

LL-004: Admin Stuck in Login Redirect Loop After MFA Enforcement

Impact: Admin user unable to access the CRM for ~30 minutes. Resolved by clearing session state.

What Happened

MFA enforcement was added to the admin authentication flow. An existing admin user who had an active session (session cookie from before MFA was required) attempted to log in. The server detected that MFA was required but not completed, and redirected to the MFA setup page. The MFA setup page checked for an authenticated session, found the old cookie, and redirected back to the dashboard. The dashboard redirected back to MFA setup. The user was stuck in an infinite redirect loop.

Root Cause

The old session cookie satisfied the "is authenticated" check but did not satisfy the new "MFA completed" check. The MFA setup page did not handle this state correctly — it needed to allow access to users who are partially authenticated (have a valid session but haven't completed MFA), not redirect them as if they were fully authenticated.

Fix Applied

• Cleared the user's session cookie manually
• User logged in fresh, completed MFA enrollment
• Added explicit error handling: if a session exists but MFA factor is not verified, redirect to a dedicated /auth/mfa page (not the setup page) with a clear message

Prevention

• Implement a dedicated MFA verification page that is accessible to partially-authenticated users (auth token present but AAL1 only, not AAL2)
• Supabase Auth supports checking aal (Authenticator Assurance Level) on the session — use this to differentiate "not logged in" from "logged in but MFA not completed"
• Test MFA enforcement flows with existing session cookies in staging before deploying to production

---

LL-005: Intermediate Failed Commit Used as Parent

Impact: GitHub push rejected. Required identifying last clean commit and recreating commit tree from scratch.

What Happened

During a complex multi-file change, a first git commit-tree attempt produced a commit with a bad tree (due to duplicate entries — see LL-006). The commit SHA was noted, the tree was fixed, and a second commit was created — but the second commit used the failed first commit as its parent, rather than the last successfully pushed commit.

When the push was attempted, GitHub rejected it with a pack validation error. The push pack contained the bad tree from the intermediate failed commit, because that commit was an ancestor of the new commit.

Root Cause

The assumption was that if a commit was created locally (even if known to be bad), it was safe to use as a parent. This is false: GitHub validates the entire ancestry of pushed objects. If any ancestor commit contains a corrupt or invalid tree, the push is rejected for all commits in that pack.

Fix Applied

1. Identified the last successfully pushed commit SHA (from git log cross-referenced with GitHub commit history)
2. Rebuilt the correct tree from scratch using Python helpers
3. Created a new commit with the clean tree and the clean parent SHA
4. Updated the ref and pushed successfully

Prevention

• CLAUDE.md § 1F explicitly states: "parent MUST be the last successfully pushed SHA" — hardcode it, never assume HEAD is clean
• After any failed commit attempt, do not use that commit as a parent for anything
• Use git log --oneline -3 before every commit to verify what HEAD actually is

---

LL-006: Duplicate Tree Entries from Shell Scripting

Impact: Push rejected by GitHub ("error: object file is empty" or "bad tree"). Required manual tree reconstruction.

What Happened

Git tree entries were built using a shell pipeline: git ls-tree HEAD | grep -v "filename.tsx" | printf "...". The intent was to remove the old entry for a file and add a new one. In practice, the grep -v pattern occasionally failed to filter the entry (due to filename similarity or special characters), and the printf appended a new entry. The result was a tree with two entries for the same filename: the old blob SHA and the new blob SHA.

git mktree accepted this input and created a tree object. git commit-tree accepted the tree. The bad tree only became visible when GitHub validated the push pack.

Root Cause

Shell-based string manipulation of git tree entries is fragile. grep -v pattern matching is not reliable for exact filename filtering, especially when filenames share prefixes. Shell pipelines also don't provide any duplicate-detection feedback.

Fix Applied

Replaced all shell-based tree building with Python helpers that: 1. Parse git ls-tree output into structured entries (mode, type, sha, name) 2. Filter out the old entry by exact name match 3. Append the new entry 4. Sort entries (required by git mktree) 5. Run git mktree --missing with the clean, deduplicated input

def ls_tree(treeish):
    result = subprocess.run(["git", "ls-tree", treeish], capture_output=True, text=True)
    entries = []
    for line in result.stdout.strip().split('\n'):
        if not line.strip():
            continue
        meta, name = line.split('\t', 1)
        mode, typ, sha = meta.split()
        entries.append((mode, typ, sha, name))
    return entries

def mktree(entries):
    lines = '\n'.join(f"{mode} {typ} {sha}\t{name}" for mode, typ, sha, name in entries)
    result = subprocess.run(
        ["git", "mktree", "--missing"],
        input=lines + '\n',
        capture_output=True,
        text=True
    )
    return result.stdout.strip()

# Usage: filter old, append new, sort, build
entries = ls_tree("HEAD:")
entries = [e for e in entries if e[3] != "filename.tsx"]  # remove old
entries.append(("100644", "blob", new_sha, "filename.tsx"))  # add new
entries = sorted(entries, key=lambda x: x[3])  # sort by name
new_tree = mktree(entries)

After building any tree, verify no duplicates before committing:

count=$(git ls-tree <tree_sha> | wc -l)
unique=$(git ls-tree <tree_sha> | awk '{print $4}' | sort | uniq | wc -l)
# count must equal unique

Prevention

• Python-only tree building is now mandated in CLAUDE.md § 1D
• Never use shell grep | printf or similar patterns for tree manipulation
• Always verify tree for duplicates before creating a commit (CLAUDE.md § 1E)

---

LL-007: RLS Policy Gaps

Impact: Security vulnerabilities. CRIT-4 and CRIT-7 were P1 security incidents discovered during internal review. H8 was a P2 discovered during UAT.

What Happened

Three separate RLS (Row Level Security) gaps were found:

CRIT-4 — Notifications table missing tenant isolation

The notifications table was added to support in-app alerts. RLS was enabled on the table, but no tenant_id policy was added — only auth.uid() checks for user-specific notifications. Because broadcast notifications have user_id = NULL, the user-specific policy didn't match them. Any authenticated user could query and read notifications from any tenant.

CRIT-7 — Document storage bucket was public

The Supabase Storage bucket used for client documents was created with public visibility (the default in Supabase Studio if not explicitly set to private). This meant any person with a direct storage URL could download any document — no authentication, no tenant check, no client_id validation.

H8 — Employee cross-employee time log read

The time_logs table had a SELECT policy that allowed any user in the tenant to read all time logs for that tenant. This was appropriate for admin users, but not for employees — an employee should only be able to read their own time logs. The policy was not scoped by role.

Root Cause

All three share the same root cause: RLS policies were not reviewed against the full access matrix (all roles × all operations × tenant isolation) when each table or resource was shipped.

The notifications table was treated as "RLS enabled = secure" without checking whether the policies covered all rows. The storage bucket had an implicit default (public) that was not verified. The time_logs policy was written for the admin case and the employee-specific restriction was not added.

Fix Applied

• CRIT-4: Added tenant_id = get_active_tenant_id() SELECT policy to notifications
• CRIT-7: Changed bucket to private, implemented signed URL generation with client_id validation in the API route, added RLS on the storage object metadata table
• H8: Added a separate SELECT policy for the employee role: employee_id = auth.uid()

Prevention

• Every new table requires an RLS review before shipping: does every role have the correct SELECT/INSERT/UPDATE/DELETE access?
• New storage buckets must be explicitly set to private — never rely on defaults
• The RLS access matrix (roles × tables × operations) should be updated in the security architecture docs whenever a new table or role is added
• See Production Incidents for the full incident records

---

Known Issues

This is the engineering-internal view of tracked bugs, limitations, and accepted risks. It contains more technical detail than the public release notes. Issues are tracked here from discovery through resolution.

---

Section 1: High Severity — Accepted Risks for v1.0

These issues were identified before v1.0 launch. The decision was made to accept the risk and ship, with mitigations noted and target fix versions assigned.

---

HIGH-1: No API Rate Limiting

Field	Detail
ID	HIGH-1
Status	Accepted (v1.0)
Target Fix	v1.1
Severity	High
Discovery	Pre-launch security review

Description

All custom API routes under /api/* are unprotected from brute force, enumeration, and DDoS attacks. There is no per-IP rate limiting, no per-user request throttling, and no global rate cap on the edge functions.

Technical Detail

Supabase Auth (/auth/v1/*) has built-in rate limiting (60 requests/hour per IP by default, configurable). This protects login and signup endpoints. However, all application API routes (/api/admin/*, /api/portal/*, /api/webhook/*) have no equivalent protection. An attacker can make unlimited requests to these endpoints.

Specific risks: - Enumeration of client IDs or lead IDs via sequential requests to /api/admin/clients/[id] - High-volume requests to report generation endpoints causing compute cost spikes - Credential stuffing if any endpoint accepts credentials directly

Current Mitigation

Vercel's infrastructure provides some DDoS protection at the edge layer. requireAdminWithTenant() protects admin routes from unauthenticated access, reducing the attack surface for enumeration. Webhook endpoints validate Stripe signatures, providing implicit rate limiting on webhook handlers.

Workaround

None — this is a gap, not a workaround situation. Monitor Vercel function logs for unusual request volume.

Planned Fix

Implement Upstash Redis rate limiting using @upstash/ratelimit on the Vercel edge. Apply rate limiting middleware to all /api/* routes. Target: v1.1.

---

HIGH-2: No PDF Generation

Field	Detail
ID	HIGH-2
Status	Accepted (v1.0)
Target Fix	v1.1
Severity	High
Discovery	UX review

Description

Invoice and contract "Download as PDF" features show the document as HTML in a new browser tab, not as a generated PDF. No PDF generation library is integrated.

Technical Detail

The edge runtime (export const runtime = 'edge') is incompatible with Node.js PDF libraries like puppeteer, pdfkit, or pdf-lib that rely on Node.js APIs. A PDF generation solution for edge runtime requires either: - A dedicated serverless function (non-edge) for PDF generation - A third-party PDF rendering service (e.g., Browserless, Gotenberg) - A client-side PDF library (e.g., jsPDF + html2canvas)

Current Mitigation

None for download. The admin portal shows a print-optimized view of invoices and contracts. Users can use browser "Print → Save as PDF" as a manual workaround.

Workaround

Browser: File → Print → Change destination to "Save as PDF". Works on all modern browsers.

Planned Fix

Evaluate @react-pdf/renderer (works in edge as it generates PDF from React components, no Node.js APIs needed) or a dedicated serverless function for PDF rendering using puppeteer-core + Chromium. Target: v1.1.

---

HIGH-3: No Audit Log

Field	Detail
ID	HIGH-3
Status	Accepted (v1.0)
Target Fix	v1.2
Severity	High
Discovery	Architecture review

Description

No audit trail exists for admin actions. Create, update, and delete operations on any record (clients, invoices, contracts, proposals, users, etc.) are not logged. There is no way to reconstruct "who did what, when" after the fact.

Technical Detail

Without an audit log: - Cannot investigate data corruption or accidental deletion - Cannot demonstrate compliance with any accountability requirement - Cannot debug "why did this record change" issues

Current Mitigation

Supabase database logs (PostgreSQL-level) capture query activity, but these are not retained long-term on the free tier and are not queryable in a structured way. Vercel function logs capture API request details but not payload content.

Workaround

For critical changes, Supabase's updated_at timestamps provide minimal breadcrumbing. No full workaround available.

Planned Fix

Implement an audit_logs table with columns: id, tenant_id, user_id, action (create/update/delete), resource_type, resource_id, before_json, after_json, created_at. Populate via a helper function called in all admin API route handlers. Target: v1.2.

---

HIGH-4: Silent Email Failures

Field	Detail
ID	HIGH-4
Status	Accepted (v1.0)
Target Fix	v1.1
Severity	High
Discovery	Code review

Description

When Resend fails to send an email (API error, rate limit, invalid recipient, etc.), the error is caught and logged to the Vercel function console, but no retry is attempted and no alert is raised. The admin is not notified of the failure, and the client never receives the email.

Technical Detail

Current pattern in email sending code:

try {
  await resend.emails.send({ ... })
} catch (error) {
  console.error('Email send failed:', error)
  // ← no retry, no alert, execution continues
}

This means invoice emails, contract notification emails, and portal invitation emails can fail silently.

Current Mitigation

Admins can manually resend emails from the admin panel (invoice detail page → Resend Email button, client record → Resend Invite). Resend dashboard shows sending history and errors.

Workaround

If a client reports not receiving an email, admin navigates to the relevant record and manually resends. Check Resend dashboard → Logs for delivery details.

Planned Fix

Implement a email_queue table: insert email jobs into the queue, process with a cron job, update status to sent/failed, retry up to 3 times with exponential backoff, alert admin on final failure via in-app notification. Target: v1.1.

---

HIGH-5: Recurring Invoice Emails Not Sent

Field	Detail
ID	HIGH-5
Status	Accepted (v1.0)
Target Fix	v1.1
Severity	High
Discovery	QA testing

Description

Auto-generated recurring invoices are created correctly in the database by the cron job, but no email notification is sent to the client. The client must be manually notified by the admin.

Technical Detail

The cron job at /api/admin/invoices/cron creates the invoice record in Supabase and marks it as sent, but does not call the email sending function. The email sending logic was only wired up in the manual "Send Invoice" admin action.

Current Mitigation

Admins must periodically check for newly-generated recurring invoices and manually send them. The invoice list shows created_at for each invoice — filter by today to find new ones.

Workaround

Admin: Invoices → filter by today's date → send any "Recurring" invoices that show "Sent" status but were created today (they were not actually emailed).

Planned Fix

Update the cron handler to call the invoice email function after creating the recurring invoice record. This is blocked on HIGH-4 (email reliability) — fix both together in v1.1.

---

HIGH-6: Signed URL 1-Hour Expiry

Field	Detail
ID	HIGH-6
Status	Accepted (v1.0)
Target Fix	v1.1
Severity	High
Discovery	UAT

Description

Supabase Storage signed URLs for document downloads expire after 1 hour. If a user loads a page with document links and then attempts to download more than 1 hour later, the link is expired and the download fails with a 400 error.

Technical Detail

Signed URLs are generated at page load time with a 3600-second TTL. For users who keep tabs open for extended periods, or for links sent via email, the URL will expire. The error message from Supabase is generic ("Object not found") which is confusing.

Current Mitigation

None — users can refresh the page to generate a new signed URL.

Workaround

Refresh the page. A new signed URL is generated on page load.

Planned Fix

Option A: Increase signed URL TTL to 24 hours (acceptable for documents behind auth — the URL itself is the credential, and the risk window is acceptable). Option B: Generate signed URLs on-demand via an API call when the user clicks download, rather than at page load time. Prefer Option B for security. Target: v1.1.

---

HIGH-7: No TOTP Recovery Codes

Field	Detail
ID	HIGH-7
Status	Accepted (v1.0)
Target Fix	v1.1
Severity	High
Discovery	Security review

Description

When users enroll in TOTP-based MFA, no recovery codes are generated or displayed. If a user loses access to their authenticator app (device lost, stolen, or replaced), they cannot log in and have no recovery path without admin intervention.

Technical Detail

Supabase Auth's MFA implementation supports TOTP enrollment but does not natively generate NIST-standard recovery codes. Recovery codes would need to be generated at enrollment time, shown once to the user, stored hashed in the database, and accepted as an alternative factor during login.

Current Mitigation

An admin with super_admin role can deactivate the locked-out user account, create a new invitation, and the user re-enrolls from scratch. This is disruptive and requires admin action.

Workaround

User: contact admin. Admin: Settings → Team Management → deactivate user → resend invitation → user re-enrolls MFA.

Planned Fix

Generate 10 single-use recovery codes at MFA enrollment. Display once with clear "save these" instructions. Store as bcrypt hashes in a mfa_recovery_codes table (columns: user_id, code_hash, used_at). Validate during login as an alternative to TOTP. Target: v1.1.

---

Section 2: Medium Severity — Deferred

These issues are real but do not block core workflows. They are logged here and will be prioritized in a future release.

ID	Description	Workaround	Target
MED-1	Client portal has no search functionality — clients cannot search their invoices, projects, or documents	Manual scrolling/filtering	v1.2
MED-2	Proposal content field renders raw Markdown in the PDF preview on Safari — full-text Markdown shown instead of rendered HTML	Use Chrome for proposal previews	v1.1
MED-3	Time log CSV export not available for employees — only accessible in admin reports module	Admin exports on employee's behalf	v1.2
MED-4	Notification badge count does not update in real-time — requires page refresh	Refresh page	v1.2
MED-5	Contract signature date is stored as server timestamp, not client-reported timestamp — can mismatch by timezone	Acceptable for v1.0 legal purposes	v1.2

---

Section 3: Low Severity / Cosmetic

ID	Description	Target
LOW-1	Mobile sidebar does not auto-close after navigation on iOS — user must tap the close button	v1.1
LOW-2	Dark mode not implemented — theme-aware colors used throughout but no dark mode toggle exposed to users	v1.2
LOW-3	Empty state illustrations are placeholder SVGs — not branded	v1.1
LOW-4	Admin table pagination resets to page 1 after any action (delete, status change)	v1.2
LOW-5	Long client company names overflow the sidebar nav without truncation on some viewport sizes	v1.1

---

Resolved Issues (v1.0)

ID	Description	Resolution
CRIT-4	Cross-tenant notification leak via missing RLS	Added tenant_id RLS policy to notifications table
CRIT-7	Client document access via direct storage URL	Made bucket private, implemented signed URLs with client validation
H8	Employee cross-employee time log read	Added employee_id = auth.uid() policy for employee role
CRIT-1	Employee invite saved with 'client' role	Fixed role assignment in invite handler

---

Vendor Notes

Operational notes, quirks, limits, and gotchas for each vendor used in the SONAN DIGITAL CRM. These are things you learn by running the stack in production — not things you find in the getting-started docs.

---

Supabase

Supabase provides PostgreSQL database, Auth, Storage, and Realtime for the CRM.

Connection

- Pooled URL (use this): `postgresql://postgres.[ref]:[password]@aws-0-[region].pooler.supabase.com:6543/postgres`
- Direct URL (migrations only): `postgresql://postgres.[ref]:[password]@db.[ref].supabase.co:5432/postgres`

The DATABASE_URL environment variable in Vercel should always be the PgBouncer URL. The direct URL is only needed for supabase db push (migrations) because PgBouncer doesn't support all PostgreSQL protocol features needed by the migration runner.

Auth

Limit / Behavior	Detail
Rate limit	60 requests/hour per IP on /auth/v1/* (configurable in Supabase dashboard → Auth → Rate Limits)
Magic link expiry	1 hour by default (configurable)
PKCE flow	Required for server-side auth in Next.js App Router — use @supabase/ssr package
Session refresh	createClient() from @supabase/ssr automatically refreshes sessions via cookie; service client does not
MFA AAL levels	AAL1 = password only, AAL2 = password + MFA factor. Use session.aal to differentiate

```typescript
// CORRECT — server component
import { createClient } from '@/lib/supabase/server'
const supabase = await createClient()
const { data: { user } } = await supabase.auth.getUser()

// WRONG — returns null user in server components
import { createServiceClient } from '@/lib/supabase/service'
const supabase = createServiceClient()
const { data: { user } } = await supabase.auth.getUser() // ← always null
```

Storage

• Buckets are either public or private at the bucket level — there is no per-object visibility setting
• Public buckets serve files without authentication via a direct URL — never put sensitive files in a public bucket
• Private buckets require signed URLs — generate with supabase.storage.from(bucket).createSignedUrl(path, expirySeconds)
• Default signed URL expiry: you set it in the createSignedUrl call — there is no global default in the SDK
• Storage RLS policies apply to the storage.objects table — write them like any other Supabase RLS policy
• Storage object paths are bucket_name/folder/file.ext — the bucket name is not part of the path in RLS policies, the bucket is identified by the bucket_id column

Migrations

# Apply migrations to linked project
supabase db push

# Regenerate TypeScript types from DB schema
supabase gen types typescript --linked > src/types/supabase.ts

# Reset local DB and re-run all migrations
supabase db reset

# Check which migrations have been applied
supabase migration list

PITR and Backups

• Free tier: daily automated backups, 7-day retention. No point-in-time recovery.
• Pro tier: PITR available with continuous WAL archiving. Required if you need to recover to a specific minute.
• Backups can be downloaded from Supabase dashboard → Project Settings → Database → Backups.
• For v1.0, daily backups are accepted (see risk register). Upgrade to Pro tier before any client with a contractual data recovery SLA.

Known SDK Quirks

• Nested FK joins return arrays — even for many-to-one relationships, FK joins always return Array<T>. Always define interfaces with array shape and access with [0]. See Lessons Learned LL-002.
• .single() throws on no rows — supabase.from('table').select().eq('id', id).single() throws a PostgrestError if no row is found. Use .maybeSingle() if the row may not exist.
• RLS errors surface as empty results — if RLS blocks a query, Supabase returns [] (not an error). If you're getting unexpectedly empty results, check RLS policies first.
• auth.uid() in RLS — returns the UUID of the authenticated user from the JWT. For service role operations, auth.uid() returns null — service role bypasses RLS entirely.

---

Vercel

Vercel hosts the Next.js application and runs the edge functions and cron jobs.

Edge Runtime Constraints

Use Web API equivalents instead:
- `crypto` → `crypto.subtle` (Web Crypto API, available in edge)
- File reading → not applicable (use Supabase Storage)
- `Buffer` → `Uint8Array` or `TextEncoder`/`TextDecoder`

Function Limits

Limit	Value
Edge function size	1 MB (gzipped)
Edge function execution time	30 seconds (configurable up to 300s on Pro)
Serverless function size	50 MB
Cold start — edge	~0ms (runs at CDN edge, always warm)
Cold start — serverless	~200ms typical

Environment Variables

• Environment variables set in Vercel dashboard are available at runtime via process.env
• They are not available in next.config.ts at build time unless explicitly added with the env: config key
• NEXT_PUBLIC_* variables are exposed to the browser — never put secrets in NEXT_PUBLIC_ variables
• After changing an env var in Vercel, you must trigger a redeploy for the change to take effect

// Available at runtime in edge functions
const key = process.env.STRIPE_SECRET_KEY // ✓

// NOT available at build time without next.config.ts env: key
// next.config.ts env: key is required for build-time access

Cron Jobs

• vercel.json crons array defines scheduled jobs
• Free tier: maximum 2 cron jobs
• Pro tier: unlimited cron jobs
• Minimum cron interval: 1 minute
• Cron jobs call an HTTP endpoint in your app — protect with CRON_SECRET in Authorization header
• Cron logs visible in Vercel dashboard → Project → Cron tab
• Crons run in UTC timezone

// vercel.json
{
  "crons": [
    {
      "path": "/api/admin/appointments/cron",
      "schedule": "0 8 * * *"
    }
  ]
}

Log Retention

Plan	Log Retention
Free	7 days
Pro	30 days
Enterprise	Configurable

For debugging production incidents older than 7 days on the free plan, logs are gone. Consider Sentry (already integrated) for error retention — Sentry retains errors for 30+ days on free tier.

Deployment

• Every push to main triggers a production deployment
• Every PR gets a preview deployment at a unique URL
• Rollback: Vercel dashboard → Deployments → select a previous deployment → Promote to Production
• Build cache: Vercel caches .next build artifacts — clear with "Redeploy without cache" if build artifacts seem stale

---

Stripe

Stripe handles payment processing, invoice payment links, and recurring billing triggers.

Key vs Mode Rules

Always verify which key is in each Vercel environment (preview vs production).

Webhook Security

Always verify webhook signatures. Never trust the raw webhook body without verification:

const sig = req.headers.get('stripe-signature')
const body = await req.text()

let event: Stripe.Event
try {
  event = stripe.webhooks.constructEvent(body, sig!, process.env.STRIPE_WEBHOOK_SECRET!)
} catch (err) {
  return NextResponse.json({ error: 'Invalid signature' }, { status: 400 })
}

Checkout and Payment Links

• Stripe Checkout sessions expire after 24 hours — if a client doesn't pay within 24 hours, the session expires and the link stops working. The invoice in the CRM remains in sent status.
• There is no webhook event for an expired Checkout session — you must check invoice age and send a reminder or generate a new payment link.
• Payment links (as opposed to Checkout sessions) do not expire and can be reused.

Webhook Retry Behavior

• Stripe retries failed webhook deliveries for up to 3 days with exponential backoff
• If your webhook handler returns a non-2xx status, Stripe retries
• If your handler takes more than 30 seconds, Stripe considers the delivery failed and retries
• Idempotency is critical: your webhook handler will be called more than once for the same event on retries. Use the Stripe event ID to deduplicate:

// Check if this event was already processed
const existing = await supabase
  .from('processed_stripe_events')
  .select('id')
  .eq('stripe_event_id', event.id)
  .maybeSingle()

if (existing.data) {
  return NextResponse.json({ received: true }) // already processed
}

Idempotency Keys

Use Stripe idempotency keys when creating payment intents or charges to prevent duplicate charges on network retry:

const paymentIntent = await stripe.paymentIntents.create(
  { amount: cents, currency: 'usd', ... },
  { idempotencyKey: `invoice-${invoiceId}-attempt-1` }
)

---

Resend

Resend sends transactional emails from the CRM (invoice notifications, portal invitations, contract alerts).

Domain Verification

• You must verify your sending domain in the Resend dashboard before sending from a custom domain
• Until verified, you can only send from @resend.dev (for testing only)
• Verification requires adding DNS records (MX, SPF, DKIM) to your domain
• Verification can take up to 48 hours to propagate

Free Tier Limits

Limit	Value
Emails per month	3,000
Emails per day	100
Custom domains	1
Team members	1

At current CRM email volume (invitations, invoices, contract notifications), 3,000/month and 100/day is adequate for early operation. Monitor Resend dashboard → Overview for usage against limits.

No Native Retry Queue

From Address Requirements

• The from address must use a verified domain — cannot send from gmail.com, yahoo.com, etc.
• Format: "SONAN Digital <noreply@yourdomain.com>"
• Using an unverified domain causes resend.emails.send() to throw 422 Unprocessable Entity

Delivery Status

• The resend.emails.send() response includes an email id and initial delivery status
• Resend does not guarantee delivery confirmation in the API response — the response confirms the email was accepted for sending, not that it was delivered
• For delivery confirmation, register a Resend webhook in the Resend dashboard → Webhooks → add endpoint
• Webhook events: email.sent, email.delivered, email.bounced, email.complained

Logs and Debugging

• Resend dashboard → Logs shows all sending activity with status, timestamp, recipient, and any error
• Logs are available for 30 days on the free tier
• If a client reports not receiving an email, check Resend logs first before resending — the email may have been delivered and marked spam, or the address may have bounced

---

Troubleshooting Guide

Step-by-step diagnosis for the most common errors and failures in the SONAN DIGITAL CRM. Each entry follows the pattern: Problem → Cause → Fix.

For quick-reference tables, see Common Fixes. For vendor-specific quirks, see Vendor Notes.

---

Git / FUSE Issues

"unable to unlink tmp_obj_*" during git hash-object

Symptoms

warning: unable to unlink tmp_obj_XXXXXX: Operation not permitted

Cause

This is a FUSE filesystem permission warning triggered when git tries to rename a temporary object file to its final location. It is a cosmetic warning — the blob is written successfully despite the warning.

Fix

Verify the blob was written correctly:

git cat-file -t <sha>
# Should return: blob

If it returns blob, the hash succeeded. Ignore the warning and continue. If git cat-file returns an error, the blob was not written — re-run the hash-object step.

---

Push rejected: "bad tree" or "packfile has broken object"

Symptoms

error: object file /path/.git/objects/... is empty
remote: error: bad tree object <sha>
error: packfile invalid

Cause

One of two scenarios:

1. Duplicate tree entries: a tree object was built with two entries for the same filename (e.g., both the old and new blob SHA for page.tsx). Git creates the tree object, but GitHub rejects it on push.
2. Bad parent commit: an intermediate failed commit (containing a bad tree) was used as the parent for a new commit. The push pack includes the bad tree from the ancestor.

Fix

Step 1: Identify the last commit that was successfully pushed to GitHub. Do not trust local HEAD.

git log --oneline -5
# Cross-reference with GitHub commit history
CLEAN_SHA="<last_pushed_sha>"

Step 2: Identify whether the new commit's tree is clean:

# Check for duplicate entries in the new commit's root tree
NEW_TREE=$(git cat-file -p <new_commit_sha> | grep tree | awk '{print $2}')
count=$(git ls-tree $NEW_TREE | wc -l)
unique=$(git ls-tree $NEW_TREE | awk '{print $4}' | sort | uniq | wc -l)
echo "Total: $count, Unique: $unique"
# Must be equal — if not, tree has duplicates

Step 3: Rebuild the commit tree from scratch using Python helpers (see CLAUDE.md § 1D). Use CLEAN_SHA as the parent — never the failed commit.

Step 4: Verify the push pack is clean:

git rev-list --objects <new_commit> ^<CLEAN_SHA> | grep <any_suspected_bad_sha>
# Must return nothing

---

Blob line count doesn't match file line count after hash-object

Symptoms

After running the FUSE-safe commit pattern:

git cat-file -p <sha> | wc -l   # returns 150
wc -l /tmp/filename.tsx          # returns 300

Cause

The file was hashed from the FUSE mount path, not from /tmp. Or the Python f.write() step was writing to the FUSE path (not /tmp) and the write was truncated.

Fix

1. Re-check where the file content was written — it must be /tmp/filename.tsx, never the FUSE workspace path
2. Re-write the file to /tmp using Python:

content = r"""...full file content..."""
with open('/tmp/filename.tsx', 'w') as f:
    f.write(content)

1. Re-hash from /tmp:

import subprocess
with open("/tmp/filename.tsx", "rb") as f:
    data = f.read()
result = subprocess.run(
    ["git", "hash-object", "-w", "--stdin"],
    input=data, capture_output=True
)
sha = result.stdout.decode().strip()

1. Verify again — both counts must match exactly before proceeding.

---

Authentication Issues

"Unauthorized" on admin API routes

Symptoms

API route returns { "error": "Unauthorized" } with HTTP 401, or a redirect to /auth/login.

Diagnosis steps

1. Check the session cookie exists: browser DevTools → Application → Cookies → look for sb-*-auth-token cookie for your Supabase project URL. If missing, user is not logged in.

2. Check the user's role in the database:

SELECT id, email, role FROM users WHERE email = 'user@example.com';

The requireAdminWithTenant() function accepts roles admin and super_admin. If the role is employee or client, the auth check fails.

1. Check the route handler:

export async function GET(req: Request) {
  const caller = await requireAdminWithTenant()
  if (!caller) return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
  // If this line is reached, caller.userId and caller.tenantId are set
}

If requireAdminWithTenant() returns null, it logs the reason to the Vercel function console. Check Vercel logs for the corresponding request.

1. Verify tenant association: the user must be associated with an active tenant. Check the tenant_users table:

SELECT * FROM tenant_users WHERE user_id = '<user_uuid>';

Fix

Root cause	Fix
Not logged in	User logs in again
Wrong role	Update users.role in Supabase dashboard or via admin panel
No tenant association	Insert row into tenant_users
Expired session	Clear cookies, re-login

---

Client cannot log into the portal

Symptoms

Client reports they cannot access the portal, magic link doesn't work, or page shows an error.

Diagnosis

1. Check invitation email delivery: Resend dashboard → Logs → search for client's email address. Was the email delivered?

2. Check magic link expiry: Supabase magic links expire after 1 hour. If the client received the email more than 1 hour ago, the link is expired.

3. Check if the client user exists in Supabase Auth: Supabase dashboard → Authentication → Users → search for the client's email.

4. Check the client record status: in the CRM admin panel, is the client's portal access enabled?

Fix

If the invitation link expired or was never received: Admin → Clients → open client record → Contacts → Resend Portal Invitation.

If the user exists in Auth but can't log in: check if their email is confirmed in Supabase Auth. If not confirmed, resend the invitation (this resends the confirmation/magic link).

---

MFA device lost — user locked out

Symptoms

User cannot log in because their TOTP app is gone (new phone, deleted app, etc.).

Fix

1. Admin → Settings → Team Management → find the user
2. Deactivate the user account
3. Create a new invitation for the same email address
4. User receives a fresh invitation email and re-enrolls in MFA from scratch

---

TypeScript / Build Errors

"Property 'name' does not exist on type '{ id: string; name: string; }[]'"

Symptoms

TypeScript error on a Supabase query result: a property access that should work according to the interface is failing because the type is actually an array.

Cause

Supabase FK joins always return arrays, even for many-to-one relationships. The interface was written as a single object but the runtime value is an array.

Fix

Update the interface to use array types and add [0] indexing everywhere:

// BEFORE (wrong)
interface TaskRow {
  id: string
  title: string
  projects: { id: string; name: string } | null  // ← wrong, this is an array
}
// Usage: task.projects?.name  ← TypeScript error

// AFTER (correct)
interface TaskRow {
  id: string
  title: string
  projects: Array<{ id: string; name: string }>  // ← matches actual shape
}
// Usage: task.projects?.[0]?.name  ← correct

---

"fs module not found" or "path module not found"

Symptoms

Build error: Module not found: Can't resolve 'fs' or similar for Node.js built-ins.

Cause

A file or imported package is using a Node.js built-in module. Edge runtime does not support Node.js APIs.

Fix

1. Find the import: grep -r "from 'fs'" src/ or grep -r "require('fs')" src/
2. Remove the import and replace with a Web API equivalent, or
3. If the import is from an npm package, find an edge-compatible alternative

Common replacements:

Node.js API	Edge Runtime Alternative
crypto.createHash()	crypto.subtle.digest()
Buffer.from()	new TextEncoder().encode()
fs.readFile()	Not applicable — use Supabase Storage

---

"amount_cents column not found" or column returns null unexpectedly

Symptoms

Invoice or proposal amount_cents returns null. Or TypeScript error on a DB column that should exist.

Cause

Wrong column name. The correct columns are:

• Invoices: subtotal_cents (not amount_cents)
• Proposals: subtotal_cents (not amount_cents)
• Content approvals decision: action (not decision)

Fix

Search for the wrong column name and replace:

grep -r "amount_cents" src/
grep -r "\.decision" src/

Replace amount_cents → subtotal_cents and .decision → .action.

---

Stripe Issues

Stripe webhook not received / invoice not marked paid after payment

Symptoms

Client pays via Stripe Checkout. Invoice status stays sent. No webhook event in Vercel logs.

Diagnosis

1. Check Stripe dashboard → Developers → Webhooks → your endpoint: look at "Recent deliveries". Was the event sent? What was the response code?

2. Check the webhook endpoint URL: is it pointing to the correct Vercel production URL (not a preview URL or localhost)?

3. Check STRIPE_WEBHOOK_SECRET: in Vercel env vars, is this the correct secret for the registered endpoint? Test and live mode each have different secrets.

4. Check Vercel function logs for the webhook route (/api/webhook/stripe): was the handler called? Did it throw an error?

Fix by root cause

Root Cause	Fix
Wrong webhook URL in Stripe	Update endpoint URL in Stripe dashboard
Wrong STRIPE_WEBHOOK_SECRET	Copy whsec_* from Stripe dashboard → Webhooks → endpoint → Signing secret
Handler throwing on signature verification	Ensure req.text() is used (not req.json()) — body must be raw string for signature verification
Handler returning non-2xx	Check and fix the handler error; re-deliver event from Stripe dashboard
Test vs live key mismatch	Verify all Stripe env vars are consistently test or live

Manual recovery

If the payment was received but the webhook failed: Stripe dashboard → Webhooks → endpoint → Recent deliveries → find the checkout.session.completed event → Resend. Or manually update the invoice status in Supabase.

---

Cron Job Issues

Cron job not running

Symptoms

Overdue invoices are not being marked overdue. Recurring invoices are not being created. Cron tab in Vercel shows no recent runs.

Diagnosis

1. Check vercel.json: does the crons array have the correct path and schedule?

{
  "crons": [
    { "path": "/api/admin/appointments/cron", "schedule": "0 8 * * *" }
  ]
}

1. Check the cron handler auth: the handler validates Authorization: Bearer ${CRON_SECRET}. Verify CRON_SECRET is set in Vercel env vars and the handler checks it correctly.

2. Check Vercel plan: free tier supports maximum 2 cron jobs. If you have more than 2 defined, extras may be silently dropped.

3. Manually trigger: test the cron endpoint directly:

curl -X POST https://your-app.vercel.app/api/admin/appointments/cron \
  -H "Authorization: Bearer $CRON_SECRET"

Check the response for errors.

Fix

Root Cause	Fix
vercel.json misconfigured	Fix schedule or path; redeploy
CRON_SECRET mismatch	Update env var in Vercel; redeploy
Too many crons on free plan	Upgrade to Pro, or combine cron handlers
Handler error	Check Vercel function logs; fix the handler

---

Document Download Issues

Document download link expired (400 error)

Symptoms

Clicking a document download link shows a 400 error from Supabase Storage, or a "Object not found" / "expired" error message.

Cause

Supabase signed URLs expire after the TTL set when the URL was generated (typically 3600 seconds / 1 hour). The URL was generated at page load time and has since expired.

Fix (immediate)

Refresh the page. A new signed URL is generated on page load.

Fix (long-term)

See Known Issue HIGH-6 — the planned fix is to generate signed URLs on-demand at download time, not at page load time.

---

Email Issues

Client not receiving email

Symptoms

Client reports they never received an invoice email, portal invitation, or contract notification.

Diagnosis steps

1. Check Resend dashboard → Logs → search for client email address → check status
2. Status codes:
3. Delivered — email reached the mail server (check spam folder)
4. Bounced — email address is invalid or rejected (hard bounce = permanent, soft bounce = temporary)
5. Complained — client marked a previous email as spam (Resend stops sending to this address)
6. No log entry — email was never sent from the CRM (check Vercel logs for errors)

Fix by root cause

Root Cause	Fix
Email sent but in spam	Ask client to check spam; mark as not spam
Email bounced (hard)	Update the email address in the client record
Resend delivery error	Check Vercel function logs; resend manually from admin panel
Email never sent (code error)	Check Vercel logs for the send attempt; fix the code path
Rate limit hit	Check Resend dashboard for daily limit (100/day on free tier)

---

Common Fixes

Quick-reference tables for the most frequent issues. Designed to be scanned fast — for full diagnosis steps, see Troubleshooting.

---

Auth Issues

Symptom	Quick Fix
"Unauthorized" on admin API	Check user role in DB (users.role must be admin or super_admin); verify session cookie exists (DevTools → Application → Cookies); check requireAdminWithTenant() is in route handler
Client can't log into portal	Admin panel → client record → Resend Portal Invitation
MFA device lost	Admin → Settings → Team Management → deactivate user → resend invitation → user re-enrolls
Session expired redirect loop	Clear all cookies for the domain; re-login from scratch
"Invalid login credentials"	Verify email is confirmed in Supabase Auth dashboard; check if user account exists
Admin invited as wrong role	Update users.role in Supabase dashboard SQL editor: UPDATE users SET role = 'admin' WHERE email = '...'

---

RLS Issues

Symptom	Quick Fix
User sees another tenant's data	Check tenant_id filter in RLS policy AND in API route query — both must filter by tenant
"new row violates row-level security policy"	Check INSERT RLS policy; verify tenant_id is included in INSERT values and matches get_active_tenant_id()
Empty results when data should exist	RLS is silently blocking — check SELECT policy for the table; test with service role client to confirm data exists
Employee sees all time logs	Apply H8 fix: add SELECT policy for employee role: USING (employee_id = auth.uid())
Portal client sees other clients' data	Check client_id filter in portal RLS policy; verify get_portal_client_id() returns correct value
Storage file returns 403	Check bucket is private; verify signed URL is generated correctly; check storage RLS on storage.objects

---

Build & TypeScript Errors

Error	Quick Fix
Module not found: Can't resolve 'fs'	Remove Node.js import; use Web APIs (edge runtime has no Node.js APIs)
Property 'x' does not exist on type '{ x: string }[]'	Supabase FK join is an array — change interface to Array<{...}> and access with [0]
Missing env var at runtime	Add to Vercel environment variables (correct environment: production/preview/development); trigger redeploy
amount_cents column not found or returns null	Use subtotal_cents — that is the correct column name on invoices and proposals
decision column not found or returns null	Use action — that is the correct column name on content_approvals
TypeScript error on Supabase response	Run supabase gen types typescript --linked > src/types/supabase.ts to regenerate types after migration
export const runtime = 'edge' missing	Add to every route file and page file that does server-side work

---

Stripe Issues

Symptom	Quick Fix
Webhook not received	Check Stripe dashboard → Webhooks → endpoint URL is correct production URL (not preview); check STRIPE_WEBHOOK_SECRET matches the endpoint
Invoice not marked paid	Re-deliver checkout.session.completed event from Stripe dashboard; or manually update invoice status in Supabase
No signatures found matching the expected signature	Handler must read body as raw text (req.text()), not parsed JSON (req.json())
Test payments failing in production	Check all Stripe keys are live keys (sk_live_*, pk_live_*), not test keys
Duplicate charge on retry	Use idempotency key when creating payment intents

---

Email Issues

Symptom	Quick Fix
Client not receiving email	Check Resend dashboard → Logs for delivery status; if delivered, ask client to check spam
Email bounced	Update email address in client record; resend
"From address not verified" error	Verify sending domain in Resend dashboard; until verified, can only send from @resend.dev
Silent email failure (no error in UI)	Check Vercel function logs for console.error('Email send failed:') entries
Recurring invoice email not sent	Known issue (HIGH-5) — admin must manually send from invoice detail page

---

Cron Job Issues

Symptom	Quick Fix
Cron not running	Check vercel.json crons config; verify CRON_SECRET env var is set; check Vercel dashboard → Cron tab
Cron runs but does nothing	Check Vercel function logs for the cron route; test manually with curl using Authorization: Bearer $CRON_SECRET
"Unauthorized" on manual cron trigger	Header must be Authorization: Bearer <CRON_SECRET> — exact match, case-sensitive
Overdue invoices not updating	Manually trigger: curl -X POST .../api/admin/invoices/cron -H "Authorization: Bearer $CRON_SECRET"

---

Document / Storage Issues

Symptom	Quick Fix
Download link expired (400 error)	Refresh the page — signed URL regenerates on page load
Upload fails silently	Check Vercel function logs; check Supabase Storage → bucket capacity; check storage RLS
Document shows to wrong client	Check client_id validation in signed URL generation API route
Storage URL returns 404	Verify file path is correct; check file exists in Supabase Storage dashboard

---

FUSE-Safe Commit — Quick Reference

Use this every time you commit to the FUSE-mounted repository. Full pattern is in CLAUDE.md.

Step 1 — Write to /tmp using Python (never to the FUSE path)

content = r"""...full file content here..."""
with open('/tmp/filename.tsx', 'w') as f:
    f.write(content)

Step 2 — Hash from /tmp

import subprocess
with open("/tmp/filename.tsx", "rb") as f:
    data = f.read()
result = subprocess.run(
    ["git", "hash-object", "-w", "--stdin"],
    input=data, capture_output=True
)
sha = result.stdout.decode().strip()
print(sha)

Step 3 — Verify blob line count (mandatory)

# Both numbers MUST match exactly
git cat-file -p <sha> | wc -l
wc -l /tmp/filename.tsx

Step 4 — Build tree with Python helpers

def ls_tree(treeish):
    result = subprocess.run(["git", "ls-tree", treeish], capture_output=True, text=True)
    entries = []
    for line in result.stdout.strip().split('\n'):
        if not line.strip(): continue
        meta, name = line.split('\t', 1)
        mode, typ, sha = meta.split()
        entries.append((mode, typ, sha, name))
    return entries

def mktree(entries):
    lines = '\n'.join(f"{mode} {typ} {sha}\t{name}" for mode, typ, sha, name in entries)
    result = subprocess.run(["git", "mktree", "--missing"], input=lines+'\n', capture_output=True, text=True)
    return result.stdout.strip()

entries = ls_tree("HEAD:")
entries = [e for e in entries if e[3] != "filename.tsx"]  # remove old
entries.append(("100644", "blob", new_sha, "filename.tsx"))  # add new
entries = sorted(entries, key=lambda x: x[3])  # sort by name
new_tree = mktree(entries)

Step 5 — Verify no duplicate entries

count=$(git ls-tree <tree_sha> | wc -l)
unique=$(git ls-tree <tree_sha> | awk '{print $4}' | sort | uniq | wc -l)
echo "$count total, $unique unique"
# Must be equal

Step 6 — Create commit (parent = last PUSHED SHA, not HEAD)

# Always hardcode the last successfully pushed SHA
PARENT="<last_pushed_sha>"
git commit-tree <tree_sha> -p $PARENT -m "feat: your commit message"

Step 7 — Update ref

with open('.git/refs/heads/main', 'w') as f:
    f.write('<new_commit_sha>\n')

Step 8 — User pushes from Windows terminal

git push origin <new_commit_sha>:refs/heads/main

---

Standard Operating Procedures

Repeatable, step-by-step procedures for common operational tasks. Follow these exactly — they encode hard-won knowledge about the correct order of operations.

---

SOP-001: Deploy to Production

Trigger: Feature or fix is ready to ship.

Prerequisites: All changes are on the dev branch (or a feature branch merged into dev).

Steps

1. Ensure all tests pass on the branch being merged.
2. Run TypeScript build: pnpm build or npm run build

3. Fix any TypeScript errors before proceeding

4. Create a Pull Request from dev → main in GitHub.

5. Title: feat: description or fix: description following conventional commits

6. Description: what changed and why; link to any related issues

7. Get code review approval from at least one other engineer.

8. Reviewer checks: no secrets hardcoded, RLS policies reviewed if new tables, edge runtime constraints respected

9. Merge the PR once approved.

10. Merging to main automatically triggers a Vercel production deployment

11. Do not merge if CI checks are failing

12. Monitor the Vercel deployment for ~5 minutes.

13. Vercel dashboard → Deployments → watch the build log for errors

14. If build fails: investigate build log, fix, push to dev, PR again

15. Monitor Sentry for 15 minutes after successful deployment.

16. Sentry dashboard → Issues → filter to last 15 minutes

17. Any new errors? Investigate immediately — consider rollback if errors are user-impacting

18. Verify key user flows in production manually.

19. Log into admin panel as an admin user
20. Spot-check the features that changed

21. Log into the client portal as a test client if portal was affected

22. Update the changelog in CHANGELOG.md with what shipped.

Rollback: Vercel dashboard → Deployments → find the previous deployment → "Promote to Production".

---

SOP-002: Apply Database Migration

Trigger: A new table, column, index, or RLS policy change is needed.

Steps

1. Write the migration SQL in supabase/migrations/ with a timestamp prefix.
2. Filename format: YYYYMMDDHHMMSS_descriptive_name.sql
3. Example: 20260215143000_add_audit_logs_table.sql

4. Include: CREATE TABLE (with IF NOT EXISTS), indexes, RLS enable, RLS policies

5. Test migration on local Supabase: bash supabase db reset # This drops and recreates the local DB, applying all migrations in order Verify the migration applied cleanly. Check for any errors.

6. Test the application locally against the reset DB:

7. Does the application start without TypeScript errors?

8. Do the features that use the new table work correctly?

9. Regenerate TypeScript types: bash supabase gen types typescript --linked > src/types/supabase.ts Commit the updated types along with the migration.

10. Apply to staging (Vercel preview environment / staging Supabase project): bash supabase db push --linked # Ensure `supabase link` points to staging project, not production

11. Verify staging works after migration. Run through the affected user flows manually.

12. Apply to production Supabase:

13. Option A (dashboard): Supabase dashboard → SQL Editor → paste migration SQL → Run
14. Option B (CLI): supabase db push --project-ref <PROD_PROJECT_REF>

15. Prefer Option A for visibility and control

16. Verify production works after migration.

17. Document the migration in the changelog: what table/column was added and why.

RLS Checklist for New Tables

Before applying any migration that creates a new table, confirm:

• [ ] ALTER TABLE new_table ENABLE ROW LEVEL SECURITY; is in the migration
• [ ] SELECT policy: tenant_id check for all roles
• [ ] INSERT policy: tenant_id check; service role only for system-inserted rows
• [ ] UPDATE policy: tenant_id check; role restrictions as appropriate
• [ ] DELETE policy: admin-only or service role only
• [ ] Employee-specific tables: does the employee role get tenant-wide access or own-rows only?
• [ ] Portal tables: is there a client_id filter for portal role?

---

SOP-003: Rotate a Secret

Trigger: A secret is compromised, a team member leaves, or a vendor requires rotation.

Steps

1. Generate the new secret/key in the provider's dashboard.
2. Stripe: dashboard → Developers → API keys → Roll key
3. Resend: dashboard → API Keys → Create API Key
4. Supabase: dashboard → Project Settings → API → Generate new key

5. CRON_SECRET: generate a new random string: openssl rand -hex 32

6. Add the new secret to Vercel environment variables.

7. Vercel dashboard → Project → Settings → Environment Variables
8. Update the variable in all environments that use it (Production, Preview, Development)

9. Do NOT delete the old value yet — keep both until the deployment is verified

10. Trigger a Vercel redeploy to pick up the new secret.

11. Vercel dashboard → Deployments → Redeploy latest production deployment

12. Verify the application works with the new secret.

13. For Stripe: make a test webhook delivery from Stripe dashboard
14. For Resend: send a test email from the admin panel

15. For Supabase keys: verify auth and DB operations work

16. Revoke the old secret in the provider dashboard.

17. Only revoke after confirming the new secret works

18. Update this secrets inventory below with rotation date.

Secrets Inventory

Secret	Provider	Last Rotated	Notes
SUPABASE_SERVICE_ROLE_KEY	Supabase	—	Rotate if team member with access leaves
NEXT_PUBLIC_SUPABASE_ANON_KEY	Supabase	—	Public key — lower risk, rotate if exposed
STRIPE_SECRET_KEY	Stripe	—	Live key — rotate immediately if exposed
STRIPE_WEBHOOK_SECRET	Stripe	—	Tied to webhook endpoint; rotate by creating new endpoint
RESEND_API_KEY	Resend	—	Rotate if email anomalies detected
CRON_SECRET	Self-generated	—	Rotate if cron endpoints are probed
SENTRY_AUTH_TOKEN	Sentry	—	Org-level token; rotate on team changes

---

SOP-004: Cron Job Monitoring

Trigger: Run after each expected cron execution, or when investigating suspected missed runs.

Steps

1. Check Vercel dashboard → Cron tab after each expected run time.
2. The "Last Run" timestamp should be within the expected window

3. Status should be green (success) — red indicates the cron ran but the handler returned a non-2xx status

4. Verify Vercel function logs for the cron route.

5. Vercel dashboard → Project → Functions → filter by the cron route path

6. Look for error logs, timeouts, or unexpected responses

7. If cron didn't run (no recent entry in Vercel Cron tab):

8. Check vercel.json — is the cron defined? Is the schedule correct?
9. Check if free tier cron limit (2 jobs) is exceeded

10. Verify the most recent production deployment included the vercel.json cron definition

11. Manually trigger the cron to verify it works: bash curl -X POST https://your-app.vercel.app/api/admin/appointments/cron \ -H "Authorization: Bearer $CRON_SECRET" \ -v # Expect HTTP 200 with a JSON response

12. Spot-check the database to verify the cron had its expected effect:

```sql -- Check invoices that should have been marked overdue SELECT id, status, due_date FROM invoices WHERE status = 'sent' AND due_date < CURRENT_DATE AND tenant_id = ''; -- Should return 0 rows if cron ran correctly

-- Check recurring invoices created today SELECT id, created_at, client_id, subtotal_cents FROM invoices WHERE created_at::date = CURRENT_DATE AND is_recurring = true; ```

1. If email was expected (recurring invoice notification, overdue reminder):
2. Check Resend dashboard → Logs for today's date
3. If cron ran but emails are missing, see Known Issue HIGH-5

---

SOP-005: Onboard New Client

See Client Onboarding Guide for the full procedure.

Quick checklist:

• [ ] Create client record in admin panel (company name, address, currency)
• [ ] Add primary contact (name, email, phone)
• [ ] Assign client to manager (assigned_manager_id)
• [ ] Create portal user and send invitation
• [ ] Set up any recurring invoices
• [ ] Upload signed contract to client documents
• [ ] Notify assigned manager

---

SOP-006: Offboard Employee

Trigger: Employee leaves the company or is terminated.

Steps

1. Deactivate the user account in the admin panel.
2. Admin → Settings → Team Management → find user → Deactivate

3. Deactivated users cannot log in and are excluded from new task assignments

4. Reassign open tasks and projects.

5. Admin → Tasks → filter by assigned user → reassign to appropriate team member

6. Admin → Projects → check for active projects with this employee assigned → update assigned team

7. Audit time logs for completeness.

8. Admin → Reports → Time Logs → filter by employee and current/recent pay period
9. Check for any open time entries (logged_date with no end) or suspicious gaps

10. Export CSV if needed for payroll

11. Notify team via appropriate channel that the account has been deactivated.

12. Revoke external access for any tools the employee had access to:

13. Supabase dashboard: remove from organization members if they had direct access
14. Vercel: remove from team if applicable
15. GitHub: remove from repository collaborators
16. Resend, Stripe, Sentry: remove from team if applicable

17. Any other shared credentials: rotate per SOP-003

18. Archive employee record (do not delete — time logs and project history reference the user ID).

---

SOP-007: Investigate a Support Request

Trigger: Client or employee reports unexpected behavior or an error.

Steps

1. Reproduce the issue (if possible) using the reporter's account or a test account with the same role.

2. Check Sentry for recent errors from the affected user or route.

3. Sentry → Issues → filter by user email or URL path

4. Check Vercel function logs for the route involved.

5. Look for error messages, unexpected status codes, or slow response times

6. Check Supabase for data issues.

7. Is the record in the expected state?

8. Are there RLS policies that might be blocking access?

9. Check Resend logs if the issue involves email delivery.

10. Check Stripe dashboard if the issue involves payments or subscriptions.

11. Document the root cause in the support ticket or issue tracker.

12. If this is a new bug: open a Known Issue entry, fix, deploy per SOP-001.

13. If this reveals a security issue: open a Production Incident record immediately.

---

AI Prompt Library

Ready-to-use prompts for Claude and other AI assistants to accelerate common CRM development tasks. Each prompt is designed to produce output that matches the SONAN DIGITAL CRM stack: Next.js 15 App Router, edge runtime, TypeScript, Supabase, Vercel.

---

Code Generation Prompts

Generate an API Route for a New CRM Resource

Use this to scaffold a complete CRUD API for any new resource (e.g., tasks, contracts, leads).

Create a Next.js 15 App Router API route for [resource name] in the SONAN DIGITAL CRM.

Stack: Next.js 15 App Router, TypeScript, Supabase, Vercel edge runtime.

Requirements:
- Export `export const runtime = 'edge'` at the top of every route file
- Use `requireAdminWithTenant()` from '@/lib/admin-auth' for authentication — if it returns null, return 401
- Use `createServiceClient()` from '@/lib/supabase/service' for all DB operations
- Filter ALL queries by `caller.tenantId` — never return data without tenant isolation
- Return `NextResponse.json()` with appropriate HTTP status codes
- Use proper TypeScript types, no `any`

Files to generate:
1. `src/app/api/admin/[resource-plural]/route.ts`
   - GET: list all [resources] for tenant (with optional query param filters: [list filters])
   - POST: create a new [resource]

2. `src/app/api/admin/[resource-plural]/[id]/route.ts`
   - GET: get single [resource] by id
   - PUT: update [resource] by id
   - DELETE: delete [resource] by id

Database table: `[table_name]`
Columns: [list columns with types]
Foreign keys: [list FK relationships]

Include proper error handling: validate required fields on POST/PUT, return 404 if resource not found, return 400 for validation errors.

---

Generate a Supabase Migration

Use this to create a complete migration for a new table including RLS.

Write a Supabase PostgreSQL migration file for the SONAN DIGITAL CRM.

New table: `[table_name]`

Standard columns to always include:
- `id UUID PRIMARY KEY DEFAULT gen_random_uuid()`
- `tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE`
- `created_at TIMESTAMPTZ NOT NULL DEFAULT now()`
- `updated_at TIMESTAMPTZ NOT NULL DEFAULT now()`

Additional columns:
[Describe each column: name, type, nullable/not null, default, FK if applicable]

Requirements:
1. CREATE TABLE with all columns
2. CREATE INDEX on tenant_id
3. CREATE INDEX on any FK columns
4. CREATE INDEX on [any columns that will be frequently filtered]
5. Enable RLS: `ALTER TABLE [table_name] ENABLE ROW LEVEL SECURITY;`
6. RLS Policies:
   - SELECT for authenticated: tenant_id = get_active_tenant_id()
   - INSERT for authenticated: tenant_id = get_active_tenant_id()
   - UPDATE for authenticated: tenant_id = get_active_tenant_id()
   - DELETE for admin role only: tenant_id = get_active_tenant_id() AND get_user_role() IN ('admin', 'super_admin')
   [Add employee-specific restrictions if applicable: AND employee_id = auth.uid()]
   [Add portal client restrictions if applicable: AND client_id = get_portal_client_id()]
7. Trigger to auto-update `updated_at` on UPDATE

Output as a single .sql file ready to place in supabase/migrations/.

---

Generate a TypeScript Interface for a Supabase Query

Use this when you have a Supabase select query with FK joins and need correct TypeScript types.

Generate a TypeScript interface for the following Supabase query result in the SONAN DIGITAL CRM.

IMPORTANT: Supabase FK joins ALWAYS return arrays, even for many-to-one relationships.
All nested FK relations must be typed as Array<T>, not T | null.
All property accesses on FK relations must use [0] indexing.

Query:
```typescript
const { data } = await supabase
  .from('[table_name]')
  .select(`
    [paste your select string here]
  `)

Generate: 1. A TypeScript interface [InterfaceName]Row that exactly matches the array/object shape Supabase returns 2. Example usage showing how to safely access nested properties using optional chaining and [0] indexing 3. A typed helper function that maps the raw rows to a cleaner application type

Stack: TypeScript strict mode.

---

### Fix Supabase FK TypeScript Error

Use this when you have an existing type error from a Supabase FK join.

Fix a TypeScript error in the SONAN DIGITAL CRM caused by a Supabase FK join being typed incorrectly.

Context: Supabase always returns FK joins as arrays (Array), even for many-to-one relationships. The current code types the join as a single object, causing runtime errors and TypeScript complaints.

Current interface (wrong):

[paste the current wrong interface]

Current code with the error:

[paste the code that's erroring]

TypeScript error message: [paste the error]

Fix the interface to use array types for all FK joins. Update all property accesses to use [0] indexing with optional chaining. Keep all existing logic and variable names the same — only change the types and property access patterns. Add a comment on each FK field explaining it's an array due to Supabase behavior.

---

### Generate a Server Component Page

Use this to scaffold a new admin panel page.

Create a Next.js 15 App Router server component page for the SONAN DIGITAL CRM admin panel.

Page: src/app/admin/[page-name]/page.tsx

Requirements: - export const runtime = 'edge' at top - Server component (no 'use client' on the page itself) - Auth check using Supabase cookie client — redirect to '/auth/login' if not authenticated - Fetch data server-side using createServiceClient() filtered by tenant_id - Pass fetched data as props to a 'use client' child component for interactivity - Use Supabase's array FK join pattern (access nested with [0])

Page purpose: [describe what this page shows] Data to fetch: [describe the data, including any FK joins] Client component name: [ComponentName]Client — receives typed props from the server component

Include: - TypeScript types for all data - Loading states (use Suspense boundary or skeleton) - Empty state when no data - Error handling if fetch fails

---

### Generate a Notification

Use this to add an in-app notification to an existing API route.

Add an in-app notification to an existing SONAN DIGITAL CRM API route.

Import: import { createNotification } from '@/lib/notifications'

Notification to create: - Type: [one of: new_lead, invoice_paid, contract_signed, contract_declined, proposal_approved, proposal_declined, support_reply] - Title: "[notification title — short, ~5 words]" - Body: [describe what dynamic value should go in the body, e.g., "the client's company name"] - Resource type: [e.g., 'lead', 'invoice', 'contract', 'proposal'] - Resource ID: [the ID variable in the route] - Resource label: [the human-readable name/title of the resource] - Link: [the admin panel URL to navigate to, e.g., /admin/leads/${lead.id}] - User ID: [null for broadcast to all admins, or specific user UUID for targeted notification]

Existing route code:

[paste the existing route handler]

Insert the createNotification call at the correct point (after the record is successfully created/updated in the DB, before the return statement). Use the correct caller.tenantId for tenantId.

---

## Debugging Prompts

### Debug "Unauthorized" on Admin API Route

Debug an "Unauthorized" error on an admin API route in the SONAN DIGITAL CRM.

Stack: Next.js 15 App Router, edge runtime, TypeScript, Supabase. Auth pattern: requireAdminWithTenant() from '@/lib/admin-auth'

Route path: [e.g., /api/admin/clients/[id]] HTTP method: [GET/POST/PUT/DELETE] Error: [paste exact error message or HTTP response]

Route handler code:

[paste the full route handler]

What the user is doing when the error occurs: [describe the action]

User details: - Role in database: [e.g., admin, employee, super_admin] - Logged in: [yes/no] - Tenant: [e.g., has a tenant record in tenant_users]

Check: is requireAdminWithTenant() returning null? Why might it? Consider: session cookie issues, role check, tenant association, edge runtime session parsing. Provide specific diagnosis and the exact fix.

---

### Debug RLS Blocking Query

Debug a Supabase RLS issue in the SONAN DIGITAL CRM where a query returns empty results or "row-level security policy violation".

Table: [table_name] Operation: [SELECT/INSERT/UPDATE/DELETE] User role: [admin/employee/portal/service] Expected result: [what should be returned] Actual result: [empty array / error message]

Query:

[paste the Supabase query]

Current RLS policies on the table (from supabase/migrations/ or Supabase dashboard):

[paste the RLS policies, or write "unknown — need to check"]

Functions used in policies (e.g., get_active_tenant_id, get_user_role): [describe or paste]

Diagnose why RLS is blocking this query. Check: is the user's session available in the RLS context? Is tenant_id being set correctly? Is the role check matching? Provide the exact SQL fix for the RLS policy.

---

### Debug Git FUSE Commit Issue

Debug a git commit issue on a FUSE-mounted repository in the SONAN DIGITAL CRM dev environment.

I'm using the FUSE-safe commit pattern from CLAUDE.md: 1. Write file content to /tmp using Python 2. Hash blob from /tmp with git hash-object --stdin 3. Verify blob line count matches /tmp file line count 4. Build tree with Python ls_tree/mktree helpers 5. Create commit with git commit-tree 6. Update .git/refs/heads/main

Error encountered: [paste the exact error message]

Steps I ran (paste the actual commands and their output): [paste]

File being committed: - Name: [filename] - Size: [approximate lines] - Path in repo: [src/...]

What I expected to happen: [describe]

Diagnose the specific failure point and provide the exact corrective steps to: 1. Fix the immediate error 2. Verify the fix worked (blob line count match, no duplicate tree entries) 3. Create a clean commit from the last successfully pushed parent SHA

---

## SQL Writing Prompts

### Write RLS Policies for a New Table

Write complete PostgreSQL RLS policies for a new table in the SONAN DIGITAL CRM multi-tenant Supabase schema.

Table: [table_name] Columns include: tenant_id UUID, [other columns]

Helper functions available in the schema: - get_active_tenant_id() — returns the tenant UUID from the current session (for admin/employee roles) - get_portal_client_id() — returns the client UUID for portal (client) role sessions - get_user_role() — returns the user's role string ('super_admin', 'admin', 'employee') - auth.uid() — returns the current user's UUID

Roles and their access requirements: - super_admin, admin: full CRUD access, filtered to their tenant_id - employee: [describe what employee can do — e.g., SELECT own rows only, INSERT own rows, no DELETE] - portal (client): [describe — e.g., SELECT rows where client_id = get_portal_client_id()] - Service role: bypasses RLS — no policies needed

Write: 1. ALTER TABLE [table_name] ENABLE ROW LEVEL SECURITY; 2. DROP + CREATE for each policy (idempotent) 3. One policy per role per operation — don't combine roles into one policy 4. Add a comment on each policy explaining what it grants

Format as a migration file ready to apply.

---

### Write a CRM Report Query

Write a PostgreSQL query for the SONAN DIGITAL CRM reports module.

Report name: [e.g., "Revenue by Client This Month"]

Tables involved: - [list tables with relevant columns]

Required filters: - Always: tenant_id = $1 (parameterized) - Date range: [column] BETWEEN $2 AND $3 - [any other filters]

Output columns: [list each output column: name, source table/calculation, display format if relevant]

Aggregations: [e.g., SUM of subtotal_cents grouped by client_id] Ordering: [e.g., ORDER BY total_revenue DESC] Limit: [e.g., top 10, or no limit]

Additional requirements: - Include a totals row using ROLLUP or a UNION ALL - Handle NULLs gracefully (COALESCE where appropriate) - All monetary values are stored as integers (cents) — do not divide by 100 in SQL, return raw cents

The query will be run via Supabase's supabase.rpc() function or .from().select(). Specify which approach fits better given the complexity.

---

## Documentation Prompts

### Write an Architecture Decision Record

Write an Architecture Decision Record for the SONAN DIGITAL CRM.

Use this exact ADR format:

ADR-NNN: [Title]

Status: Accepted / Proposed / Superseded
Date: [YYYY-MM-DD]
Deciders: SONAN DIGITAL Engineering

Context

[Why was this decision needed? What problem were we solving? What constraints existed?]

Decision

[What was decided, stated clearly and specifically.]

Alternatives Considered

Option	Pros	Cons	Reason Rejected
[Option A]
[Option B]

Consequences

Positive: - [benefit 1]

Negative / Trade-offs: - [trade-off 1]

Neutral: - [neutral consequence]

Future Considerations

[What might cause this decision to be revisited?]

---

Decision to document: [describe the decision] Context: [describe why this decision came up, what problem it solved] Alternatives considered: [list what else was evaluated and why it was rejected] Stack context: Next.js 15, edge runtime, TypeScript, Supabase, Vercel — any decision should account for these constraints.

---

### Write a Troubleshooting Entry

Write a troubleshooting entry for the SONAN DIGITAL CRM knowledge base.

Format:

[Problem title — be specific, e.g., "Stripe webhook not received after payment"]

Symptoms [Exact error message or behavior the user sees]

Diagnosis [Numbered steps to identify the root cause]

Fix by Root Cause [Table: Root Cause | Fix]

Prevention [How to avoid this in the future, or what monitoring to set up]

---

Problem to document: [describe the issue] What causes it: [describe the root cause(s)] How to fix it: [describe the fix(es)] How to prevent it: [describe prevention] Stack context: Next.js 15, Supabase, Vercel, Stripe, Resend, edge runtime.

Write in the same clinical, precise tone as the rest of the troubleshooting guide — no filler, no preamble.

---

# Production Incidents

This log captures all P1 and P2 production incidents for the SONAN DIGITAL CRM. Every incident that impacts users or data security must have a record here, even if resolved quickly.

<div class="alert alert-info"><div class="alert-icon">ℹ️</div><div class="alert-body"><div class="alert-title">Severity Definitions</div><ul>
<li><strong>P1</strong> — Critical: data loss, security breach, or complete service unavailability for all users</li>
<li><strong>P2</strong> — High: significant feature unavailable, data integrity risk, or security concern for a subset of users</li>
<li><strong>P3</strong> — Medium: degraded performance or partial feature failure (tracked in Known Issues, not here)</li>
</ul></div></div>
---

## How to Open an Incident

1. Create a new section below using the template
2. Fill in what you know immediately — don't wait to have all answers
3. Update the timeline in real-time as the incident progresses
4. Complete the Root Cause and Fix sections once resolved
5. Assign follow-up items with owners and due dates before closing

---

## Incident Template

Copy this template for each new incident.

---

### INC-YYYY-NNN: Short Description

| Field | Value |
|---|---|
| **Incident ID** | INC-YYYY-NNN |
| **Date** | YYYY-MM-DD |
| **Severity** | P1 / P2 |
| **Duration** | X hours Y minutes |
| **Impact** | Who was affected, what they could not do |
| **Reporter** | Name or "automated monitoring" |
| **Responder(s)** | Names |
| **Status** | Open / Resolved |

#### Timeline

| Time (UTC) | Event |
|---|---|
| HH:MM | Incident detected / reported |
| HH:MM | Engineer assigned |
| HH:MM | Root cause identified |
| HH:MM | Fix deployed or workaround applied |
| HH:MM | Incident resolved and confirmed |

#### Root Cause

_Describe what caused the incident._

#### Fix Applied

_Describe exactly what was changed to resolve the incident._

#### Prevention Actions

_What specific changes will prevent this from happening again?_

#### Follow-up Items

| Item | Owner | Due Date | Status |
|---|---|---|---|
| | | | |

---

## Historical Incidents

---

### INC-2026-001: Cross-Tenant Notification Leak (CRIT-4)

| Field | Value |
|---|---|
| **Incident ID** | INC-2026-001 |
| **Date** | 2026-01-15 (discovered during internal security review) |
| **Severity** | P1 (security) |
| **Duration** | Not user-facing at discovery — no production exposure confirmed |
| **Impact** | Any authenticated user could read all notifications from any tenant via the Supabase API |
| **Reporter** | Engineering (internal security review) |
| **Responder(s)** | Engineering team |
| **Status** | Resolved |

#### Timeline

| Time (UTC) | Event |
|---|---|
| 10:00 | Security review identified notifications table lacks tenant_id RLS policy |
| 10:05 | Severity assessed as P1 — cross-tenant data exposure via missing RLS |
| 10:10 | Confirmed: `SELECT * FROM notifications` with any valid auth token returned rows from all tenants |
| 10:20 | Fix written: RLS policy adding `tenant_id = get_active_tenant_id()` to notifications SELECT |
| 10:35 | Migration applied to production Supabase |
| 10:40 | Verified: authenticated users can now only read their own tenant's notifications |
| 10:45 | Incident resolved |

#### Root Cause

The `notifications` table was created with RLS enabled but no `tenant_id` policy was added. Only a `user_id` policy existed for user-specific notifications. Broadcast notifications (where `user_id IS NULL`) had no policy at all — they were visible to all authenticated users across all tenants.

The gap occurred because adding RLS was treated as a checkbox ("RLS enabled = secure") rather than a review of the full policy set (every role × every operation × tenant isolation).

#### Fix Applied

```sql
-- Added to notifications table RLS policies

-- SELECT: tenant isolation
CREATE POLICY "notifications_select_tenant"
ON notifications
FOR SELECT
TO authenticated
USING (tenant_id = get_active_tenant_id());

-- INSERT: only service role (notifications created via service client)
-- No INSERT policy for authenticated role — service role bypasses RLS

-- UPDATE: tenant isolation (for marking read)
CREATE POLICY "notifications_update_own"
ON notifications
FOR UPDATE
TO authenticated
USING (tenant_id = get_active_tenant_id())
WITH CHECK (tenant_id = get_active_tenant_id());

Prevention Actions

1. Added "RLS review checklist" requirement before any new table ships: SELECT/INSERT/UPDATE/DELETE policies must be explicitly reviewed for each role (authenticated, employee, portal, service)
2. Added notifications RLS gap to Lessons Learned (LL-007)
3. Added cross-tenant read test to security test suite

Follow-up Items

Item	Owner	Due Date	Status
Add automated RLS smoke test for each new table	Engineering	2026-02-01	Open
Review all existing tables for tenant_id RLS completeness	Engineering	2026-02-01	Resolved
Document RLS access matrix in security architecture docs	Engineering	2026-02-15	In Progress

---

INC-2026-002: Client Document Access via Direct Storage URL (CRIT-7)

Field	Value
Incident ID	INC-2026-002
Date	2026-01-15 (discovered during security review, same session as CRIT-4)
Severity	P1 (security)
Duration	Not user-facing at discovery
Impact	Any person with a direct Supabase storage URL could download any client document without authentication
Reporter	Engineering (internal security review)
Responder(s)	Engineering team
Status	Resolved

Timeline

Time (UTC)	Event
10:50	Security review tested storage bucket — confirmed bucket was set to public
10:52	Confirmed: unauthenticated request to direct storage URL returned file content with HTTP 200
10:55	Severity assessed as P1 — all client documents exposed to anyone with a URL
11:00	Plan: make bucket private, implement signed URLs, add client_id validation
11:30	Bucket changed to private in Supabase Storage dashboard
11:35	Confirmed: direct storage URLs now return 403
11:45	API route updated to generate signed URLs with client_id validation
12:00	Storage RLS policies added to storage.objects
12:10	End-to-end test: client can download their own documents via signed URL; cannot access other client documents
12:15	Incident resolved

Root Cause

The Supabase Storage bucket was created via the Supabase Studio dashboard. The default visibility for new buckets in Supabase Studio is public. The "public" setting was not noticed or explicitly set to private during initial setup.

The document API route was written to construct direct storage URLs (not signed URLs), which only work on public buckets. This meant that even when the bucket was made private, the API would break — requiring changes to both the bucket setting and the URL generation logic.

Fix Applied

1. Bucket made private in Supabase Storage dashboard.

2. API route updated to generate signed URLs with client validation:

// /api/portal/documents/[id]/download/route.ts
export const runtime = 'edge'

export async function GET(req: Request, { params }: { params: { id: string } }) {
  const caller = await requirePortalAuth()
  if (!caller) return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })

  const supabase = createServiceClient()

  // Verify this document belongs to the caller's client
  const { data: doc } = await supabase
    .from('client_documents')
    .select('id, storage_path, client_id')
    .eq('id', params.id)
    .eq('client_id', caller.clientId)  // ← client_id check
    .maybeSingle()

  if (!doc) return NextResponse.json({ error: 'Not found' }, { status: 404 })

  // Generate short-lived signed URL
  const { data: signedUrl } = await supabase.storage
    .from('client-documents')
    .createSignedUrl(doc.storage_path, 3600)

  if (!signedUrl) return NextResponse.json({ error: 'Failed to generate URL' }, { status: 500 })

  return NextResponse.redirect(signedUrl.signedUrl)
}

3. Storage RLS policies added to storage.objects:

CREATE POLICY "client_documents_storage_select"
ON storage.objects
FOR SELECT
TO authenticated
USING (
  bucket_id = 'client-documents'
  AND (
    -- Admin access: tenant match
    get_active_tenant_id() IS NOT NULL
    OR
    -- Portal access: client match via metadata
    (storage.foldername(name))[1] = get_portal_client_id()::text
  )
);

Prevention Actions

1. Storage bucket privacy checklist added to new feature DR review — default must be private, explicitly set
2. All future storage access must go through API routes that validate ownership, never direct URLs
3. Added CRIT-7 to Lessons Learned (LL-007)

Follow-up Items

Item	Owner	Due Date	Status
Increase signed URL expiry or switch to on-demand generation	Engineering	v1.1	Open (tracked as HIGH-6)
Add integration test: unauthenticated direct storage URL must return 403	Engineering	2026-02-01	Open

---

INC-2026-003: Employee Cross-Employee Time Log Read (H8)

Field	Value
Incident ID	INC-2026-003
Date	2026-01-20 (discovered during UAT)
Severity	P2
Duration	Resolved in UAT, never reached production
Impact	Employee users could read all other employees' time logs within the same tenant
Reporter	QA (UAT testing)
Responder(s)	Engineering team
Status	Resolved

Timeline

Time (UTC)	Event
14:00	UAT tester reports they can see other employees' time logs in the employee portal
14:05	Confirmed in staging: employee_a can query and view employee_b's time logs
14:10	Root cause identified: time_logs SELECT RLS policy only checks tenant_id, not employee_id
14:20	Fix written: separate RLS policy for employee role restricting to own rows
14:30	Migration applied to staging
14:35	Verified: employee_a can no longer see employee_b's time logs
14:40	Migration applied to production
14:45	Incident closed

Root Cause

The time_logs SELECT RLS policy was written for admin access (tenant-level) and not role-differentiated:

-- Original policy (too permissive for employee role)
CREATE POLICY "time_logs_select"
ON time_logs
FOR SELECT
TO authenticated
USING (tenant_id = get_active_tenant_id());

This allowed any authenticated user in the tenant — including employees — to read all time logs for the tenant.

Fix Applied

Added a role-specific policy for the employee role:

-- Admin role: can see all time logs in tenant (existing policy, no change)
CREATE POLICY "time_logs_select_admin"
ON time_logs
FOR SELECT
TO authenticated
USING (
  tenant_id = get_active_tenant_id()
  AND get_user_role() IN ('admin', 'super_admin')
);

-- Employee role: can only see own time logs
CREATE POLICY "time_logs_select_employee"
ON time_logs
FOR SELECT
TO authenticated
USING (
  tenant_id = get_active_tenant_id()
  AND get_user_role() = 'employee'
  AND employee_id = auth.uid()
);

Prevention Actions

1. RLS review checklist now includes explicit check: "does the employee role get access to only their own rows, or all tenant rows?"
2. Added H8 to Lessons Learned (LL-007)
3. Added employee cross-read test to UAT checklist

Follow-up Items

Item	Owner	Due Date	Status
Audit all employee-accessible tables for similar cross-employee read gaps	Engineering	2026-02-01	Resolved
Add automated test: employee cannot read another employee's time logs	Engineering	2026-02-15	Open