Incident Response

This page documents the incident classification system, the end-to-end response process, escalation paths, the post-mortem template, and communication templates for client-facing outages.

---

1. Incident Classification

Classify every incident as early as possible. Classification determines the response speed and communication requirements.

Priority	Name	Definition	Response Window	Examples
P1	Critical / Full Outage	The application is completely unavailable, or a core workflow is broken for all users	15 minutes	Homepage returns 500, login broken for all users, payments failing for all clients, database unreachable
P2	Major Feature Degraded	A key feature is broken for a significant portion of users, but the application is otherwise available	1 hour	Invoices cannot be created, email delivery down, file uploads failing, cron jobs not running
P3	Partial / Minor Feature	A feature is degraded for some users, or an edge case is broken	4 hours	Search returns incorrect results, sorting on a table broken, notifications delayed, minor display issues affecting workflow
P4	Cosmetic / Low Impact	Visual issues, minor UX problems, or non-urgent bugs with no operational impact	Next business day	Button color wrong, typo in UI text, report CSV has an extra column

---

2. Incident Response Steps

a. Detect

Incidents are detected through one of three channels:

• Automated alert — UptimeRobot downtime notification or Sentry error spike alert fires to the #crm-alerts Slack channel and primary engineer's email.
• User or client report — A client or staff member reports an issue via support ticket, Slack, or direct message.
• Engineering discovery — An engineer notices an anomaly during routine work or monitoring review.

When detected: Immediately check the Sentry dashboard and Vercel logs to assess scope and severity before classifying.

---

b. Classify

Apply the classification table above. Key questions:

• Is the application completely unreachable? → P1
• Can users log in? → If no, P1
• Is payment processing broken? → P1 or P2
• Is it affecting all users or a subset? → All = higher priority
• How many clients are actively impacted right now? → More = higher priority

---

c. Acknowledge

The first engineer to see the alert claims ownership of the incident within the response window:

• Post in #crm-alerts: "@channel I'm picking up the [P1/P2] incident — [brief description]. Investigating now."
• Do not have multiple engineers independently investigating without coordination — assign one lead; others assist on request.

---

d. Communicate

Internal communication: - Post status updates to #crm-alerts every 15 minutes for P1, every 30 minutes for P2. - Tag the Tech Lead and Operations Manager for P1.

External communication (clients): - For P1 incidents lasting more than 15 minutes, the Operations Manager sends a client communication (see templates below). - For P2 incidents lasting more than 1 hour, consider proactive communication if the affected feature is client-visible (invoice access, portal login, etc.). - For P3/P4, no proactive client communication unless a client has already reported the issue to you.

Update the status page (UptimeRobot public page) immediately for P1: - Set the affected monitors to "Under Maintenance" to suppress repeated alerts while the incident is active.

---

e. Investigate

Systematic investigation steps for production incidents:

1. Check recent deployments — Vercel Dashboard → Deployments. Did an incident start within 30 minutes of a deployment? That deployment is the prime suspect.

2. Check Sentry — Look for new issues or spikes that started around the incident time. Check the stack trace for the root cause.

3. Check Vercel edge logs — Filter by the affected route. Look for 500 errors, timeouts, or unexpected response patterns.

4. Check Supabase — Supabase Dashboard → Database → Logs. Look for query errors, connection pool exhaustion, or RLS policy failures.

5. Check external providers:

6. Stripe status: https://status.stripe.com
7. Supabase status: https://status.supabase.com
8. Vercel status: https://www.vercel-status.com
9. Resend status: https://status.resend.com

If a provider has an active incident, that is likely the root cause — monitor their status page and wait for resolution.

1. Check environment variables — If a recent deployment changed env vars, verify the values are correct in Vercel Dashboard → Settings → Environment Variables.

---

f. Mitigate

Mitigate before fully resolving when possible — stop the bleeding, then fix the cause.

Scenario	Mitigation
Bad deployment	Immediately rollback in Vercel Dashboard → Deployments → last known-good deployment → "Promote to Production"
Database query causing load	Identify the query in Supabase logs; if it's from a new feature, disable that feature's route temporarily
Stripe webhook loop	Disable the webhook endpoint in Stripe Dashboard temporarily; re-enable after fixing the handler
Leaked secret	Immediately revoke and rotate (see Secrets Management emergency procedure)
Email delivery down	Resend failures affect notifications only — the app remains functional; inform users of the delay

---

g. Resolve

Resolution is complete when:

• The root cause is fixed (not just mitigated).
• The fix is deployed to production.
• Affected functionality is verified to be working in production.
• Any data inconsistencies caused by the incident are corrected.
• External provider incidents are resolved and services confirmed normal.

Declare resolution:

Post in #crm-alerts: "[P1/P2] incident RESOLVED. [Brief summary of root cause and fix]. All services normal. Post-mortem to follow within 48h."

Update the UptimeRobot status page to reflect the incident is resolved.

---

h. Document — Post-Mortem

For P1 and P2 incidents, a post-mortem must be completed within 48 hours of resolution. Use the template in Section 4.

---

3. Escalation Path

Monitoring Alert or User Report
         │
         ▼
  Primary Engineer
  (claim ownership within response window)
         │
         ├── P3/P4: Engineer resolves without escalation
         │
         ├── P2 not resolved in 1 hour ──► Tech Lead
         │
         └── P1 immediately ──────────────► Tech Lead + Operations Manager
                                                     │
                                                     └── Not resolved in 2 hours ──► CTO
                                                                                           │
                                                                                           └── Vendor support (Vercel / Supabase / Stripe) as needed

Role	Responsibility
Primary Engineer	Technical investigation and fix
Tech Lead	Decision-making on mitigations (rollback, disable feature, etc.); coordinates engineering resources
Operations Manager	Client communication, status page updates, internal stakeholder updates
CTO	Final authority on major decisions (extended rollback, data loss disclosure, vendor engagement)

---

4. Post-Mortem Template

Copy and fill in within 48 hours of P1/P2 resolution.

## Incident Post-Mortem

**Date:** YYYY-MM-DD
**Severity:** P1 / P2
**Duration:** HH:MM (from first detection to full resolution)
**Incident Lead:** [Name]
**Participants:** [Names]

---

### Impact

- Number of users affected:
- Features affected:
- Client-facing impact:
- Data loss or corruption: Yes / No (describe if yes)

---

### Timeline

| Time (UTC) | Event |
|---|---|
| HH:MM | Incident detected via [alert / user report / engineering] |
| HH:MM | Engineer claimed ownership |
| HH:MM | Root cause identified |
| HH:MM | Mitigation applied |
| HH:MM | Fix deployed |
| HH:MM | Resolution confirmed |

---

### Root Cause

[Clear, specific description of what failed and why. Avoid blame — focus on the system or process failure.]

---

### Fix Applied

[What was changed to resolve the incident.]

---

### Prevention Actions

| Action | Owner | Due Date |
|---|---|---|
| [e.g., Add input validation to the invoice creation route] | [Name] | YYYY-MM-DD |
| [e.g., Add unit test for the failing edge case] | [Name] | YYYY-MM-DD |

---

### Follow-Up Items

- [ ] Update runbook if this incident revealed a gap
- [ ] Add monitoring/alert for this failure mode
- [ ] Review if similar issues exist in related code

---

5. Client Communication Templates

Use these templates for P1 incidents that are client-visible. Send from the Operations Manager's email.

Initial Notification (send within 15 minutes of P1 declaration)

Subject: SONAN DIGITAL Platform — Service Disruption Notice

Dear [Client Name / "Valued Client"],

We are writing to inform you that we are currently experiencing a service disruption affecting [describe: the client portal / invoice access / [specific feature]].

Our engineering team is actively investigating and working to restore full service as quickly as possible.

We will provide an update within [30 minutes / 1 hour].

We sincerely apologize for the inconvenience. If you have an urgent need during this period, please contact us directly at [support email / phone].

— The SONAN DIGITAL Team

---

Update During Extended Incident (send every 30–60 minutes for ongoing P1)

Subject: SONAN DIGITAL Platform — Service Disruption Update

Dear [Client Name / "Valued Client"],

This is an update on the ongoing service disruption we notified you about at [time].

Current status: [The issue has been identified and a fix is being deployed. / We have identified the root cause and are working with [provider name] to resolve it. / We are continuing to investigate.]

Expected resolution: [Estimated time, or "We will provide another update by [time]."]

We appreciate your patience and apologize for the disruption.

— The SONAN DIGITAL Team

---

Resolution Notice

Subject: SONAN DIGITAL Platform — Service Restored

Dear [Client Name / "Valued Client"],

We are pleased to inform you that the service disruption that began at [time] has been fully resolved as of [time].

All platform features are now operating normally. If you experience any further issues, please do not hesitate to contact us at [support email].

We apologize for the inconvenience this caused and are committed to preventing similar incidents. A full review of the root cause has been completed.

Thank you for your patience and understanding.

— The SONAN DIGITAL Team

---

Client Onboarding

This guide is for admin staff performing the onboarding of a new agency client into the SONAN DIGITAL CRM. Follow each step in order. The target is to have a client fully onboarded — with portal access and a scheduled kickoff — within 24 hours of contract signing.

---

Overview

Step	Action	Done by	Expected time
1	Create client record	Admin	5 min
2	Add primary contact	Admin	5 min
3	Assign account manager	Admin	2 min
4	Create initial proposal (if applicable)	Admin / Account Manager	15–30 min
5	Set up portal access — send invitation	Admin	5 min
6	Verify invitation received and password created	Admin (confirm with client)	Up to 24 hours
7	Schedule kickoff meeting	Admin / Account Manager	5 min
8	Brief client on portal and send documentation	Admin / Account Manager	10 min

---

Step 1: Create Client Record

1. Log in to the CRM admin panel.
2. Navigate to Clients → New Client (or click the + New Client button in the Clients list).
3. Fill in all required fields:

Field	Notes
Company Name	Official company name as it will appear on invoices
Email	Company/billing email address
Phone	Primary company phone number
Address	Full billing address — required for invoice generation
Industry	Optional but useful for reporting
Status	Set to Active

1. Click Save.
2. Note the new client's ID from the URL — you will need it to associate contacts, proposals, and projects.

---

Step 2: Add Primary Contact

Every client must have at least one contact record, and at least one contact should be marked as primary.

1. Open the newly created client record.
2. Click Contacts → Add Contact.
3. Fill in:

Field	Notes
Name	Full name of the primary contact person
Email	Contact's personal work email — this is the email that will receive the portal invitation
Phone	Direct phone number
Title	Job title (e.g., CEO, Marketing Manager)
Is Primary	Check this box — designates this contact as the main point of contact

1. Click Save.

---

Step 3: Assign Account Manager

1. Open the client record.
2. Find the Account Manager field (or Assigned Manager — this maps to assigned_manager_id in the database).
3. Select the relevant team member from the dropdown.
4. Click Save.

The assigned account manager will receive notifications related to this client (new messages, proposal responses, invoice payments, etc.).

---

Step 4: Create Initial Proposal (If Applicable)

If a proposal has not yet been signed, or if a new scope of work needs to be created:

1. Navigate to Proposals → New Proposal.
2. Select the Client created in Step 1.
3. Fill in the proposal title, description, valid until date, and line items.
4. Set status to Draft.
5. Review and set to Sent when ready to share with the client.

The client will be able to view and approve the proposal from their portal once they have portal access (Step 5).

If the client already has a signed contract and this onboarding is for a new project cycle, skip this step.

---

Step 5: Set Up Portal Access — Send Invitation

The client portal invitation sends an email to the primary contact with a link to create their password and access the portal.

1. Open the client record.
2. Click Portal Access or Send Invitation (location may vary — look in the client record action menu or the Contacts section).
3. Confirm the invitation will be sent to the primary contact's email address.
4. Click Send Invitation.

The system will: - Create a Supabase Auth user for the contact's email with role client. - Send an invitation email via Resend with a password setup link (valid for 24 hours by default).

---

Step 6: Verify Invitation Received and Account Created

This is a critical verification step — do not mark onboarding complete until confirmed.

1. Within 4 hours of sending the invitation, follow up with the client to confirm the invitation email was received.
2. Check their spam/junk folder if not received.

3. If not received after 15 minutes, see Common Issues below.

4. Confirm the client has successfully created their password and can log in:

5. Ask the client to confirm they can access https://yourdomain.com and see their dashboard.

6. Alternatively, check the CRM for a last_sign_in timestamp on the client's user record.

7. Verify the client can see relevant data — confirm they can view their proposals, project status, and any shared files from their portal.

---

Step 7: Schedule Kickoff Meeting

1. Coordinate with the client and the assigned account manager to find a meeting time.
2. Schedule the kickoff in the CRM Appointments module or the team calendar tool.
3. Send a calendar invite to:
4. The client's primary contact
5. The assigned account manager
6. Any other relevant team members

The kickoff meeting agenda should cover: - Introduction to the account team - Review of scope and deliverables - Communication and feedback expectations - Walk-through of the client portal - Q&A

---

Step 8: Brief Client on Portal and Share Documentation

After the client has portal access:

1. Send the client the Client Portal Guide link:
2. Share the documentation portal URL and direct them to the Client Guide section.
3. Briefly explain the key portal features:
4. Viewing and approving proposals
5. Signing contracts electronically
6. Making invoice payments
7. Tracking project progress
8. Submitting support tickets
9. Provide the support contact email for any portal access issues.

---

Onboarding Checklist (Quick Reference)

Print or copy this checklist for use during actual onboarding:

CLIENT ONBOARDING CHECKLIST
Client: ________________________  Date: ______________
Onboarded by: __________________

[ ] 1. Client record created (company name, email, phone, address, status = Active)
[ ] 2. Primary contact added (name, email, phone, title, is_primary = true)
[ ] 3. Account manager assigned
[ ] 4. Initial proposal created (or N/A — contract already signed)
[ ] 5. Portal invitation sent to: ________________________ (email)
[ ] 6. Client confirmed invitation received
[ ] 7. Client confirmed successful login to portal
[ ] 8. Client can view their proposals / projects in portal
[ ] 9. Kickoff meeting scheduled: ____________ (date/time)
[10] Client briefed on portal features and documentation link shared

---

Timeline Target

Milestone	Target
Client record created	Same day as contract signing
Portal invitation sent	Within 4 hours of contract signing
Client confirms portal access	Within 24 hours of invitation
Kickoff meeting scheduled	Within 48 hours of contract signing
Kickoff meeting held	Within 5 business days of contract signing

---

Common Issues

Invitation email not received

1. Ask the client to check their spam / junk folder first.
2. Verify in Resend Dashboard that the email was delivered successfully (look for the contact's email in sent emails).
3. If Resend shows delivery but the client still cannot find it, the email may have been filtered by the client's corporate mail gateway. Ask their IT team to whitelist @resend.dev (or your sending domain).
4. If the email was not sent (Resend shows no record), check that the contact's email was correctly entered in Step 2 and resend the invitation.

Client is confused about MFA / authentication

The client portal does not require TOTP MFA by default (only admin/staff accounts require MFA). The client simply sets a password via the invitation link. If they are seeing a two-factor prompt, verify their account was created with the correct role (client, not admin).

Invitation link has expired

1. Go to the client's contact record.
2. Click Resend Invitation.
3. Notify the client that a new link has been sent and that the previous link is no longer valid.
4. The new link is also valid for 24 hours.

Client cannot see their data after logging in

1. Verify the client's user account is linked to the correct client record (the client_id on their Supabase Auth user profile must match).
2. Verify RLS policies are allowing the client to read their own data.
3. Check Sentry for any errors occurring when the client accesses their dashboard.

---

Employee Onboarding

This guide is for admin staff responsible for adding new team members to the SONAN DIGITAL CRM. It covers the invitation process, role assignment, MFA setup, project assignment, and the corresponding offboarding procedure.

---

Overview

Step	Action	Done by	Expected time
1	Create employee invite	Admin	5 min
2	Set role	Admin	2 min
3	Verify invite received	Admin (confirm with employee)	Up to 1 hour
4	Employee sets password and logs in	Employee	10 min
5	Guide employee through MFA setup	Admin / Employee	10–15 min
6	Assign to relevant projects	Admin / Team Lead	10 min
7	Brief on time logging expectations	Admin / Team Lead	10 min
8	Provide documentation portal access	Admin	5 min
9	Add to team communication channels	Admin	5 min

---

Step 1: Create Employee Invite

1. Log in to the CRM admin panel.
2. Navigate to Settings → Team Management.
3. Click Invite Team Member (or + Invite Employee).
4. Enter the new employee's work email address.
5. Proceed to Step 2 before sending — the role must be set correctly.

---

Step 2: Set Role

Select the appropriate role for the new team member:

Role	Access Level	Use for
employee	Can log time, view assigned projects and tasks, access employee-facing modules	Most new hires — developers, designers, project coordinators
admin	Full access to all CRM modules, including client data, invoices, proposals, settings	Senior staff, account managers, team leads, operations staff

Important: The admin role grants access to all client financial data, contracts, and system settings. Only assign admin access where it is genuinely required.

Once the role is confirmed, click Send Invitation.

The system will: - Create a Supabase Auth user with the selected role. - Send an invitation email via Resend to the new employee's email address.

---

Step 3: Verify Invite Received

1. Follow up with the new employee to confirm the invitation email arrived.
2. Ask them to check their spam / junk folder if not received within 15 minutes.
3. If the email was not delivered, check the Resend Dashboard for the send record and delivery status.
4. If the email address was entered incorrectly, deactivate the incorrectly-created account (Settings → Team → deactivate) and send a new invitation with the correct email.

The invitation link is valid for 24 hours.

---

Step 4: Employee Sets Password and Logs In

The employee:

1. Clicks the invitation link in the email.
2. Sets a strong password (minimum 12 characters, mix of uppercase, lowercase, numbers, symbols recommended).
3. Is redirected to the CRM admin panel on first successful login.

Confirm with the employee that they can access the dashboard and see the navigation appropriate to their role.

---

Step 5: Guide Employee Through MFA Setup

Multi-factor authentication (TOTP) is required for all staff accounts.

Have the employee complete MFA setup on their first login session:

1. The employee navigates to Profile / Account Settings → Security → Enable Two-Factor Authentication.
2. They will see a QR code.
3. They open an authenticator app (Google Authenticator, Authy, or 1Password) and scan the QR code.
4. They enter the 6-digit TOTP code to confirm setup.
5. They are shown backup codes — instruct them to save these codes securely (in a password manager, not in plain text or email). Backup codes are single-use recovery codes if they lose their authenticator device.

If an employee loses access to their TOTP device: An admin can reset MFA for the user via Settings → Team Management → [User] → Reset MFA. The employee will then complete MFA setup again on next login.

---

Step 6: Assign to Relevant Projects

1. Navigate to Projects in the CRM admin panel.
2. Open each project the new employee will be working on.
3. Go to the project's Team or Members section.
4. Add the employee as a team member.
5. Confirm their role on the project (e.g., Developer, Designer, Project Manager).

Repeat for each active project the employee is joining.

---

Step 7: Brief on Time Logging Expectations

All employees are expected to log time accurately for client billing and project reporting. Cover the following during the briefing:

• How to log time: Projects → select project → Tasks → select task → Log Time (or Time Logs section).
• Required fields: hours, date, optional note.
• Logging frequency: Time should be logged daily or at least weekly — not retrospectively at month end.
• Rounding policy: Agree on the team's standard (e.g., nearest 15 minutes).
• What counts as billable vs. non-billable time (if applicable).
• How to correct a time log: Logged time can be edited by the employee or an admin.

---

Step 8: Provide Documentation Portal Access

Share the documentation portal URL with the new employee:

1. Send them the link to this documentation portal.
2. Point them specifically to the Employee Guide section (if available), which covers:
3. How to use the time logging module
4. How to view and update assigned tasks
5. How to submit project updates
6. How to use the support / communication modules

If there is a team internal wiki or onboarding doc, share that link as well.

---

Step 9: Add to Team Communication Channels

Ensure the new employee is added to:

• [ ] Slack (or team's primary messaging tool) — invite via the workspace admin panel.
• [ ] Relevant project channels in Slack.
• [ ] Google Workspace / Microsoft 365 group or team (if applicable).
• [ ] Team calendar (share relevant calendars, add to recurring stand-ups or team meetings).
• [ ] Any project management tools used alongside the CRM (Notion, Linear, Jira, etc.).
• [ ] GitHub organization / team (if applicable, for engineering staff).

---

Onboarding Checklist (Quick Reference)

EMPLOYEE ONBOARDING CHECKLIST
Employee Name: _________________  Start Date: __________
Role: __________________________  Onboarded by: ________

[ ] 1. Invite sent to: ________________________ (email)
[ ] 2. Role confirmed: employee / admin (circle one)
[ ] 3. Employee confirmed invite received
[ ] 4. Employee set password and successfully logged in
[ ] 5. MFA (TOTP) set up and verified
[ ] 6. Backup codes saved by employee (confirmed)
[ ] 7. Assigned to projects: _______________________________
[ ] 8. Time logging expectations discussed
[ ] 9. Documentation portal link shared
[10] Added to Slack and relevant channels
[11] Added to calendar / team meetings

---

Offboarding Checklist

When a team member leaves the organization, follow this checklist on their last day or immediately upon notice of departure:

Immediate actions (same day)

• [ ] Deactivate the CRM account — Settings → Team Management → select the employee → Deactivate Account. This immediately revokes their access to the CRM and all client data.
• [ ] Revoke Vercel access (if applicable) — Vercel Dashboard → Team Settings → Members → Remove.
• [ ] Revoke Supabase access (if applicable) — Supabase Dashboard → Project Settings → Team → Remove.
• [ ] Revoke GitHub access — GitHub Organization → People → remove or change role.
• [ ] Revoke Sentry access — Sentry → Organization Settings → Members → Remove.
• [ ] Remove from Slack — Slack workspace admin → Deactivate account.
• [ ] Remove from Google Workspace / M365 (if applicable).
• [ ] Change any shared passwords the employee had access to (admin panel credentials, shared tools).

Before deactivation (if time allows)

• [ ] Reassign open tasks — navigate to the employee's task list and reassign any open/in-progress tasks to another team member.
• [ ] Reassign as account manager — search for clients where assigned_manager_id is the departing employee and reassign to a new manager.
• [ ] Audit time logs — confirm all time logs up to the last working day are submitted and accurate. Time logs cannot easily be added by the employee after account deactivation.
• [ ] Transfer project ownership — if the employee was a project lead, assign a new lead before deactivating.

Within one week

• [ ] Rotate any secrets the employee had access to — if they had access to Vercel environment variables or were listed as a Stripe/Supabase admin, rotate the relevant secrets per the Secrets Management rotation procedure.
• [ ] Document the offboarding in the team log with the date, employee name, and confirmation that access was revoked.