Runbooks
Operational procedures: backfills, migrations, purge helpers, test OTP.
Operational procedures for the deployed backend. Everything here touches real data — read the invariants in Schema first.
Backfill jobs (convex/migrations/backfill.ts)
| Job | What it repairs |
|---|---|
repairActiveVouchersCounter | activeVouchers:<userId> sharded counters from actual voucher rows (probe: voucherCounterProbe) |
backfillDuitRayaCodeStats / clearDuitRayaCodeStats | codeStats aggregate for claim codes |
rebuildUsersByRole, rebuildOrderTotals, rebuildOrdersByChannel, rebuildVoucherCounts | Clear and refill one aggregate. The other aggregates have clear* and backfill* pairs (see Schema) |
*Probe queries (voucherCountsProbe, orderChannelProbe, productRatingsProbe, codeDenomProbe, …) | Compare an aggregate with a bounded scan of its table. Trust a rebuild only once its probe returns ok: true |
A rebuild clears its aggregate before it refills it, so the dashboard numbers it feeds drop to zero and climb back while it runs. On a deployment that already serves those numbers, run one only when its probe reports drift. voucherCounts starts empty on a deployment that gets it for the first time, so run rebuildVoucherCounts and then voucherCountsProbe there after the deploy.
Order stamp backfill (convex/migrations/orderStamps.ts)
Run once per deployment after the stamp-outcome backend ships there:
bunx convex run migrations/orderStamps:fillOrderStampFields '{}'It lowercases profile emails and fills buyerEmail, buyerPhone, shippingPhone and stampOutcome on older orders. It never awards a stamp.
Points and tiers removal
Deploy commit f3c378a1, archive with bash scripts/archive-points-tables.sh <staging|prod>, run migrations/removePoints:removePointsData '{"archived": true}', deploy feca3de3, rebuild the dashboard, then deploy develop HEAD. Loyalty has the full order and the flags.
Full data reset (convex/migrations/resetAnalytics.ts)
This file wipes a deployment's customer data. clearAllAggregates empties all eight aggregates. deleteUserData then deletes every row of orders, stamps, vouchers, reviews, notifications, carts, wishlists, webhook events and the other user-generated tables. deleteAppUsers deletes every apps-role profile, and backfillAggregatesForAdmins refills usersByRole for the staff profiles that remain. Shopify mirror tables, campaign and template configuration, and the Better Auth tables stay. Never run it against production.
Rollback helpers for Shopify writes
removeVoucherCodeFromShopify— removes a single redeem-code from its campaign node.testing/debug:purgeAvailablePoolRows— clears unclaimed Duit Raya pool rows.
Remember: never deleteShopifyDiscount on a campaign-node GID.
Running one-off internal functions
bunx convex run <module>:<function> '<json args>' from packages/backend runs internal queries, mutations and actions directly. It targets dev by default; add --deployment staging or --prod for the others. The CLI skips every auth check in the function, so a self-rescheduling worker started this way keeps running against that deployment's data until it finishes.
Test phone OTP
With ALLOW_TEST_PHONE_OTP=1 on a non-production deployment, four fixed E.164 numbers accept fixed codes (see TEST_PHONE_NUMBERS in convex/auth.ts). The flag fails closed: SITE_URL naming the production host (or unset) refuses it. Never set it on precious-rat-335; env:sync-staging hard-refuses production for exactly this reason.
Environment sync
bun run env:sync-staging copies env vars dev → staging (skips JWKS, refuses production). For any other direction, do it manually and deliberately.