[WIP] Downgrade SQLite engine to 2026-05-12 build (012a04ed) to isolate OOM regression

[chiragsalian]

Explanation of Change

• Reverts the SQLite amalgamation (libstuff/sqlite3.c, libstuff/sqlite3.h, libstuff/sqlite3ext.h) from the current build 45f6fccc (2026-07-09) back to 012a04ed (2026-05-12), taken from commit c6d195fd ("new sqlite version from dan"). Code taken from the before this June 5th PR.
• SQLITE_VERSION is unchanged (3.54.0) — only the hctree/engine build (SOURCE_ID) changes. No other files touched.

Why

• The monthly "Account Manager Retention Report" — a large read-only SELECT run via readdb/CustomQuery, doing ~12,763 full-table scans under BEGIN CONCURRENT — has run for ~3 years and now aborts with out-of-memory: {SQLITE} Code: 7, Message: statement aborts at ...: out of memory → 402 Bad query.
• Fleet-wide CustomQuery out-of-memory events were 0 through June, onset 2026-07-06. App-side Bedrock changes in the window were cleared (logging / rate-limiting / write-path locking). The only memory-relevant change underneath this path is the SQLite/hctree engine, which was bumped 3× between the report's last success (012a04ed, May 12) and failure (69941fdc, Jun 17).
• 012a04ed is the last engine build the report ran on cleanly, so this reverts to it.

Tests

• Deploy this branch to a single non-critical follower (not leader / secondary-leader).
• Re-run the report query on that node.
• OOM disappears → confirms the engine bump is the cause. OOM persists → engine is exonerated; look elsewhere.

Caveat

• The amalgamation is reverted ~2 months while keeping current Bedrock C++. Watch CI for any compile incompatibility.

[chiragsalian]

oh i should have posted. Once i got back from parental leave i tested the same query to determine if the issue persists but it did not.
Previously this query,

sudo readdb.sh -csv "WITH lastPurchases AS ( SELECT p.accountID, a.email, SUBSTR(SUBSTR(a.email, INSTR(a.email, '@') + 1), INSTR(SUBSTR(a.email, INSTR(a.email, '@') + 1), '@') + 1) as billingOwnerDomain, p.message, strftime('%m', p.created) as purchaseMonth, COALESCE(JSON_EXTRACT(p.message, '$.subscription.userCount'), 0) AS subscriptionSize, COALESCE(JSON_EXTRACT(p.message, '$.totalActorCount'), 0) AS actorsCount, COALESCE(JSON_EXTRACT(p.message, '$.isApprovedAccountant'), 0) as isApprovedAccountant, COALESCE(JSON_EXTRACT(p.message, '$.isApprovedAccountantClient'), 0) as isApprovedAccountantClient, COALESCE(JSON_EXTRACT(p.message, '$.subscription.endDate'), '') AS subscriptionEndDate, (COALESCE(JSON_EXTRACT(p.message, '$.subscription.autoRenew'), true) = false) AS hasAutoRenewDisabled, COALESCE(JSON_EXTRACT(p.message, '$.subscription.type') = 'invoicing2018', 0) as isInvoicedCustomer, COALESCE(JSON_EXTRACT(p.message, '$.approvedAccountantAccountIDs[0]'), 0) as accountantID FROM purchases p INNER JOIN accounts a ON (p.accountID = a.accountID) WHERE JSON_TYPE(p.message, '$.totalActorCount') IS NOT NULL AND p.created >= DATE('now','start of month', '-3 month') AND CAST(strftime('%d', p.created) as INT) < 5 AND JSON_EXTRACT(p.message, '$.billingType') != 'free2021' ), domains AS ( SELECT p.billingOwnerDomain as domain, d.accountID as domainID, am.email as accountManagerEmail, am.accountID as accountManagerID, COALESCE((e.accountID IS NOT NULL), 0) as isProvisioned FROM lastPurchases p INNER JOIN accounts d ON (d.email = '+@' || p.billingOwnerDomain) LEFT JOIN nameValuePairs n ON (n.accountID = domainID AND n.name = 'domain_member' AND JSON_VALID(n.value)) LEFT JOIN accounts am ON (am.accountID = JSON_EXTRACT(n.value, '$.settings.accountManager')) LEFT JOIN nameValuePairs e ON (e.accountID = domainID AND e.name = 'private_expensifyCardSettings' AND JSON_VALID(e.value)) GROUP BY domain ), domainAdmins AS ( SELECT d.domain, da.accountID, da.email FROM domains d INNER JOIN sharedNameValuePairs sa ON (sa.ownerAccountID = d.domainID AND sa.name GLOB 'expensify_adminPermissions_*') INNER JOIN accounts da ON (sa.accountID = da.accountID) ), domainUsers AS ( SELECT *, MAX(weight) FROM ( SELECT d.domain, a.accountID, 1 as weight FROM domains d INNER JOIN accounts a ON (d.domain = SUBSTR(SUBSTR(a.email, INSTR(a.email, '@') + 1), INSTR(SUBSTR(a.email, INSTR(a.email, '@') + 1), '@') + 1)) UNION ALL SELECT da.domain, da.accountID, 0 as weight FROM domainAdmins da ) GROUP BY accountID ), lastInteraction AS ( SELECT du.domain, MAX(ra.created) as lastInteractionDate FROM domainUsers du INNER JOIN reportActions ra ON (du.accountID = ra.accountID AND ra.created > DATE('now', '-6 months') AND ra.action = 'ADDCOMMENT') INNER JOIN reportNameValuePairs t INDEXED BY reportNameValuePairsType ON (ra.reportID = t.reportID AND t.name = 'type' AND t.value = 'chat') WHERE EXISTS (SELECT 1 FROM reportActions ra2 INNER JOIN domains d ON (ra2.accountID = d.accountManagerID) WHERE ra2.reportID = ra.reportID AND d.domain = du.domain) AND JSON_VALID(ra.message) AND JSON_TYPE(ra.message, '$.html') IS NOT NULL AND JSON_TYPE(ra.message, '$.whisperedTo') IS NULL GROUP BY domain ), purchasesInfoPerDomain AS ( SELECT p.accountID as mainBillingOwnerID, p.billingOwnerDomain, SUM(IIF(p.purchaseMonth = strftime('%m', date('now', '-3 month')), p.actorsCount, 0)) as actorsCount4MonthAgo, SUM(IIF(p.purchaseMonth = strftime('%m', date('now', '-2 month')), p.actorsCount, 0)) as actorsCount3MonthAgo, SUM(IIF(p.purchaseMonth = strftime('%m', date('now', '-1 month')), p.actorsCount, 0)) as actorsCount2MonthAgo, SUM(IIF(p.purchaseMonth = strftime('%m'), p.actorsCount, 0)) as actorsCountLastMonth, MAX(IIF(p.purchaseMonth = strftime('%m'), p.isApprovedAccountant, 0)) as isApprovedAccountant, MAX(IIF(p.purchaseMonth = strftime('%m'), p.isApprovedAccountantClient, 0)) as isApprovedAccountantClient, MAX(IIF(p.purchaseMonth = strftime('%m'), p.accountantID, 0)) as accountantID, SUM(IIF(p.purchaseMonth = strftime('%m'), p.subscriptionSize, 0)) as subscriptionSize, SUM(IIF(p.purchaseMonth = strftime('%m') AND p.subscriptionSize > 0 AND p.subscriptionSize < p.actorsCount, (p.actorsCount - p.subscriptionSize), 0)) as overage, SUM(IIF(p.purchaseMonth != strftime('%m') OR p.subscriptionSize > 0, 0, p.actorsCount)) as ppu, MAX(IIF(p.purchaseMonth = strftime('%m'), p.subscriptionEndDate, '')) as subscriptionEndDate, MAX(IIF(p.purchaseMonth = strftime('%m'), p.hasAutoRenewDisabled, 0)) as hasAutoRenewDisabled, MAX(IIF(p.purchaseMonth = strftime('%m'), p.isInvoicedCustomer, 0)) as isInvoicedCustomer FROM lastPurchases p GROUP BY p.billingOwnerDomain HAVING MAX(p.actorsCount) OR MAX(p.subscriptionSize) > 0 ) SELECT pd.billingOwnerDomain, ( SELECT GROUP_CONCAT(DISTINCT(p.email)) FROM lastPurchases p WHERE p.billingOwnerDomain = pd.billingOwnerDomain AND p.purchaseMonth = strftime('%m') ) as billingOwners, IIF( pd.billingOwnerDomain IN ('afis.ch','aol.com','att.net','bellsouth.net','bills.expensify.com','btinternet.com','cheerful.com','chromeexpensify.com','comcast.net','cox.net','evernote.user','expensify.cash','expensify.sms','gmail.com','gmail.con','googlemail.com','hey.com','hotmail.co.uk','hotmail.com','hotmail.fr','hotmail.it','icloud.com','live.com','mac.com','mail.ru','me.com','msn.com','outlook.com','privaterelay.appleid.com','qq.com','sasktel.net','sbcglobal.net','verizon.net','yahoo.ca','yahoo.co.in','yahoo.co.uk','yahoo.com.br','yahoo.com','ymail.com','pm.me','protonmail.ch','protonmail.com','proton.me','rigl.ch','spacehotline.com','tafmail.com','jeeviess.com','wolfandcranebar.tech','cryptohistoryprice.com','vomoto.com'), 1, 0 ) isPublic, pd.isApprovedAccountant, pd.isApprovedAccountantClient, IIF( pd.accountantID > 0, (SELECT SUBSTR(SUBSTR(a.email, INSTR(a.email, '@') + 1), INSTR(SUBSTR(a.email, INSTR(a.email, '@') + 1), '@') + 1) FROM accounts a WHERE a.accountID = pd.accountantID), '' ) as accountantDomain, (SELECT JSON_EXTRACT(nt.value, '$.selected') FROM nameValuePairs nt WHERE nt.accountID = pd.mainBillingOwnerID AND nt.name = 'timeZone') as timeZone, strftime('%m') as currentMonth, pd.actorsCount4MonthAgo, pd.actorsCount3MonthAgo, pd.actorsCount2MonthAgo, pd.actorsCountLastMonth, pd.subscriptionSize, pd.overage, pd.ppu, pd.isInvoicedCustomer, pd.subscriptionEndDate, ( SELECT COUNT(1) FROM domainUsers a INNER JOIN nameValuePairs n ON (a.accountID = n.accountID AND n.name GLOB 'expensify_policy*') WHERE a.domain = pd.billingOwnerDomain AND JSON_EXTRACT(n.value, '$.type') NOT IN ('personal', 'free') ) as policyCount, pd.hasAutoRenewDisabled, d.isProvisioned, COALESCE(( SELECT 1 FROM accounts du INNER JOIN cards c ON (du.accountID = c.accountID) WHERE SUBSTR(SUBSTR(du.email, INSTR(du.email, '@') + 1), INSTR(SUBSTR(du.email, INSTR(du.email, '@') + 1), '@') + 1) = pd.billingOwnerDomain AND bank = 'Expensify Card' AND JSON_VALID(c.message) AND JSON_EXTRACT(c.message, '$.unapprovedExpenseLimit') > 0 AND state = 3 LIMIT 1 ), 0) as useExpensifyCard, ( SELECT COUNT(*) calls FROM nameValuePairs nvp, JSON_EACH(nvp.value) WHERE nvp.accountID = d.accountManagerID AND nvp.name = 'expensify_callsCompleted' AND DATETIME(JSON_EXTRACT(JSON_EACH.value, '$.date')) BETWEEN DATETIME('now', 'start of month', '-12 months') AND DATETIME('now', 'start of month', '-1 second') AND lower(JSON_EXTRACT(JSON_EACH.value, '$.domain')) = pd.billingOwnerDomain ) as totalCallsLast12M, DATE(li.lastInteractionDate) as lastInteractionDate, d.accountManagerEmail as assignedAM, ( SELECT GROUP_CONCAT(DISTINCT(SUBSTR(SUBSTR(d.email, INSTR(d.email, '@') + 1), INSTR(SUBSTR(d.email, INSTR(d.email, '@') + 1), '@') + 1))) AS domains FROM sharedNameValuePairs snvps INNER JOIN accounts d ON (snvps.ownerAccountID = d.accountID) WHERE snvps.accountID = mainBillingOwnerID AND snvps.name = 'domain_member' ) as domains, ( SELECT GROUP_CONCAT(DISTINCT(ad.email)) FROM domainUsers a INNER JOIN nameValuePairs n ON (a.accountID = n.accountID AND n.name GLOB 'expensify_policy*') INNER JOIN sharedNameValuePairs s ON (s.ownerAccountID = n.accountID AND s.name = n.name) INNER JOIN accounts ad ON (ad.accountID = s.accountID) WHERE a.domain = pd.billingOwnerDomain AND JSON_EXTRACT(n.value, '$.type') NOT IN ('personal', 'free') AND (s.permissions LIKE '%own%' OR s.permissions LIKE '%admin%') ) as policyAdmins, (SELECT GROUP_CONCAT(da.email) FROM domainAdmins da WHERE da.domain = pd.billingOwnerDomain) as domainAdmins FROM purchasesInfoPerDomain pd LEFT JOIN domains d ON (d.domain = pd.billingOwnerDomain) LEFT JOIN lastInteraction li ON (li.domain = pd.billingOwnerDomain);" > RAW_Data.csv

Timed out in 2m 30s with OOO.

But i tested it out yesterday and it ran for 5minutes without OOO. I terminated the query since it can take hours to complete.

So either the OOO timeout is larger or the original issue isn't present. Either way something has improved since last time so until we can reproduce the original issue again its currently a non issue and hence this PR is not needed atm.

[flodnv]

It's possible that the problem was fixed in the email Billing query failed with error #7 (out of memory)

[chiragsalian]

Nice yes, very likely.