Describe the bug
fetchAndFilter in shared-libs/cht-datasource/src/local/libs/doc.ts fills a page by fetching more rows when the filter rejects some. It then sets the next cursor with currentSkip + currentLimit - overFetchCount.
That subtraction counts documents, but skip counts rows. When a rejected row sits after the first surplus accepted row, the cursor lands too far to the right, and every accepted document in the gap is lost. Those documents were fetched and passed the filter. .slice(0, limit) discarded them, and the cursor never goes back for them.
The noMoreResults check at line 184 also runs before the full page check at line 188, so a surplus on the last fetch is discarded and the caller gets cursor: null.
Nothing logs and nothing throws. The page comes back full and the cursor chain looks healthy, so a caller cannot detect the loss.
The paths that reach this are the ones whose filter can reject a row: Contact.v1.getPage and Report.v1.getPage with an ids qualifier, where an id is missing or points at another type; the phones qualifier on Contact.v1.getPage and Contact.v1.getUuidsPage, where the phone view emits a document that is not a configured contact type; and offline freetext search for contacts and reports. The contact type, byForms, person, place and target paths do not reach it, because their view keys on the property the filter tests, so nothing is rejected.
There is a second, smaller problem in the same file. fetchAndFilterIds builds its dedup Set inside one page request, so a document can come back on two pages. Only the offline freetext views emit a document more than once, and their one consumer, getIntersection in shared-libs/search/src/search.js, already removes duplicates. A cross page guarantee needs the cursor to carry state, which is a larger change, so this ticket does not ask for it. The rule worth recording is that a caller must dedupe when the view can emit a document more than once.
To Reproduce
Store five documents that are not contacts, then three that are. Read them back by id with a page size of 2.
curl -s -u medic:password -X POST http://localhost:5984/medic/_bulk_docs \
-H 'Content-Type: application/json' -d '{"docs":[
{"_id":"ff-bad-1","type":"data_record","form":"X","reported_date":1700000000000},
{"_id":"ff-bad-2","type":"data_record","form":"X","reported_date":1700000000000},
{"_id":"ff-bad-3","type":"data_record","form":"X","reported_date":1700000000000},
{"_id":"ff-bad-4","type":"data_record","form":"X","reported_date":1700000000000},
{"_id":"ff-bad-5","type":"data_record","form":"X","reported_date":1700000000000},
{"_id":"ff-good-1","type":"person","name":"FF Good 1","reported_date":1700000000000},
{"_id":"ff-good-2","type":"person","name":"FF Good 2","reported_date":1700000000000},
{"_id":"ff-good-3","type":"person","name":"FF Good 3","reported_date":1700000000000}]}'
curl -s -u medic:password "http://localhost:5988/api/v1/contact?ids=ff-bad-1,ff-bad-2,ff-bad-3,ff-bad-4,ff-bad-5,ff-good-1,ff-good-2,ff-good-3&limit=2"
At a page size of 2 the answer is ["ff-good-1","ff-good-2"] with cursor: null. At a page size of 3 it is all three. ff-good-3 is a contact in the requested list, and at a page size of 2 it never arrives, while the null cursor reports the list as complete.
Expected behavior
The page holds limit documents. The cursor points at the first row the page did not use. cursor: null means the result set is complete.
Environment
- App:
cht-datasource, reached through api and webapp
- Version: 4.18.0 and later. The function is unchanged since 4.12.0, but no caller could reject a row before 4.18.0. The released 5.3.0 and 5.3.1 both carry it.
One suggestion for whoever picks this up. Stopping the scan when the page is full, counting the rows consumed, and taking the cursor from that count fixes it. A sweep of every accept and reject pattern up to length 10, at page sizes 1 to 5, gives 1007 wrong results now and none with that change, at the cost of about 9% more requests. Every existing test that asserts an exact cursor value keeps its value, including test/local/libs/doc.spec.ts:608, the one unit test with a real rejection. The regression test belongs in the describe('fetchAndFilter') block of shared-libs/cht-datasource/test/local/libs/doc.spec.ts, and the smallest fixture is six rows at a page size of 2, where rows 1, 2 and 6 fail the filter.
Describe the bug
fetchAndFilterinshared-libs/cht-datasource/src/local/libs/doc.tsfills a page by fetching more rows when the filter rejects some. It then sets the next cursor withcurrentSkip + currentLimit - overFetchCount.That subtraction counts documents, but
skipcounts rows. When a rejected row sits after the first surplus accepted row, the cursor lands too far to the right, and every accepted document in the gap is lost. Those documents were fetched and passed the filter..slice(0, limit)discarded them, and the cursor never goes back for them.The
noMoreResultscheck at line 184 also runs before the full page check at line 188, so a surplus on the last fetch is discarded and the caller getscursor: null.Nothing logs and nothing throws. The page comes back full and the cursor chain looks healthy, so a caller cannot detect the loss.
The paths that reach this are the ones whose filter can reject a row:
Contact.v1.getPageandReport.v1.getPagewith an ids qualifier, where an id is missing or points at another type; the phones qualifier onContact.v1.getPageandContact.v1.getUuidsPage, where the phone view emits a document that is not a configured contact type; and offline freetext search for contacts and reports. The contact type,byForms, person, place and target paths do not reach it, because their view keys on the property the filter tests, so nothing is rejected.There is a second, smaller problem in the same file.
fetchAndFilterIdsbuilds its dedupSetinside one page request, so a document can come back on two pages. Only the offline freetext views emit a document more than once, and their one consumer,getIntersectioninshared-libs/search/src/search.js, already removes duplicates. A cross page guarantee needs the cursor to carry state, which is a larger change, so this ticket does not ask for it. The rule worth recording is that a caller must dedupe when the view can emit a document more than once.To Reproduce
Store five documents that are not contacts, then three that are. Read them back by id with a page size of 2.
At a page size of 2 the answer is
["ff-good-1","ff-good-2"]withcursor: null. At a page size of 3 it is all three.ff-good-3is a contact in the requested list, and at a page size of 2 it never arrives, while the null cursor reports the list as complete.Expected behavior
The page holds
limitdocuments. The cursor points at the first row the page did not use.cursor: nullmeans the result set is complete.Environment
cht-datasource, reached through api and webappOne suggestion for whoever picks this up. Stopping the scan when the page is full, counting the rows consumed, and taking the cursor from that count fixes it. A sweep of every accept and reject pattern up to length 10, at page sizes 1 to 5, gives 1007 wrong results now and none with that change, at the cost of about 9% more requests. Every existing test that asserts an exact cursor value keeps its value, including
test/local/libs/doc.spec.ts:608, the one unit test with a real rejection. The regression test belongs in thedescribe('fetchAndFilter')block ofshared-libs/cht-datasource/test/local/libs/doc.spec.ts, and the smallest fixture is six rows at a page size of 2, where rows 1, 2 and 6 fail the filter.