Syncing issues with rebuilt Nouveau index after archiving

[jkuester]

Describe the bug

Replication errors have been observed on one production instance when a Nouveau index was dropped and rebuilt after documents had been purged from the CouchDB using the new archiving endpoint.

Extensive investigation has identified several CouchDB issues that are involved:

• [Bug]: Error querying Nouveau indexes after purging apache/couchdb#6118
• [Bug]: View requests fail with a 500 compaction error if queried in between purges apache/couchdb#6132

We are working with the Couch team to fix/validate these issues and plan to include the patched version of Couch in an upcoming CHT release.

The core problem, as it is experienced in the CHT, is that re-building a Nouveau index (such as the one used for driving the user replication logic) from scratch (e.g. updating the design doc or dropping/re-adding the design doc) when documents have been purged from the Couch database (e.g. with the /archive endpoint) simply fails. The Nouveau service hits an error. Without that index, the server is unable to calculate which docs belong to which user and so users are not able to sync.

Expected behavior
It should work with no errors.

Environment

• Version: 5.3.0

Additional context
@eljhkrr @witash Please supply any additional corrections or details! 🙏

[jkuester]

I was doing some poking around for a workaround, specifically to the invalid_start_purge_seq error from apache/couchdb#6118. (That is the persistent error that blocks replication.)

Couch/Nouveau just communicate with each other over HTTP and Claude dug up some details for me that you can set the Nouveau purge_seq value via POST http://nouveau:5987/.index/<node>/<shard>/<index_signature> with the body:

{
  "match_purge_seq": 0,
  "purge_seq": "<actual_purge_seq>",
  "commit": true
}

I have done some testing and it seems to work that if you have an index stuck on the invalid_start_purge_seq error, you can unblock the index rebuild by just manually setting the actual purge sequence for the DB into Nouveau with this endpoint.

You can use this shell script (run it in the CouchDB's container) to automatically set the correct values. It works for a single-node Couch instance, but should be easily ported for multi-node Couch if needed.

@witash can you see anything that I am missing here? Seems like this would get the indexes rebuilt without missing anything or needing a custom Couch image. Of course, if the indexes got re-built again later, the invalid_start_purge_seq error would return and the script would have to re-run. But, do you think this might work to unblock anyone who hits this error before we get an actual Couch fix released?

[jkuester]

For the record, the production instance which hit this issue is now reporting that functionality has been restored after updating to use a Couch image built from the main branch of the Couch repo. This is consistent with what we expected based on our tests.

There are still a couple ancillary problems documented in the issues linked above that we are trying to chase down and resolve before moving forward with a patch release to resolve this issue.