Send configured headers from the GeoBlacklight harvester - #213
Draft
thatbudakguy wants to merge 4 commits into
Draft
thatbudakguy wants to merge 4 commits into
thatbudakguy wants to merge 4 commits into
Conversation
json 3 removed the quirks_mode keyword, which json-schema still passes to JSON.parse (json-schema 6.2.0, the current release, does it at validator.rb:447). Every schema validation raises ArgumentError as a result, so all of GeoCombine's validation specs fail. Gemfile.lock isn't checked in, so CI resolves gems fresh on every run and started installing json 3.0.2 as soon as it was released; a checkout with an older lockfile doesn't reproduce it. CI on main passed before that and fails now without a line of source having changed. Constrained in the Gemfile rather than the gemspec so it binds development and CI without propagating to applications that depend on geo_combine. Worth dropping once json-schema stops using quirks_mode. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
thatbudakguy
added this pull request to stack #218
September 17, 2026 23:43
Refactors #index onto a private #each_page enumerator, so the documents harvested from a site can be used for something other than indexing them into Solr (e.g. writing them to disk for your own OpenGeoMetadata repository). Mirrors the docs_to_index seam in GeoCombine::Harvester. Also fixes documents the document transformer omits being sent to Solr as nulls: #index mutated the page with map! and discarded the compacted copy. Closes #207 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moves the requests a harvest makes into an HttpClient that waits out the crawl delay before each one. Previously the delay was applied once per page of search results, so a Blacklight 7 harvest could issue 100 document requests back to back before it took effect. Each request continues to get its own connection rather than reusing one, which is likewise more likely to be accepted by a WAF; the client is now the single place that behavior is decided. Fractional delays work too. A response class builds a client for itself when it isn't given one, and passes along the logger it was given so request logging doesn't quietly go somewhere else. Harvester request specs now stub HTTP with webmock rather than mocking Net::HTTP.get, since the request path is no longer a single class method. Closes #209 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Headers can be configured globally or per site, and are sent with every request a harvest makes. This is a way to get the harvester past a firewall or bot detection (Stanford skips Turnstile for requests carrying a particular header), to authenticate it, or just to give it a User-Agent that identifies it to the sites being harvested. Header names and values are coerced to strings, so symbols work in configuration where Net::HTTP would otherwise reject them. Closes #210 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
thatbudakguy
force-pushed
the
210-configured-headers
branch
from
September 17, 2026 23:43
7c58bcd to
4968894
Compare
thatbudakguy
marked this pull request as draft
September 17, 2026 23:45
thatbudakguy
removed this pull request from stack #218
September 18, 2026 20:18
thatbudakguy
added this pull request to stack #219
September 18, 2026 20:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #210
Headers can be configured globally or per site, and are sent with every request a harvest makes:
If you configure headers at the global level, they get merged with the per-site header configuration.