Skip to content

Send configured headers from the GeoBlacklight harvester - #213

Draft
thatbudakguy wants to merge 4 commits into
209-per-request-crawl-delayfrom
210-configured-headers
Draft

thatbudakguy wants to merge 4 commits into
209-per-request-crawl-delayfrom
210-configured-headers

Conversation

@thatbudakguy

@thatbudakguy thatbudakguy commented Sep 17, 2026 •

Copy link
Copy Markdown
Member

Closes #210

Headers can be configured globally or per site, and are sent with every request a harvest makes:

GeoCombine::GeoBlacklightHarvester.configure do
  {
    headers: { 'User-Agent' => 'GeoCombine harvester (you@example.edu)' }, # all sites
    earthworks: {
      host: 'https://earthworks.stanford.edu/',
      headers: { 'foo' => 'bar' }, # earthworks only
      params: { f: { schema_provider_s: ['Stanford'] } }
    }
  }
end

If you configure headers at the global level, they get merged with the per-site header configuration.

json 3 removed the quirks_mode keyword, which json-schema still passes to
JSON.parse (json-schema 6.2.0, the current release, does it at
validator.rb:447). Every schema validation raises ArgumentError as a
result, so all of GeoCombine's validation specs fail.

Gemfile.lock isn't checked in, so CI resolves gems fresh on every run and
started installing json 3.0.2 as soon as it was released; a checkout with
an older lockfile doesn't reproduce it. CI on main passed before that and
fails now without a line of source having changed.

Constrained in the Gemfile rather than the gemspec so it binds
development and CI without propagating to applications that depend on
geo_combine. Worth dropping once json-schema stops using quirks_mode.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thatbudakguy
thatbudakguy added this pull request to stack #218 September 17, 2026 23:43
thatbudakguy and others added 3 commits September 17, 2026 16:43
Refactors #index onto a private #each_page enumerator, so the documents
harvested from a site can be used for something other than indexing them
into Solr (e.g. writing them to disk for your own OpenGeoMetadata
repository). Mirrors the docs_to_index seam in GeoCombine::Harvester.

Also fixes documents the document transformer omits being sent to Solr as
nulls: #index mutated the page with map! and discarded the compacted copy.

Closes #207

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Moves the requests a harvest makes into an HttpClient that waits out the
crawl delay before each one. Previously the delay was applied once per
page of search results, so a Blacklight 7 harvest could issue 100
document requests back to back before it took effect.

Each request continues to get its own connection rather than reusing one,
which is likewise more likely to be accepted by a WAF; the client is now
the single place that behavior is decided. Fractional delays work too.

A response class builds a client for itself when it isn't given one, and
passes along the logger it was given so request logging doesn't quietly
go somewhere else.

Harvester request specs now stub HTTP with webmock rather than mocking
Net::HTTP.get, since the request path is no longer a single class method.

Closes #209

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Headers can be configured globally or per site, and are sent with every
request a harvest makes. This is a way to get the harvester past a
firewall or bot detection (Stanford skips Turnstile for requests carrying
a particular header), to authenticate it, or just to give it a User-Agent
that identifies it to the sites being harvested.

Header names and values are coerced to strings, so symbols work in
configuration where Net::HTTP would otherwise reject them.

Closes #210

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thatbudakguy
thatbudakguy marked this pull request as draft September 17, 2026 23:45
@thatbudakguy
thatbudakguy removed this pull request from stack #218 September 18, 2026 20:18
@thatbudakguy
thatbudakguy added this pull request to stack #219 September 18, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support sending configured headers in the GeoBlacklightHarvester

1 participant