This assumes that your cached endpoint is using Varnish and has the xkey module enabled.
You can take a look at our custom varnish-post repository that comes with all required configuration for the cached endpoint.
npm install # Install dependencies
cp example.env .env # Copy environment variables file
# Open your editor, and fill the environment variables in the `.env` file
npm run start # Start the script| Name | Description | Default Value |
|---|---|---|
| CACHE_ENDPOINT | The URL of the cache endpoint | "" |
| CACHE_ENDPOINT_USERNAME | The username for the cache endpoint | "" |
| CACHE_ENDPOINT_PASSWORD | The password for the cache endpoint | "" |
| CACHE_DEFAULT_ENTRY_NAME | The default entry name for the cache | "default" |
| CACHE_TAG_HEADER | The header name for the cache tag | "xkey" |
| SUPPORT_URL_ENCODED | Whether to clear the cache for the URL-encoded version of the dataset URI | "true" |
| SPARQL_ENDPOINT_URL | The URL of the SPARQL endpoint | "" |
| SPARQL_USERNAME | The username for the SPARQL endpoint | "" |
| SPARQL_PASSWORD | The password for the SPARQL endpoint | "" |
| S3_ENABLED | Whether to use S3 for caching | "false" |
| S3_LAST_TIMESTAMP_KEY | The key for the last timestamp file in S3 | "last_timestamp.txt" |
| S3_SIMPLE_DATE_WORKAROUND_KEY | The key for the simple date workaround file in S3 | "simple_date_workaround.txt" |
| S3_BUCKET | The S3 bucket name | "default" |
| S3_ACCESS_KEY_ID | The S3 access key ID | "" |
| S3_SECRET_ACCESS_KEY | The S3 secret access key | "" |
| S3_REGION | The S3 region | "default" |
| S3_ENDPOINT | The S3 endpoint | "" |
| S3_SSL_ENABLED | Whether to use SSL for S3 | "false" |
| S3_FORCE_PATH_STYLE | Whether to force path style for S3 | "false" |
If S3_ENABLED is set to true, the first time you run the script you might see an error message saying that the last timestamp file does not exist. This is expected, and the script will create the file automatically at the end of the first run, and will update that file every time it runs.
You will not see this error message again after the first run.
You might also get a similar error about a simple date workaround file. This is also expected, and the script will create the file automatically at the end of the first run, and will update that file every time it runs.
Using S3 allows us to trick a bit for the cases where dateModified returned by the SPARQL query is a date and not a dateTime.
The trick makes sure that the cache is invalidated for this entry only the first time, and the day after the dateModified date.
Without this trick, the cache would be invalidated every time the script runs until the day after its value.
The tests run the script for real against the services in compose.yaml, so Docker needs to be running:
npm run test:services # Start the services and wait for them to be ready
npm test # Run the tests
docker compose down -v # Stop the services once you are done| Service | Image | Role |
|---|---|---|
oxigraph |
ghcr.io/oxigraph/oxigraph |
The SPARQL endpoint the script queries. Its store lives in a tmpfs and starts empty. |
sparql-proxy |
node + test/fixtures/xkey-proxy.js |
Tags every response with the IRIs the query mentions, so that Varnish has something to purge. |
varnish |
ghcr.io/zazuko/varnish-post |
The cache the script clears. |
s3 |
adobe/s3mock |
The S3 implementation the state files are written to. |
The sparql-proxy service stands in for the production SPARQL proxy: Varnish can only invalidate by xkey if the backend tags its responses with one, and Oxigraph does not do that on its own. Responses that mention no dataset are tagged with the default entry name instead.
The s3 service is S3Mock, a test double rather than an object store: it accepts any credentials and throws its data away when it stops. The script only ever reads and writes two small objects, so nothing here depends on a real S3 implementation.
If the default ports are taken on your machine, set OXIGRAPH_PORT, VARNISH_PORT or S3_PORT — the tests read the same variables:
VARNISH_PORT=9090 docker compose up -d --wait
VARNISH_PORT=9090 npm testThe tests run the script from an empty working directory, so your own .env file is never picked up, and with TZ=UTC, because the detection of a dateModified that is really a date rather than a timestamp depends on the local timezone.
This project is licensed under the MIT License - see the LICENSE file for details.