GeoCombine

A Ruby toolkit for managing geospatial metadata, including:

• harvesting and indexing records from OpenGeoMetadata
• converting geospatial metadata records between standards
• migrating OpenGeoMetadata records from older to newer schema versions

Installation

Tip

If you're using GeoBlacklight, GeoCombine is already bundled with your installation – you can skip this section.

Add this line to your application's Gemfile:

gem 'geo_combine'

And then execute:

bundle install

Or install it yourself as:

gem install geo_combine

Usage

Downloading metadata from OpenGeoMetadata

Schema version

By default, GeoCombine will only fetch and index records using the current schema version. If you instead want to index records using an older schema (e.g. because your GeoBlacklight instance is an older version), you can set the SCHEMA_VERSION environment variable:

# Only fetch and index schema version 1.0 records
export SCHEMA_VERSION=1.0

Restricted records

By default, GeoCombine will only fetch and index records that are not marked as restricted. If you want to include restricted records, you can set the OGM_SKIP_RESTRICTED environment variable to false:

export OGM_SKIP_RESTRICTED=false

GitHub API token

GeoCombine uses the GitHub API to find OpenGeoMetadata repositories. Unauthenticated requests are limited to 60 per hour, and geocombine:clone makes one for each repository. If you hit the limit, set the GITHUB_TOKEN environment variable to a GitHub personal access token to raise it; the token only needs read access to public repositories:

export GITHUB_TOKEN=your-token

Clone OpenGeoMetadata repositories locally

bundle exec rake geocombine:clone

Will clone all OpenGeoMetadata repositories containing metadata matching the SCHEMA_VERSION into ./tmp/opengeometadata. Location of the OpenGeoMetadata repositories can be configured using the OGM_PATH environment variable.

OGM_PATH='my/custom/location' bundle exec rake geocombine:clone

You can also specify a single repository:

bundle exec rake geocombine:clone[edu.stanford.purl]

Note: If you are using zsh, you will need to use escape characters in front of the brackets:

bundle exec rake geocombine:clone\[edu.stanford.purl\]

Update local OpenGeoMetadata repositories

bundle exec rake geocombine:pull

Runs git pull origin master on all cloned repositories in ./tmp/opengeometadata (or custom path with configured environment variable OGM_PATH).

You can also specify a single repository:

bundle exec rake geocombine:pull[edu.stanford.purl]

Note: If you are using zsh, you will need to use escape characters in front of the brackets:

bundle exec rake geocombine:pull\[edu.stanford.purl\]

Index GeoBlacklight documents

Tip

If you're using GeoBlacklight, GeoCombine will automatically detect the path to your Solr index if you run it in the directory where GeoBlacklight is installed.

To index into Solr, GeoCombine requires a Solr instance that is running the GeoBlacklight schema:

bundle exec rake geocombine:index

If Blacklight is installed in the ruby environment and a solr index is configured, the rake task will use the solr index configured in the Blacklight application (this is the case when invoking GeoCombine from your GeoBlacklight installation). If Blacklight is unavailable, the rake task will try to find a Solr instance running at http://localhost:8983/solr/blacklight-core.

You can also set the Solr instance URL using SOLR_URL:

SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:index

Withdrawn records

An OpenGeoMetadata repository can withdraw records by listing them in a withdrawn.json file at its root:

{
  "$schema": "https://opengeometadata.org/schema/ogm-withdrawals-1.0.json",
  "ogm_version": "1.0",
  "withdrawn": [
    { "id": "stanford-bb058zh0946", "date": "2026-03-14", "reason": "superseded" }
  ]
}

After indexing, geocombine:index deletes the records these files list from your Solr index. It is careful about what it deletes:

• A record is only deleted if a withdrawn.json lists its id (or layer_slug_s, for SCHEMA_VERSION=1.0). Removing a record's file from a repository does not delete it from your index.
• A record that is still present in the repository is never deleted, even if it is also listed as withdrawn. GeoCombine logs a warning instead.
• A withdrawn.json that can't be read is logged and skipped; it doesn't stop indexing.
• Repositories listed in OGM_SKIP_REPOS can't delete anything.
• If the harvest stops before it has walked every record, nothing is deleted.

GeoCombine keeps no record of what it has deleted. Every run deletes every withdrawn record again, which changes nothing for records that are already gone. Deletes match by id alone, so a withdrawn id that matches a record you indexed from somewhere other than OpenGeoMetadata will delete that record too.

To delete withdrawn records without reindexing everything, run:

bundle exec rake geocombine:withdraw

Logging

Some of the tools and scripts in this gem use Ruby's Logger class to print information to $stderr. By default, the log level is set to Logger::INFO. For more verbose information, you can set the LOG_LEVEL environment variable to DEBUG:

export LOG_LEVEL=DEBUG

Indexing local documents

To index a collection of records in a local directory, you can pass the directory as a rake argument:

bundle exec rake geocombine:index\[/path/to/your/files\]

Or you can provide it as OGM_PATH:

OGM_PATH=/path/to/your/files rake geocombine:index

For a single file, it works the same way:

bundle exec rake geocombine:index\[/path/to/your/file.json\]

Or:

OGM_PATH=/path/to/your/file.json rake geocombine:index

Converting metadata

Converting metadata into GeoBlacklight JSON

GeoCombine provides several classes representing different metadata standards that implement the #to_geoblacklight method for generating records in the GeoBlacklight JSON format:

GeoCombine::Iso19139 # ISO 19139 XML
GeoCombine::OGP # OpenGeoPortal JSON
GeoCombine::Fgdc # FGDC XML
GeoCombine::EsriOpenData # Esri Open Data Portal JSON
GeoCombine::CkanMetadata # CKAN JSON

An example for converting an ISO 19139 XML record:

# Create a new ISO19139 object
> iso_metadata =  GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')

# Convert to GeoBlacklight's metadata format
> iso_metadata.to_geoblacklight

# Output it as JSON instead of a Ruby hash
> iso_metadata.to_geoblacklight.to_json

Some formats also support conversion into HTML for display in a web browser:

# Create a new ISO19139 object
> iso_metadata =  GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')

# Convert ISO to HTML
> iso_metadata.to_html

Converting metadata into Aardvark JSON

GeoCombine::Fgdc and GeoCombine::Iso19139 also implement #to_aardvark, which converts FGDC or ISO 19139 XML into the OGM Aardvark schema and returns a GeoCombine::GeoblacklightAardvark:

# Create a new Fgdc object
> fgdc_metadata = GeoCombine::Fgdc.new('./tmp/opengeometadata/edu.tufts/0/108/220/208/fgdc.xml')

# Convert to the Aardvark schema
> aardvark = fgdc_metadata.to_aardvark

# Validate it against the Aardvark JSON schema
> aardvark.valid?
=> true

# Output it as JSON instead of a Ruby hash
> aardvark.to_json

Aardvark defines some fields that FGDC and ISO have no source for, such as gbl_suppressed_b, and the links in a record may not be the ones you want to publish. Pass fields in and they are merged into the transformed record, replacing any derived values:

> fgdc_metadata.to_aardvark(
    'id' => 'tufts-ecuador-drilling-towers',
    'schema_provider_s' => 'Tufts',
    'dct_references_s' => { 'http://schema.org/url' => 'https://example.edu/catalog/tufts-1' }.to_json
  )

Note about schema_provider_s and id:

# Without a provider, id and provider are derived from the record
> fgdc_metadata.to_aardvark.metadata['id']
=> "tufts-university-gis-center-ecuador50kdrillingtower11"

# With one, the provider is used as the id prefix
> fgdc_metadata.to_aardvark('schema_provider_s' => 'Tufts').metadata['id']
=> "tufts-ecuador50kdrillingtower11"

# Or set the id yourself
> fgdc_metadata.to_aardvark('id' => 'tufts-ecuador-drilling-towers').metadata['id']
=> "tufts-ecuador-drilling-towers"

The FGDC transformation fills in dct_references_s from the citation's online linkage and the network resources of each digital form, classified by URL and by the format they're distributed in: a landing page, downloads (a labeled list when there's more than one), FGDC and HTML metadata, and OGC (WMS, WFS, WCS, WMTS) and Esri service endpoints. A browse graphic URL becomes the thumbnail, and gbl_wxsIdentifier_s is the layer an OGC request asks for. References you pass in replace the derived ones.

ISO 19139 records (including ISO 19115-2 gmi:MI_Metadata) work the same way. Their ids use the last segment of the fileIdentifier, falling back to the dataSetURI, the citation identifier and the title:

> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')

> iso_metadata.to_aardvark('schema_provider_s' => 'Stanford').metadata['id']
=> "stanford-bb338jh0716"

The ISO transformation also fills in dct_references_s, from the dataset URI, downloads, and OGC (WMS, WFS, WCS, WMTS) and Esri service links, along with gbl_wxsIdentifier_s from the name of an OGC link. References you pass in replace the derived ones.

Some values can't be derived reliably and are left for you to pass in: gbl_resourceType_sm for vector data whose geometry is only described as composite (used for both lines and polygons), and pcdm_memberOf_sm and dct_isPartOf_sm, which take the ids of other Aardvark records.

Migrating metadata

You can use the GeoCombine::Migrators to migrate metadata from one schema to another.

Currently, the only migrator is GeoCombine::Migrators::V1AardvarkMigrator which migrates from the GeoBlacklight v1 schema to the Aardvark schema

# Load a record in geoblacklight v1 schema
record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))

# Migrate it to Aardvark schema
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).run

Some fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:

# You can store this mapping as a JSON or CSV file and load it into a hash
id_map = {
  'My Collection 1' => 'institution:my-collection-1',
  'My Collection 2' => 'institution:my-collection-2'
}

GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).run

Tests

To run the tests, use:

bundle exec rake spec

Some tests need a real Solr core running the GeoBlacklight schema. They delete documents, so they only run when SOLR_INTEGRATION_URL points at a core you don't mind them changing:

SOLR_INTEGRATION_URL=http://localhost:8983/solr/blacklight-core bundle exec rake spec

Contributing

1. Fork it ( https://github.com/[my-github-username]/GeoCombine/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request