Data governance for a federated repository ecosystemDraft

A while ago I wrote a post for the BROCCOLI team arguing that they should borrow a few ideas from data mesh architecture. BROCCOLI is building a data lake: it harvests OpenAlex, OpenAIRE, Crossref and the Dutch CRIS systems into one place and refines them through bronze, silver and gold layers into dashboards. Data products, data contracts and a marketplace fit that picture well.

At first sight, the Dutch Repository Federation (DURF) looks like a different animal. The publications themselves, and the authoritative records about them, stay where they are: at the universities. The information chain in the position paper runs through systems that already exist and are run by different parties:

• seventy-odd repositories and CRIS systems at the universities and medical centres;
• OpenAIRE's harvester, Graph and Broker;
• the KB's e-Depot and URN:NBN resolver;
• the Netherlands Research Portal;
• publishers at one end and the world's search engines at the other.

The DURF proposal puts it plainly: the project "does not build a single technical infrastructure with a single point of failure. Rather it leverages existing components and standards … and creates the necessary links between them."

So no lake, then? That's what I thought when I started this post. But follow the metadata, not the files, and a lake shows up after all:

• The OpenAIRE Graph harvests the metadata of all Dutch research output, source by source. That's bronze.
• It then deduplicates it and unites it with everything else it knows, from Crossref, ORCID, ROR, DataCite and thousands of other repositories. That's silver.
• Finally it serves that out for specific uses: the Netherlands Research Portal, the KB e-Depot and, through the e-Depot, the URN:NBN resolver. That's gold.

That's a data lake with medallion layers. It just isn't ours: OpenAIRE runs it, and the federation pays for it through its licence contributions. It turns out to be the key to how the whole chain hangs together, so I've given it its own section below.

Two things carry over from the BROCCOLI post unchanged, and in a federation they matter even more: data as a product, and data contracts first.

The idea in one paragraph

Every link in the DURF chain relies on the link before it. The KB can only preserve a file if the repository says where the file is. The Portal can only show a university's output if the records carry that university's ROR ID. Google Scholar can only index an article if the landing page has the right meta tags. Today those dependencies are mostly implicit, and they hold as long as someone happens to remember them. My proposal is to make each one explicit. At every point where data passes from one party to another, the delivering party treats what it hands over as a product, owned by a named data team. It then writes down what it promises in a data contract: which fields, in which format, how complete, how fresh, and how you can check. The receiving party builds against the contract, not against whatever it happened to find last week.

What it looks like

Here's the DURF information chain again, redrawn as data products. Each dashed frame is one data product, with the party that owns it. Each purple box is a contract: an output port that the producer promises to keep. The contract lives with the producer, because the producer is the one making the promise. Purple arrows are the dependencies the contract covers.

The seven example contracts are in the contracts/ folder of this repository. They're all drafts. Each one is owned by one work package of the DURF project, which gets it agreed and adopted. More on that below.

The OpenAIRE Graph is our data lake

In the BROCCOLI post, the medallion layers were the backbone of the design. They're here too; they just sit inside one party. Here's the same table for DURF:

C2 (landing pages for search engines) and C3 (deliveries from publishers) sit outside the lake. They go straight from or to the repositories.

Seeing the Graph as a lake makes three things clear.

DURF doesn't run the lake, so it steers at the edges. BROCCOLI builds and owns its whole lake, from bronze to gold. DURF only owns what goes in and what comes out. The bronze contract (C1) and the gold contracts (C4, C5, C6) are DURF's grip on the lake. Silver, the matching and deduplication, is OpenAIRE's craft and OpenAIRE's responsibility. DURF's lever there is the licence, negotiated through NaMeCo, not a say in the pipeline.

This time we produce our own bronze. BROCCOLI has to take OpenAlex as it comes, so its bronze contracts can only describe what arrives. In DURF, the Dutch bronze comes from the federation's own members. That makes C1 a real promise, not a description: when the Portal needs a ROR ID in gold, the fix is at the source that feeds bronze, and that source is one of us.

A lake is not a single point of failure here. The proposal says DURF shouldn't build a single point of failure, and it doesn't. The Graph holds a copy of the metadata, not the only copy. The authoritative records and the full texts stay at the universities. The bronze can be harvested again at any time. The Graph is published as open dumps on Zenodo. And the e-Depot keeps its own preserved copy. If the lake ever went away, it could be rebuilt from the sources. That's what federating around a lake, rather than into one, means.

There's a nice bonus. SURF's BROCCOLI lake reads the same data for two of its bronze sources. Its openaire_publications is the Dutch slice of the Graph, from the community data dump or the community filter in the Graph API: exactly C5. Its cris_<institution> harvests the same CRIS endpoints that C1 covers. So the gold that DURF gets from OpenAIRE is bronze for BROCCOLI: lakes feeding lakes, held together by contracts. BROCCOLI doesn't need its own bronze contracts for these two sources. It can reuse C1 and C5, and when a university improves its metadata to meet C1, the Portal and BROCCOLI's quality dashboard both see it.

Every university gets a data team

In data mesh, a data product isn't just a dataset. It's the data plus the code that produces it, the contract that describes it, and above all a team that's accountable for it. For DURF that means every participating institution runs its repository or CRIS as a data product called cris_publications, owned by a small data team with three roles:

• Data owner: the library director or head of research support. Accountable for the institution keeping its contracts, and the person who signs the position paper.
• Data steward: the repository or CRIS manager. Looks after metadata quality, reads the validator reports and acts on the Broker's suggestions.
• Data engineer: the functional or technical CRIS administrator. Keeps the OAI-PMH endpoint, the CERIF mapping and the landing pages running.

That sounds like a lot of new people, but it isn't. These are the people who already run the repository. The DURF proposal counts on "approximately 0.1–0.2 FTE per institution allocated from existing repository and CRIS managers' responsibilities" after the project ends. What changes isn't the headcount. It's that their work now has a clear shape: here is our product, here is what we promise, here is how anyone can check.

The same goes for the other parties. OpenAIRE runs openaire_graph as one data product with three output ports: enrichment events for the institutions, the Dutch slice of the Graph for the Portal and the indexes, and a preservation feed for the KB. The KB runs edepot_preservation and reports back what it has safely stored. Each has its own data team, its own tooling and its own release rhythm. The only thing the others see is the contract.

This is also what the position paper means when it says parties, not systems, take on commitments. A CRIS can't sign a pledge. The data team behind it can, and a data contract is how that pledge becomes specific enough to check. Each contract makes one cell of the roles and responsibilities matrix concrete: the matrix says who can expect something from whom, the contract says exactly what.

The first contract: a Dutch application profile

Everything downstream depends on C1, the contract between each institution and OpenAIRE. If a record reaches the Graph without a DOI, the Graph can't deduplicate it. If it arrives without a ROR ID, the Portal can't attribute it to the right university. Without a file location, the KB has nothing to preserve. Without an access right, nobody downstream knows what they may show.

The good news is we don't have to invent C1. The OpenAIRE Guidelines for CRIS Managers 1.2 already define the structure and vocabularies:

• CERIF-XML over OAI-PMH, with metadata prefix oai_cerif_openaire and the set openaire_cris_publications;
• COAR vocabularies for resource types, access rights and version types.

What the guidelines can't know is what Dutch consumers downstream rely on. So C1 takes the guidelines as its base and tightens a handful of fields. That tightened contract is the Dutch application profile the DURF proposal wants to develop, replacing the current NL-DIDL/MODS requirements.

Written down as a data contract, the relevant part looks like this. Most of the file is descriptions, because the descriptions are the point: a CRIS manager at any university should be able to read it and know what to do.

# contracts/institutions/durf-cris-publications-v1.odcs.yaml (shortened)
apiVersion: v3.1.0
kind: DataContract
id: durf-cris-publications-v1
name: Institutional publications, Dutch application profile of the
      OpenAIRE Guidelines for CRIS Managers 1.2
version: 0.1.0
status: draft
domain: institutional-research-output
dataProduct: cris_publications
customProperties:
  - property: durfTheme
    value: 2
  - property: oaiPmhMetadataPrefix
    value: oai_cerif_openaire
  - property: replaces
    value: NL-DIDL/MODS (EduStandaard)
team:
  name: institution-data-team
  members:
    - { username: data-owner,    role: Data owner }
    - { username: data-steward,  role: Data steward }
    - { username: data-engineer, role: Data engineer }
slaProperties:
  - { property: latency, value: 7, unit: d }        # in CRIS → on the endpoint
  - { property: retention, value: persistent }      # deletions are announced, never dropped
servers:
  - server: production
    type: api
    location: https://cris.institution.example/ws/oai?verb=ListRecords&...
schema:
  - name: publication
    physicalName: cerif:Publication
    properties:
      - name: doi
        physicalName: DOI
        description: >-
          Mandatory whenever the publication has one (MA). The OpenAIRE
          guidelines leave it optional; NL requires it, because it is the
          main deduplication key in the Graph and for the indexes.
      - name: authors
        items:
          properties:
            - name: orcid                   # MA in NL (R in OpenAIRE)
            - name: affiliation_ror         # MA in NL: at least the own institution
      - name: access_rights
        required: true                      # M in NL: e-Depot, Portal and indexes branch on it
      - name: embargo_end_date              # NL addition, for Taverne
      - name: file_locations                # MA for every file, whatever its access rights
    quality:
      - type: custom
        engine: openaire-provide-validator
        description: >-
          The OpenAIRE PROVIDE validator for the CRIS Guidelines 1.2 profile
          passes with no errors. This is the contract test.
      - type: text
        description: Every author affiliated with the producing institution
          carries the institution's ROR ID.

Two things I like about this. First, the authoritativeDefinitions in the full file point to the OpenAIRE guidelines. So the Dutch profile is visibly a thin layer on top of an international standard, not a national fork. Second, the quality section says out loud what the contract test is: the OpenAIRE PROVIDE validator, which already runs on every harvest. The Dutch additions, like "every own author has the ROR ID", are simple enough to check next to it.

Who owns the contract itself, as opposed to the data? In a federation, not the individual institution: then there would be seventy slightly different profiles. The DURF proposal has the answer ready. A NL Research Information Content Board, formed through EduStandaard, stewards the profile. Institutions implement it. The board changes it through a public process. That's federated governance in its purest form: we agree on a few rules together, and each data team makes sure they're met.

OpenAIRE and the KB are data teams too

It would be easy to read all this as "the universities must deliver, and everyone else consumes". But the chain only works if the promises run both ways. In the diagram, four of the seven contracts are kept by OpenAIRE or the KB, not by the universities:

• C4, enrichment events. Once the Graph has aggregated and deduplicated a record against every other source, the Broker tells the institution what it could add: a missing DOI, an ORCID, a funder, an open access copy found elsewhere. The contract fixes the topics, the trust score, and that every event points to the record's own ID in the source CRIS. It also says how long an event stays available. That's the feedback loop of theme 2, and it only works if the institution can rely on it.
• C5, the Dutch Graph slice. The dataset behind the Netherlands Research Portal, and the open Dutch slice that other indexes and EOSC nodes can take from the Graph API and data dumps. The key promise: every record harvested through C1 is in there, and the source CRIS ID travels with it, so any consumer can trace a record back to its producer.
• C6, the e-Depot feed. An incremental feed of Dutch publications, one row per full-text file: URL, MIME type, checksum and access right. Deletions are announced, never silently dropped. The KB builds one harvester instead of seventy, and gets records that are already deduplicated.
• C7, preservation status. The KB's side of the bargain. For every file it pulled, it reports back whether it's safely preserved, under which URN:NBN, and whether the last fixity check passed. The institution writes the URN:NBN back into its CRIS, and it flows into C1 again as an identifier. The loop closes.

A word on the "Broker to Graph" and "Graph to e-Depot" steps. PROVIDE, the Graph and the Broker are all run by OpenAIRE, so I've drawn them as one data product with three output ports, rather than three products with contracts between them. Contracts matter where responsibility changes hands. Inside OpenAIRE, OpenAIRE may well use contracts between its own teams, and that would be good practice. But DURF doesn't need to see them. What DURF needs is a firm promise at each point where data leaves OpenAIRE: towards the institutions (C4), the Portal and the indexes (C5), and the KB (C6). In lake terms, bronze, silver and gold all live inside this one product, and DURF's contracts sit at its edges: C1 going in, C4, C5 and C6 coming out.

Contracts with parties outside DURF

Two of the contracts cross the federation's border, and they behave differently.

C3, publisher deliveries, runs from publishers into the repositories: JATS metadata (CC0) plus the full-text files, delivered over SWORD, routed on the institution's ROR ID. Publishers don't sign the DURF position paper. So this contract only has force where it's written into an agreement, such as the open access terms of a national read-and-publish deal. Without one, it still has a use: it describes exactly what a DURF repository is ready to receive. Handing a publisher a contract that fits on one page makes that conversation a lot easier.

C2, discovery, runs the other way, from the repositories to Google Scholar, OpenAlex, BASE, CORE and anyone else with a crawler. Those consumers will never sign anything either. Here the contract is a one-sided promise: every publication gets a stable landing page with citation_* meta tags, a working citation_pdf_url, and schema.org JSON-LD, all listed in a sitemap. DURF checks that promise on behalf of the consumers.

The monitors in the proposal are contract tests

This is where it clicked for me. The DURF proposal's software management plan lists five monitors to build. Put them next to the contracts:

Every monitor is a contract test, and every contract gets a monitor. That gives DURF a nice design rule: no monitor without a contract, and no contract without a monitor. The contract says what "good" means. The monitor shows, per institution, how far off we are. And because the contract is written by the whole federation, a red cell on a dashboard isn't one party pointing at another. It's everyone looking at the same agreement.

We might not even have to build the dashboard from scratch. The ORI Quality Monitoring dashboard from PID2Portal already shows data stewards how complete their institution's metadata is, per source. A DURF tab that reads the contract tests would sit very naturally next to it.

What this means for each work package

The DURF proposal splits the project into six work packages, the themes, each with its own activities and months. So how does all this land in the project plan? The short version:

• Every contract has one work package that owns it and gets it agreed and adopted.
• The activities in that work package write the contract, build what it needs, or test it. None of this is extra work on top of the plan. It gives work the plan already contains a concrete, checkable output.
• Theme 1 owns no single contract. It provides the agreements the contracts are attached to.

The months are those in the proposal.

Theme 1 · Governance of the Federation and the Portal

Owns: the agreements that the contracts belong to.

Theme 2 · Metadata Quality Improvement

Owns: C1 (the Dutch application profile) and C4 (enrichment events).

Theme 3 · Full Text Capture Enhancement

Owns: C3 (publisher deliveries), together with Theme 1 for the agreements with publishers.

Theme 4 · KB e-Depot Archiving

Owns: C6 (the e-Depot feed) and C7 (preservation status).

Theme 5 · Distribution & Discovery Optimization

Owns: C2 (landing pages and sitemaps).

Theme 6 · Portal Development and Enhancement

Owns: C5 (the Dutch slice of the Graph).

Each contract file records the same thing in two properties, workPackage and workPackageActivities, so you can see from the YAML which part of the project it belongs to.

Contracts first, starting from the consumer

In the BROCCOLI post, my main plea was to start from the user journey and not from the data. The same holds here, just with the chain instead of a dashboard. Don't start by asking "what can our CRIS export?". Start at the far end of the chain and work back:

1. What do the end consumers need? The e-Depot needs files with checksums. The Portal needs ROR IDs and access rights. The indexes need DOIs and PDF links. That fixes the gold contracts C5 and C6, plus C2.
2. What must OpenAIRE receive to deliver that? That fixes C1, the contract for what goes into bronze. It's why C1 is stricter than the plain OpenAIRE guidelines. In between, ask OpenAIRE whether its silver can connect the two ends.
3. What do the institutions need back to keep their promise? That fixes C4 and C7, and what they need from publishers fixes C3.
4. Only then: the mapping in each CRIS, the OAI-PMH configuration, the harvesting and feeds. Each data team builds its own part with its own tools, and tests it against the contract.

A house rule to go with it: no field in the Dutch profile without a consumer downstream who needs it. Every NL tightening in C1 names who relies on it. If nobody does, it goes. That keeps the profile small, and it gives the Content Board a simple test for every change request.

How contracts change

Contracts are YAML files in a git repository, so they change the way code does: through a pull request, reviewed by the parties who depend on them. I'd suggest a few simple rules.

• Adding an optional field is a minor version (0.2.0). Consumers can ignore it.
• Making a field mandatory, or removing one, is a new major version (durf-cris-publications-v2). It runs next to v1 for an agreed transition period, and no consumer is broken overnight.
• Changes to C1 go through the Content Board. Changes to the other contracts are agreed between producer and consumers, with DURF governance as referee.
• Every contract says in its customProperties which work package owns it, which activities in the proposal it delivers, and which arrow in the diagram it governs. That keeps the position paper, the project plan and the contracts from drifting apart.

The move from NL-DIDL/MODS to the CERIF profile is, in these terms, exactly such a major version change. We'll learn how to run a transition period by doing that one well.

Things I'm not sure about yet

• CERIF and Dublin Core. The proposal talks about translating NL-DIDL/MODS to "OpenAIRE CERIF and DC guidelines". Not every repository is a CRIS; some will expose the OpenAIRE Guidelines for Literature Repositories instead. In contract terms, that's one logical schema with two physical output ports. I think it should be one contract with two servers rather than two contracts, but I'd like to hear from the Content Board.
• Restricted and embargoed files. The KB should preserve those too, but the Graph only has what's public. The e-Depot may need direct, authenticated access to each repository for those files. That would be an eighth contract, between the KB and each institution.
• How Broker events arrive. Broker subscriptions, API pull, or COAR Notify straight into the repository? C4 fixes the content and leaves the channel open. Maybe that's right; maybe it should pick one.
• The exact field lists. The obligation levels I give for the OpenAIRE guidelines, and the Graph field names in C5 and C6, are my reading. They need checking against the published XSD and the current Graph API before anyone relies on them.
• Who owns what in the project. Which work package owns which contract is my reading of the proposal. The theme coordinators may well divide it differently, and they should have the final say.
• Depending on someone else's lake. If the Graph merges two Dutch records wrongly, or a release is late, the Portal and the e-Depot feel it. C5 and C6 give DURF service levels to point at, but silver stays out of our hands. I think that's the right trade-off for a federation of our size. But the exit route, harvesting our own bronze again from the sources, should stay open and be tested now and then.
• Enforcement. A contract between federation members is backed by the position paper and the NaMeCo agreements. What happens when an institution doesn't keep it for a year? I think the honest answer is "it shows up red on the dashboard, and the other members ask about it". That may well be enough. But it's worth saying out loud.

So, where to start? A timeline

Put together, this is how the contracts could move through the project. The months and themes are the proposal's own.

Meanwhile, the contracts live in this repository. datacontract lint should run on every pull request, and each contract should be linked from the pledge in the position paper that it makes concrete.

None of this needs a lake of our own, a new platform or a big central team. It needs each party to say out loud what it delivers, write that down in a format the others can read and test, and keep the promise. That's what makes Dutch research output resilient, open and global: not one strong system, but a chain of kept promises.

The contracts are drafts and so is this post. If you run a repository, a CRIS, the Graph or the e-Depot and you think I've got your part wrong, please open an issue or send a pull request. That's how contracts are supposed to change anyway.