pyEuropePMC

Multi-source search

UnifiedSearch sends one query to several literature sources in parallel, rewrites it for each source, and merges the answers into one deduplicated list of LiteratureResult records. Each source client can also be used on its own. This page covers both, the source registry, and the normalization helpers the clients share.

Quick start

from pyeuropepmc.features.search import UnifiedSearch

with UnifiedSearch() as searcher:  # Europe PMC, PubMed and arXiv
    results, report = searcher.search("chronic fatigue syndrome", limit=25)

print(f"{len(results)} records, {report.duplicates_removed} duplicates removed")
print(report.metadata["source_counts"])  # records returned by each source
print(report.metadata["source_errors"])  # {source: message} for sources that failed
for paper in results[:5]:
    print(paper.source, paper.publication_year, paper.title)

UnifiedSearch is also exported as pyeuropepmc.UnifiedSearch.

UnifiedSearch

UnifiedSearch(sources=None, dedup_mode=DedupMode.BALANCED, timeout=30, rate_limit_delay=None, api_key=None, max_workers=None, translate=True, credentials=None, primary=..., primary_limit_factor=2.0, rate_limits=None, source_concurrency=1)

Parameter Type Default Description
sources list[str] or None None Source keys from the registry; None means ["europepmc", "pubmed", "arxiv"]. An unknown key raises ValueError
dedup_mode DedupMode DedupMode.BALANCED Duplicate matching mode, from pyeuropepmc.features.enrich.merger; see Deduplication
timeout int 30 Request timeout in seconds, passed to each client that accepts it
rate_limit_delay float or None None Delay passed to every client, replacing the client’s own default. None means 1.2 seconds, or 3 for arXiv, DBLP and Zenodo; see Rate limits
api_key str or None None Shortcut for credentials={"api_key": ...}
max_workers int or None None Thread-pool size: how many sources one search() queries at once; None uses one thread per source
translate bool True Rewrite the query for each source; see Query translation
credentials dict or None None Credentials for sources that accept them; see Credentials
primary str or None "europepmc" if selected, otherwise None Source whose records anchor the merge and win ties; None treats all sources equally
primary_limit_factor float 2.0 The primary source is asked for limit times this many records
rate_limits dict[str, float] or None None Delay in seconds per source; wins over rate_limit_delay for the sources it names. A name that is not in sources raises ValueError
source_concurrency int or dict[str, int] 1 How many calls one source may get at the same time from all threads sharing the object; see Rate limits

Clients are created at the first search. Use UnifiedSearch as a context manager or call close() to close them.

search(query, limit=25, sort=None, sources=None, diagnostics=False, **kwargs) -> tuple[list[LiteratureResult], MergeReport]

Parameter Type Default Description
query str required Query string
limit int 25 Records requested from each source; the primary source is asked for limit × primary_limit_factor
sort str or None None "relevance", "date" or "citations", translated into each source’s own sort vocabulary; see below
sources list[str] or None None Query only these configured sources
diagnostics bool False Add a status record per source to report.metadata["source_status"] and matched_sources to every record; see Source status and provenance
**kwargs     Passed to the sources whose search() accepts them

sort takes one of three canonical values, which UnifiedSearch rewrites into what each source expects:

sort Europe PMC PubMed arXiv OpenAlex Semantic Scholar
"relevance" default order relevance default order relevance default order
"date" P_PDATE_D desc pub_date date date publicationDate:desc
"citations" (alias "citation_count") CITED desc not supported not supported citation_count citationCount:desc

A source that cannot sort the way you asked keeps its own default order instead of receiving a value it would reject. Any other sort value is passed through unchanged, so a source-native value such as sort="P_PDATE_D desc" still works when Europe PMC is the only source you query.

A failing source does not stop the others. report is a MergeReport, and report.metadata holds:

Key Content
source_counts Records each source returned, before deduplication
source_errors {source: message} for sources that failed, including "client not initialised"
effective_limits {source: {"rate_limit_delay", "max_concurrency"}} for the queried sources: the limits that applied to this search
source_status With diagnostics=True: the status record of each source; see Source status and provenance
source_times Seconds from the start of the parallel requests until each source finished
sources_used The sources queried
primary_source, primary_records, added_by_source With a primary source: its name, how many merged records match one of its records, and how many new records each other source added

Sources named in a call

search(query, sources=[...]) may name any registered source, not only the ones the object was built with. A source that is not yet in use gets its client on first use, with the same credentials, timeout and delay as the others and one slot of concurrency, and a source that cannot be created (a missing optional package) is reported as unavailable with the reason instead of silently contributing nothing. An unknown name raises ValueError before any request.

Source status and provenance

report.metadata always says how many records each source returned (source_counts) and which sources raised (source_errors). It does not say why a source returned nothing, or whether it was asked at all. With diagnostics=True the result keeps its shape and gains two things:

results, report = searcher.search("chronic fatigue syndrome", limit=10, diagnostics=True)
for name, row in report.metadata["source_status"].items():
    print(name, row["status"], row["records"], row["error"])
print(results[0].source, results[0].matched_sources)  # europepmc ['europepmc', 'pubmed']
Field Content
status ok: records came back. empty: the source answered with none; a client that caught its own failure and returned nothing (an HTTP 403, a page that is not JSON) is error instead, not empty. error: the request raised. rate_limited: it raised HTTP 429 or a rate-limit message. unavailable: the client could not be created (a missing optional package, for instance), so no request was made
records Records returned, before deduplication
seconds Seconds from the start of the parallel requests until the source finished; null for unavailable
limit Records asked for (the primary source is asked for limit × primary_limit_factor)
query, sort The query after translation and the sort value actually sent (null when the source cannot sort that way); null for unavailable
error null, or {"type", "message", "http_status"}; for unavailable the message is why the client could not be created
retry_after Seconds from the server’s Retry-After header for a rate_limited source, when it sent a number
rate_limit_delay, max_concurrency The limits that applied to the source
waited_seconds How long the call waited for a free slot under max_concurrency before it ran

The rows are plain JSON. A failing or unavailable source never stops the others, with or without diagnostics; search_all does not take the option.

search_all

search_all(query, limit=25, **kwargs) -> tuple[dict[str, list[LiteratureResult]], dict[str, str]]

Queries every configured source without deduplication and returns (results_by_source, errors_by_source):

from pyeuropepmc.features.search import UnifiedSearch

with UnifiedSearch(sources=["pubmed", "arxiv", "clinicaltrials"], rate_limit_delay=3.0) as searcher:
    per_source, errors = searcher.search_all("biomarkers", limit=10)

for source, papers in per_source.items():
    print(f"{source}: {len(papers)} records")
print(errors)

Query translation

With translate=True the query is rewritten for each source by pyeuropepmc.features.search.translate_query(query, source). For TITLE:"gene editing" AND CRISPR:

Sources Query sent
europepmc, pubmed, clinicaltrials TITLE:"gene editing" AND CRISPR, passed through
arxiv all:"gene editing CRISPR"; a query that already uses arXiv prefixes such as ti: or cat: is sent as written
openalex, semantic_scholar, core, zenodo, doaj, dblp, hal gene editing CRISPR, with field tags, Boolean operators, quotes and parentheses removed

Field syntax is not converted between Europe PMC and PubMed, so a fielded query written for one of them reaches the other as written. translate_query() never raises and returns the query unchanged for an unknown source.

Sources

from pyeuropepmc.features.search import registry

print(registry.available_sources())
print(registry.available_sources(installed_only=True))  # skip sources whose optional package is missing
print(sorted(registry.source_capabilities("europepmc")))  # ['date_filter', 'fulltext', 'get_paper', 'search']
Key Client Covers Credentials
europepmc EuropePMCLiteratureAdapter, wrapping SearchClient Europe PMC  
pubmed PubMedClient PubMed through NCBI E-utilities email
arxiv ArxivClient arXiv preprints; see arXiv client  
clinicaltrials ClinicalTrialsClient ClinicalTrials.gov; see ClinicalTrials.gov client  
semantic_scholar SemanticScholarLiteratureAdapter Semantic Scholar; needs pip install "pyeuropepmc[semanticscholar]" api_key
openalex OpenAlexLiteratureAdapter OpenAlex email
zenodo ZenodoClient Zenodo datasets, software and publications  
doaj DOAJClient Directory of Open Access Journals articles  
dblp DBLPClient DBLP computer-science bibliography  
hal HALClient HAL open archive  
core COREClient CORE open-access aggregator; needs a free CORE API key api_key

ORCID and NIH iCite are enrichment clients that look up known identifiers, not search sources; see ORCID and iCite clients.

Credentials

from pyeuropepmc.features.search import UnifiedSearch

with UnifiedSearch(
    sources=["europepmc", "pubmed", "semantic_scholar", "core"],
    credentials={"api_key": "YOUR_KEY", "email": "you@example.org"},
) as searcher:
    results, report = searcher.search("CRISPR", limit=10)

Rate limits

Client Own default rate_limit_delay (seconds)
PubMedClient 0.35
ClinicalTrialsClient 0.5
EuropePMCLiteratureAdapter, OpenAlexLiteratureAdapter, DOAJClient, HALClient, COREClient 1.0
SemanticScholarLiteratureAdapter 1.2
ArxivClient, ZenodoClient, DBLPClient 3.0

UnifiedSearch sets each client’s delay itself, in this order:

  1. rate_limits[source], when you name the source there.
  2. rate_limit_delay, when you pass it: the same delay for every source, as before.
  3. Otherwise 1.2 seconds (Semantic Scholar’s limit without a key), but never less than the provider asks for: arXiv, DBLP and Zenodo get 3 seconds. Up to 4.4.0 these three were paced at 1.2 seconds too, which is faster than arXiv’s published limit of one request every three seconds.
searcher = UnifiedSearch(
    sources=["europepmc", "pubmed", "arxiv", "openalex"],
    rate_limits={"pubmed": 0.4, "openalex": 0.2, "arxiv": 5.0},  # seconds between requests
    source_concurrency={"pubmed": 2},                          # calls to PubMed at the same time
)
print(searcher.limits)
# {'europepmc': {'rate_limit_delay': 1.2, 'max_concurrency': 1}, 'pubmed': {...}, ...}

Concurrency has two levels. max_workers is how many sources one search() queries at once (default: all of them). source_concurrency is how many calls one source gets at the same time from every thread that shares the UnifiedSearch object (default 1 for each source). A single search() makes one call per source, so the second level only matters when searches overlap, as they do in a server where several agents share one object; the later calls wait for the earlier ones instead of hitting the provider together. waited_seconds in the diagnostics shows how long a call waited. The delay is enforced by the source’s client: each HTTP request starts at least that long after the previous one of the same client, whichever thread made it (measured against arXiv with three overlapping searches: requests exactly 3.0 s apart). So the delay holds even with source_concurrency above 1; the cap only decides how many calls may be in flight and queue.

The effective limits are in every result. report.metadata["effective_limits"] maps each queried source to its rate_limit_delay and max_concurrency, so the limits that applied are recorded with the data; the diagnostics rows repeat them next to the error and Retry-After of a source that was rate limited.

Safe defaults for repeated and large searches. Keep source_concurrency at 1, keep the default delays or raise them (rate_limits={"arxiv": 5.0}), pass credentials for the sources that raise their limit with a key or an e-mail address (Semantic Scholar, NCBI, OpenAlex, Unpaywall), request fewer records per source rather than more, and look for rate_limited in the diagnostics before retrying. A server that is rate limited keeps its Retry-After in source_status; waiting that long is the polite response. UnifiedSearch does not re-ask a failed source on its own (see When a source fails); the clients retry only what a retry can fix: transport failures and HTTP 408/425/429/5xx, up to five tries with exponential backoff (1, 2, 4, 8 s) or, when the response carries a Retry-After, the wait the server asked for. A server that asks for more than 30 seconds is not asked again: the error carries the advice in context["retry_after"]. The Semantic Scholar adapter asks again up to four times on HTTP 429 the same way.

When a source fails

A source that fails never stops the others, but its records are missing from the merged result. With diagnostics=True the status of each source says what happened and what helps:

status Meaning What to do
ok Records came back  
empty The source answered with no records The query matched nothing there; try another source or a looser query
rate_limited HTTP 429 or a rate-limit message Wait retry_after seconds if it is given, then ask that source again, with a longer rate_limits[source] for repeated searches
error The request failed: a rejected key, a block page, an outage Read error["message"] and error["http_status"]; another source is the fallback
unavailable The client could not be created Install the optional package the message names

Three failures seen in live tests on 26 October 2026, and the same pattern for other providers:

A fallback order. Nothing in the library chooses a replacement for you, so decide the order for your question and add the next source when one failed (the sources that worked are asked again):

FALLBACK = {                       # who to ask when the first choice failed
    "semantic_scholar": "openalex",
    "dblp": "openalex",
}

results, report = searcher.search(query, sources=["europepmc", "pubmed", "semantic_scholar"],
                                  diagnostics=True)
status = report.metadata["source_status"]
failed = [n for n, row in status.items() if row["status"] in {"error", "rate_limited"}]
fallbacks = [FALLBACK[n] for n in failed if n in FALLBACK and FALLBACK[n] not in status]
if fallbacks:                     # one merged result again, with the fallbacks added
    kept = [n for n in status if n not in failed]
    results, report = searcher.search(query, sources=kept + fallbacks, diagnostics=True)

Suggested starting points, not measured rankings: for biomedical questions Europe PMC and PubMed first, then OpenAlex for the citation counts and works they miss, then Semantic Scholar when you have a key; for computer science arXiv and OpenAlex first, with DBLP and Semantic Scholar as additions that may be blocked. Over MCP the same applies to unified_search(diagnostics=True): its sources field has the status of each source.

Missing optional packages

If a selected source cannot be created, for example semantic_scholar without the semanticscholar package, UnifiedSearch logs a warning, skips the source and reports "client not initialised" in source_errors. registry.load_source(name, **kwargs) instead raises OptionalDependencyError (from pyeuropepmc._optional_imports, a subclass of ImportError) with the pip install command.

Register a source

from pyeuropepmc.features.search import registry

registry.register_source(
    registry.SourceSpec(
        name="my_repo",
        target="my_package.clients:MyRepoClient",  # imported on first use
        extras=("my_sdk",),                        # modules the client needs
        capabilities=frozenset({"search", "get_paper"}),
        credential_kwargs=("api_key",),
    )
)

SourceSpec(name, target, extras=(), pip_extra=None, capabilities=frozenset({"search"}), credential_kwargs=()). Registering an existing name raises ValueError unless you pass replace=True. registry.load_entry_point_sources(group="pyeuropepmc.sources") calls each zero-argument function published under that entry-point group; call it before creating UnifiedSearch.

The client class must accept rate_limit_delay, timeout or the credential names as keyword arguments (others are dropped) and return LiteratureResult objects from search(query, limit=...). Its records can carry the source’s own name, for example LiteratureResult(source="my_repo", ...); the deduplicator ranks a name it does not know below the built-in sources.

Using a source client directly

The clients in pyeuropepmc.features.search share search(query, limit=25, sort=None, **kwargs) -> list[LiteratureResult] and get_paper(identifier) -> LiteratureResult | None, and can be used as context managers. The Semantic Scholar and OpenAlex adapters have the same two methods but no context-manager support.

PubMed

from pyeuropepmc.features.search import PubMedClient

with PubMedClient(email="you@example.org") as client:
    papers = client.search("ME/CFS", limit=10, sort="date")
    paper = client.get_paper("32791984")
    batch = client.get_papers_batch(["32791984", "32882182"])

print(paper.title, paper.doi, len(batch))

PubMedClient(rate_limit_delay=0.35, timeout=15, cache_config=None, email=None, tool_name="pyeuropepmc"):

Semantic Scholar and OpenAlex

from pyeuropepmc.features.literature.adapters import (
    OpenAlexLiteratureAdapter,
    SemanticScholarLiteratureAdapter,
)

semantic_scholar = SemanticScholarLiteratureAdapter(api_key=None)
papers = semantic_scholar.search("machine learning", limit=10)

openalex = OpenAlexLiteratureAdapter()
works = openalex.search("CRISPR", limit=10, sort="citation_count")

print(len(papers), len(works))

SemanticScholarLiteratureAdapter(enrichment_client=None, api_key=None, rate_limit_delay=1.2, timeout=15) passes sort to Semantic Scholar as given. OpenAlexLiteratureAdapter(enrichment_client=None, rate_limit_delay=1.0, timeout=15, cache_config=None) accepts sort values "citation_count", "date" and "relevance". Both classes are also exported by pyeuropepmc.features.literature. Semantic Scholar answers HTTP 429 to most requests, with a key as well as without one, so a search is made in up to four attempts (the first plus three retries, each paced by the client’s delay, waiting as Retry-After says as seconds or as a date, else 1.5 s, 3 s, 6 s, never more than 10 s) before it is reported as rate_limited; the key is read from SEMANTIC_SCHOLAR_API_KEY when api_key is not passed, and a key the service does not accept is an HTTP 403 error, not an empty source.

Zenodo, DOAJ, DBLP, HAL and CORE

from pyeuropepmc.features.search import COREClient, DBLPClient, DOAJClient, HALClient, ZenodoClient

with ZenodoClient() as client:
    datasets = client.search_datasets("metabolomics", limit=10)

with DOAJClient() as client:
    articles = client.search("open access publishing", limit=10)

with DBLPClient() as client:
    cs_papers = client.search("knowledge graph", limit=10)

with HALClient() as client:
    hal_papers = client.search("neuroscience", limit=10)

with COREClient(api_key="YOUR_CORE_KEY") as client:
    core_papers = client.search("machine learning", limit=10)

for paper in datasets + articles + cs_papers + hal_papers + core_papers:
    print(paper.source, paper.title)

LiteratureResult

All clients return pyeuropepmc.models.LiteratureResult, a Pydantic model:

Field Type Description
doi str or None Lower-cased, without a https://doi.org/ prefix
pmid, pmcid str or None  
title str or None  
authors list[Author] or None Author(name, orcid=None, affiliation=None, institution=None, position=None)
publication_year int or None  
journal str or None  
abstract str or None  
citation_count int or None  
source str The source’s name, stripped and lower-cased. The built-in sources use europepmc, pubmed, semanticscholar, openalex, crossref, unpaywall, arxiv, clinicaltrials, zenodo, doaj, dblp, hal, core and icite; a registered source uses its own name. A value that is not a plain identifier (lower-case letters and digits joined by _, . or -) raises ValidationError
source_id str Identifier within the source
extra_metadata dict or None Source-specific values
semantic_scholar_data, openalex_data, pubmed_data dict or None Raw source records

Semantic Scholar records have source="semanticscholar", while the registry key is semantic_scholar. The model accepts additional fields. to_paper_entity() converts a result to a PaperEntity, and merge(other) returns a new result that fills this result’s empty fields from other.

Normalization utilities

The clients clean identifiers and text with functions from pyeuropepmc.features.literature:

from pyeuropepmc.features.literature import (
    is_valid_doi,
    normalize_abstract,
    normalize_author_name,
    normalize_doi,
    normalize_mesh_terms,
    normalize_paper_title,
    normalize_pmid,
)

print(normalize_doi("https://doi.org/10.1000/ABC.123"))              # 10.1000/abc.123
print(is_valid_doi("10.1000/xyz"), is_valid_doi("abc"))              # True False
print(normalize_author_name("John Smith"))                           # Smith, John
print(normalize_author_name("Smith JA"))                             # Smith, JA
print(normalize_paper_title("  CRISPR   screens.  "))                # CRISPR screens
print(normalize_abstract("BACKGROUND: Some   text."))                # Some text.
print(normalize_mesh_terms(["neoplasms", "Neoplasms", " Humans "]))  # ['neoplasms', 'Humans']
print(normalize_pmid("PMID:12345678"))                               # 12345678
Function Returns
normalize_doi(doi) Lower-cased DOI without resolver or doi: prefix; None if it is not a DOI
is_valid_doi(doi), is_valid_pmid(pmid) bool
normalize_pmid(pmid) The digits as a str, or None
normalize_author_name(name) "First Last" becomes "Last, First"; "Last, First" is kept; "Last Initials", the form PubMed and Europe PMC use, becomes "Last, Initials" ("Smith JA" → "Smith, JA"). An all-capitals name such as "WANG LI" is read as "First Last", because nothing marks the surname
normalize_author_list(authors) The list of author dicts with normalized names
normalize_paper_title(title) Unicode NFKC, collapsed whitespace, trailing punctuation removed
normalize_abstract(abstract, strip_headers=True) Section labels such as BACKGROUND: removed, whitespace collapsed
normalize_mesh_terms(terms) Trimmed terms with case-insensitive duplicates removed
normalize_journal_title(journal), normalize_affiliation(affiliation), normalize_to_nfkc(text) Unicode and whitespace cleanup

See also

This site is open source. Improve this page.