pyEuropePMC

External API Enrichment

The enrichment module provides integration with external academic APIs to enhance paper metadata with additional information from multiple sources.

Supported APIs

CrossRef

Unpaywall

Semantic Scholar

OpenAlex

Architecture

BaseEnrichmentClient

Base class providing common functionality:

Individual API Clients

Each API has a dedicated client that extends BaseEnrichmentClient:

Professional Library Integration

PyEuropePMC now uses danielnsilva/semanticscholar v0.12.0 professional library as the backend for Semantic Scholar API operations. This integration provides:

Feature Professional Library Old API Wrapper
Type Safety ✅ Full typed responses (Paper, Author, Venue) ⚠️ Dict-based
Async Support ✅ Native async/await ❌ Sync only
Bulk Search ✅ /paper/search/bulk endpoint ⚠️ paginated
Typed Objects ✅ Dataclass models ❌ Raw dicts
Recommended ✅ For new projects ⚠️ Legacy

When to Use Each Client

Use ProfessionalSemanticScholarClient (Recommended for new projects):

Use SemanticScholarClient (Legacy):

PaperEnricher Orchestrator

High-level interface that:

Usage

Basic Usage

from pyeuropepmc import PaperEnricher, EnrichmentConfig

# Configure enrichment
config = EnrichmentConfig(
    enable_crossref=True,
    enable_semantic_scholar=True,
    enable_openalex=True
)

# Enrich a paper
with PaperEnricher(config) as enricher:
    result = enricher.enrich_paper(doi="10.1371/journal.pone.0308090")

    # Access merged data
    print(result["merged"]["title"])
    print(result["merged"]["citation_count"])

Advanced Configuration

from pyeuropepmc import PaperEnricher, EnrichmentConfig
from pyeuropepmc.cache.cache import CacheConfig

# Configure with all options
config = EnrichmentConfig(
    # Enable/disable APIs
    enable_crossref=True,
    enable_unpaywall=True,
    enable_semantic_scholar=True,
    enable_openalex=True,

    # API-specific settings
    unpaywall_email="your@email.com",
    crossref_email="your@email.com",
    semantic_scholar_api_key="your-api-key",
    openalex_email="your@email.com",

    # Performance settings
    cache_config=CacheConfig(enabled=True, ttl=86400),
    rate_limit_delay=1.0
)

with PaperEnricher(config) as enricher:
    result = enricher.enrich_paper(doi="10.1234/example")

Using Individual Clients

from pyeuropepmc.enrichment import CrossRefClient, OpenAlexClient

# Use CrossRef directly
with CrossRefClient(email="your@email.com") as client:
    data = client.enrich(doi="10.1234/example")
    print(data["citation_count"])

# Use OpenAlex directly
with OpenAlexClient(email="your@email.com") as client:
    data = client.enrich(doi="10.1234/example")
    print(data["topics"])

Semantic Scholar Recommendations API

SemanticScholarClient also supports Semantic Scholar Recommendations API v1:

from pyeuropepmc.enrichment import SemanticScholarClient

with SemanticScholarClient(api_key="your-api-key") as client:
    # GET /recommendations/v1/papers/forpaper/{paper_id}
    recs_for_one = client.get_recommendations_for_paper(
        paper_id="649def34f8be52c8b66281af98ae884c09aef38b",
        limit=500,
    )

    # POST /recommendations/v1/papers/
    recs_for_many = client.get_recommendations_for_papers(
        positive_paper_ids=["649def34f8be52c8b66281af98ae884c09aef38b"],
        negative_paper_ids=["ArXiv:1805.02262"],
        limit=500,
    )

Using the Professional Library Directly

For advanced use cases, you can use ProfessionalSemanticScholarClient directly:

from pyeuropepmc.enrichment import ProfessionalSemanticScholarClient

# Configure with API key (optional but recommended for higher rate limits)
client = ProfessionalSemanticScholarClient(
    api_key="your-api-key",
    rate_limit_delay=1.0  # 1 second delay between requests
)

# Typed responses with Paper objects
paper = client.get_paper("10.1038/nature12373")
print(f"Title: {paper.title}")
print(f"Authors: {len(paper.authors)}")
print(f"Citations: {paper.citation_count}")

# Bulk search (faster, no relevance ranking, up to 10M results)
results = client.search_paper("cancer", bulk=True, limit=50)

# Regular search with relevance ranking
results = client.search_paper("machine learning", bulk=False, limit=100)

# Get author information
author = client.get_author("1724609")
print(f"Author: {author.name}")
print(f"Paper count: {author.paper_count}")

# Get venue information
venue = client.get_venue("12345")
print(f"Venue: {venue.name}")
print(f"Paper count: {venue.paper_count}")

Migrating from Old Library to Professional Library

Old Code:

from pyeuropepmc.enrichment import SemanticScholarClient

client = SemanticScholarClient()
results = client.search_paper("cancer")

New Code:

from pyeuropepmc.enrichment import ProfessionalSemanticScholarClient

client = ProfessionalSemanticScholarClient()
results = client.search_paper("cancer", bulk=False)  # explicit

Migration Benefits:

  1. ✅ Typed responses (Paper, Author, Venue) instead of dicts
  2. ✅ Bulk search endpoint for faster operations
  3. ✅ Better async support
  4. ✅ More consistent API

Backward Compatibility:

Data Merging Strategy

When multiple sources provide the same information, PaperEnricher uses the following priority:

Bibliographic Data

Metrics

Open Access

Additional Metadata

Error Handling

The enrichment module provides robust error handling:

Graceful Degradation

Retry Logic

Validation

Performance

Caching

Rate Limiting

Best Practices

  1. Enable caching for repeated queries
  2. Provide emails for polite pools (faster response)
  3. Use API keys where available (higher limits)
  4. Start with fewer APIs and add as needed
  5. Handle missing data gracefully

Authentication & API Keys

The simplest and most secure way to configure API keys is using environment variables:

# Linux/Mac (add to ~/.bashrc or ~/.zshrc)
export SEMANTIC_SCHOLAR_API_KEY="your-api-key-here"
export UNPAYWALL_EMAIL="your@email.com"
export CROSSREF_EMAIL="your@email.com"
export OPENALEX_EMAIL="your@email.com"

# Windows (PowerShell)
$env:SEMANTIC_SCHOLAR_API_KEY="your-api-key-here"
$env:UNPAYWALL_EMAIL="your@email.com"

Programmatic Configuration

from pyeuropepmc.enrichment import EnrichmentConfig, ProfessionalSemanticScholarClient

# Configure with all options
config = EnrichmentConfig(
    # Enable/disable APIs
    enable_crossref=True,
    enable_unpaywall=True,
    enable_semantic_scholar=True,
    enable_openalex=True,

    # API-specific settings
    unpaywall_email="your@email.com",
    crossref_email="your@email.com",
    semantic_scholar_api_key="your-api-key",
    openalex_email="your@email.com",

    # Performance settings
    rate_limit_delay=1.0
)

with ProfessionalSemanticScholarClient(
    api_key=config.semantic_scholar_api_key,
    rate_limit_delay=config.rate_limit_delay
) as client:
    paper = client.get_paper("10.1038/nature12373")

Authentication Priority

  1. Environment variables (highest priority)
  2. Programmatic configuration
  3. Default (no authentication) - Limited to free tier

Rate Limits

API Free Tier Limit With Authentication Recommended Delay
CrossRef No strict limit Polite pool 1.0-2.0s
Unpaywall 100,000/day Same 0.5-1.0s
Semantic Scholar 100 req/5 min 300 req/5 min 1.5-3.0s
OpenAlex 10 req/sec 100 req/day 0.2-0.5s

Rate Limiting Recommendations

Use Case Delay Notes
Hobby projects 2.0s Conservative, respectful usage
Research projects 1.0s Balanced for moderate workloads
Production with caching 0.5s Use caching to reduce requests
Bulk operations N/A Use bulk=True to minimize API calls

Best Practices

  1. Enable caching for repeated queries
  2. Use bulk=True for search operations (minimizes API calls)
  3. Provide emails for polite pools (faster response)
  4. Use API keys where available (higher limits)
  5. Monitor response headers for rate limit status
  6. Handle 429 (rate limit) responses with exponential backoff

Testing

The enrichment module includes comprehensive tests:

Run tests:

pytest tests/enrichment/

Examples

See examples/09-enrichment/ for:

Enrichment Demo Notebook

The enrichment_demo.ipynb notebook provides an interactive demonstration of enrichment capabilities with detailed data analysis:

Basic Enrichment Example

Advanced Enrichment Example

Detailed Data Analysis

The notebook includes comprehensive data exploration sections:

Available Fields per Service

Lists all top-level fields available from each service and shows data types for each field. This helps understand the complete data structure returned by each API.

Service Comparison Summary

A clean comparison table showing what each service provides:

Service Fields Title Authors Citations Abstract OA Topics License Funding
CROSSREF 16 ✅ ✅ ✅ ✅ ❌ ❌ ✅ None
SEMANTIC_SCHOLAR 13 ✅ ✅ ✅ ✅ ❌ ❌ ❌ ❌
OPENALEX 20 ✅ ✅ ✅ ❌ ✅ ✅ ❌ ❌

This comparison helps users understand the strengths of each API and plan their enrichment strategy accordingly.

API Documentation

For detailed API documentation, see:

Future Enhancements

Potential future additions:

This site is open source. Improve this page.