Scientific Papers S3 Storage System — Technical Plan

Project: Scalable PDF repository with semantic search
Timeline: 4 weeks to production-ready prototype
Manager: Макс
Date: 2026-02-25

Executive Summary

Build a cost-efficient, deployable academic paper storage system combining S3 object storage (AWS or Yandex for Russia/EU compliance), PostgreSQL with vector embeddings, and semantic search. Prototype targets 100K–1M papers with sub-second search latency. Budget-conscious: AWS for proof-of-concept (~$200/mo for 10TB), MinIO for self-hosted scale (breakeven at 7 months vs AWS).

1. Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                     CLIENT LAYER                                 │
│              (Web UI / API / Search Interface)                   │
└────────────────────────┬────────────────────────────────────────┘
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ▼                ▼                ▼
    ┌─────────┐    ┌──────────┐    ┌──────────────┐
    │ S3 API  │    │Ingestion │    │ Search API   │
    │Gateway  │    │Service   │    │(Semantic)    │
    └────┬────┘    └────┬─────┘    └──────┬───────┘
         │              │                  │
         └──────────────┼──────────────────┘
                        │
        ┌───────────────┼───────────────┐
        │               │               │
        ▼               ▼               ▼
    ┌──────────┐  ┌──────────────┐ ┌────────────┐
    │S3 Storage│  │PostgreSQL +  │ │ Qdrant/    │
    │(PDFs)    │  │Metadata DB   │ │ Weaviate   │
    │~100K-1M  │  │(JSONB)       │ │ (Vectors)  │
    │objects   │  │              │ │ 768D       │
    └──────────┘  └──────────────┘ └────────────┘
         │              │               │
         └──────────────┼───────────────┘
                        │
         ┌──────────────┴──────────────┐
         │                             │
         ▼                             ▼
    ┌─────────────┐          ┌──────────────────┐
    │Embedding    │          │Metadata Extract  │
    │Service      │          │(PyMuPDF + LLM)   │
    │(sentence-   │          │                  │
    │transformers)│          │(Async queue)     │
    └─────────────┘          └──────────────────┘

Data Flow

1. Ingestion: PDF upload → S3 bucket → async extraction service
2. Metadata: Extract title, authors, DOI, abstract via PyMuPDF + Claude/GPT-4
3. Embedding: Generate 768-dim vectors (sentence-transformers) → Qdrant
4. Query: Full-text search (PostgreSQL) + semantic search (Qdrant) → hybrid ranking

2. Storage & Cost Analysis

For 100K Papers (avg 3MB each = 300GB)

For 1M Papers (avg 3MB = 3TB)

Recommendation: AWS S3 + Glacier for read-heavy, budget < $500/mo. MinIO after reaching 20TB+ with frequent access.

3. Metadata Schema (PostgreSQL)

Core Table: papers

CREATE TABLE papers (
  id UUID PRIMARY KEY,
  doi VARCHAR(255) UNIQUE,
  title TEXT NOT NULL,
  abstract TEXT,
  authors JSONB,  -- [{name, orcid, affiliation}, ...]
  publication_year INT,
  journal VARCHAR(255),
  conference VARCHAR(255),
  keywords TEXT[],
  s3_key VARCHAR(512) UNIQUE,  -- s3://bucket/papers/{doi}.pdf
  created_at TIMESTAMP,
  updated_at TIMESTAMP,

  -- Citation metrics
  citation_count INT DEFAULT 0,
  h_index_contrib INT,  -- for computing author h-index
  fwci FLOAT,  -- Field Weighted Citation Impact
  normalized_citations FLOAT,

  -- Content indices
  full_text_index TEXT,  -- for FTS
  embedding_id UUID,  -- link to Qdrant

  -- Metadata
  pdf_size_bytes BIGINT,
  page_count INT,
  language VARCHAR(10),
  openaccess BOOLEAN
);

CREATE INDEX idx_doi ON papers(doi);
CREATE INDEX idx_year ON papers(publication_year);
CREATE INDEX idx_keywords ON papers USING GIN(keywords);
CREATE INDEX idx_fwci ON papers(fwci DESC);

Essential Fields

• Identifiers: DOI, arXiv ID, PDF checksum
• Authors: JSONB array with name, ORCID, h-index, affiliation
• Citations: citation_count, h-index, FWCI (field-normalized impact)
• Content: abstract, keywords, full-text index for FTS
• Storage: S3 key + file metadata (size, pages, language)

4. Technology Stack Recommendation

Storage Layer

• Primary: AWS S3 Standard (proof-of-concept) → S3 Intelligent-Tiering (scale)
• Alternative: Yandex Object Storage (Russia/EU + no data residency concerns)
• Self-hosted scale: MinIO on commodity hardware (breakeven: 7 months @ 20TB+)

Metadata & Search

• Database: PostgreSQL 15+ with pgvector extension (768-dim vectors)
• Vector Search: Qdrant (self-hosted, open-source, 41 QPS @ 99% recall @ 50M vectors)
• Alternative: pgvector alone (cheaper, <100K vectors); Weaviate (hybrid search, graph context)
• Full-Text Search: PostgreSQL native (GIN indices on keywords, abstract)

Ingestion Pipeline

• PDF Processing: PyMuPDF (2-3x faster than PDFMiner)
• Metadata Extraction: Claude API (titles, abstracts) + regex/heuristics (DOI, authors)
• Embedding: sentence-transformers (all-MiniLM-L6-v2, 384-dim) or OpenAI (768-dim)
• Queue: Celery/Bull + Redis for async processing
• Orchestration: Python async + AWS Lambda (scale) or standalone workers (MVP)

API & Frontend

• Backend: FastAPI (Python) + asyncio
• Search: Hybrid (FTS → Qdrant semantic rerank)
• Frontend: React + TailwindCSS (optional for MVP — CLI/API first)

5. Implementation Timeline (4 Weeks)

Week 1: Foundation

• Days 1–2: Provision AWS S3 bucket + RDS PostgreSQL; set up local MinIO (test)
• Days 3–4: Design metadata schema; set up Qdrant (Docker)
• Day 5: Build FastAPI scaffold + S3 upload endpoint

Week 2: Ingestion

• Days 1–2: Implement PyMuPDF + metadata extraction service
• Days 3–4: Add embedding pipeline (sentence-transformers); integrate Qdrant ingestion
• Day 5: Build async ingestion queue (Celery/Redis)

Week 3: Search

• Days 1–2: Full-text search (PostgreSQL FTS + indexing)
• Days 3–4: Semantic search endpoint (Qdrant queries)
• Day 5: Hybrid search (combine rankings); add caching layer

Week 4: Polish & Deployment

• Days 1–2: Unit tests; 100K paper load test; cost profiling
• Days 3–4: Docker Compose for reproducible deployment; documentation
• Day 5: Demo + cost report; prepare for manager handoff

MVP Scope: Ingest 10K papers → search (FTS + semantic) → latency <500ms

6. Cost Estimates (Prototype Phase)

AWS Setup (100K papers, 6-month pilot)

MinIO Self-Hosted (Scale, year 1)

7. Deployment Strategy

Phase 1 (Week 1–2): Local Dev

docker-compose up -d  # PostgreSQL + Qdrant + MinIO
python scripts/ingest_test.py  # 100 papers

Phase 2 (Week 3–4): Cloud MVP

• AWS RDS (PostgreSQL 15)
• AWS S3 (single region, no replication)
• Qdrant on t3.large EC2
• 10K paper ingestion + search validation

Phase 3 (Post-Pilot, Week 5–8): Production Hardening

• Multi-region S3 replication
• RDS automated backups + 30-day retention
• Qdrant cluster (3+ replicas)
• Monitoring: DataDog/Prometheus
• CDN for metadata + cached results

8. Tech Comparison Matrix

For Manager: Start AWS (proof-of-concept), migrate to MinIO at >20TB if access patterns justify.

9. Risks & Mitigation

10. Success Criteria (Prototype)

• Ingest 100K papers in <4 hours
• Search latency <500ms (p99)
• Semantic search recall >85% (vs manual gold set)
• Monthly ops cost <$200
• Deployment reproducible via Docker Compose
• Zero downtime ingestion (async pipeline)

11. Next Steps (Manager Review)

1. Approve budget: $1,500 for 6-month AWS pilot
2. Assign team: 1 backend eng (Python) + 1 DevOps (infra)
3. Data source: Confirm paper sources (arXiv, CrossRef, institutional repos)
4. Success metric: Weekly cost tracking + performance dashboard

Sources & References

• AWS S3 Pricing 2026
• MinIO Cost Analysis
• Qdrant Performance Benchmarks
• pgvector vs Qdrant Comparison
• Vector Database 2026 Guide
• OpenMetadata S3 Integration
• Scientific Metadata Standards (ORCID)
• FWCI Citation Metrics