Scientific Papers Storage — Implementation Checklist

Week 1: Foundation Setup

Infrastructure

• Create AWS account (or use existing); enable S3 + RDS
• Provision S3 bucket (versioning + MFA delete enabled)
• Launch RDS PostgreSQL t3.medium (Multi-AZ disabled for MVP)
• Generate AWS credentials (IAM user with S3 + RDS permissions)
• Set up VPC security groups (RDS: allow EC2; S3: public read on metadata)

Local Development Environment

• Docker Compose stack: PostgreSQL 15 + Qdrant + MinIO (optional)
• Python 3.11+ environment; uv/pip with requirements:
  fastapi>=0.104 sqlalchemy>=2.0 psycopg[binary]>=3.1 pymupdf>=1.23 sentence-transformers>=2.2 qdrant-client>=2.7 boto3>=1.26 redis>=5.0 celery>=5.3
• Git repo initialized; .env for secrets (AWS keys, DB URLs)

Database & Search Setup

• Create PostgreSQL schema (papers table + indices)
• Install pgvector extension (or leave for later)
• Spin up Qdrant: single node (test mode)
• Verify connectivity from dev machine

Week 2: Ingestion Pipeline

PDF Processing

• Build ingest/extractor.py:
• Input: PDF file
• Output: title, abstract, authors, DOI, page_count, full_text
• Use PyMuPDF for parsing; fallback: Tesseract for scanned pages
• Test on 50 diverse papers (published, preprint, scanned)

Metadata Extraction

• Implement DOI extractor (regex + CrossRef API fallback)
• Author parser: extract name, ORCID (regex), affiliation
• Mock LLM extraction (use heuristics for MVP)
• Store in PostgreSQL papers table

Embedding Generation

• Download sentence-transformers model (all-MiniLM-L6-v2, 384-dim)
• Build embed/generator.py: abstract → 384-dim vector
• Upsert vectors to Qdrant collection (papers_embeddings)

Queue & Async Processing

• Set up Redis (local Docker or AWS ElastiCache)
• Build Celery tasks for ingestion (pdf → extract → embed → store)
• Implement retry logic (3 attempts with exponential backoff)
• Add monitoring: task counters + error logs to CloudWatch

Batch Ingestion

• Script: scripts/ingest_batch.py (read from folder or S3 prefix)
• Test: ingest 100 papers, verify quality
• Performance target: 500–1000 papers/hour

Week 3: Search & API

Full-Text Search

• Create PostgreSQL FTS index on title + abstract + keywords
• Endpoint: GET /search/full-text?q=quantum&limit=20
• Implement ranking (TF-IDF or BM25)

Semantic Search

• Endpoint: GET /search/semantic?q=neural+networks&limit=20
• Query → embed (same model) → Qdrant similarity search
• Return: (paper_id, score, title, abstract, relevance_snippet)

Hybrid Search

• Endpoint: GET /search?q=..&mode=hybrid
• Combine FTS (weight: 0.3) + semantic (weight: 0.7)
• Re-rank by FWCI (if available)

API Endpoints (FastAPI)

POST /papers/upload          # Upload PDF
GET  /papers/{id}            # Fetch metadata + S3 download link
GET  /search/full-text       # Full-text search
GET  /search/semantic        # Semantic search
GET  /search                 # Hybrid search
GET  /papers/by-author/{name}# Author search
GET  /stats                  # Ingestion/search stats

Caching

• Redis caching for search results (TTL: 1 hour)
• Query deduplication (popular searches)

Week 4: Testing & Deployment

Testing

• Load test: 10K paper dataset, 100 concurrent searches
• Latency profiling (FTS vs semantic vs hybrid)
• Metadata extraction validation (spot-check 100 papers)
• Vector quality (manual relevance assessment on 10 queries)

Docker & Deployment

• Create docker-compose.prod.yml (PostgreSQL + Qdrant + API)
• Build Dockerfile for FastAPI service
• Push to ECR or Docker Hub
• Deploy to AWS ECS or EC2 + systemd

Monitoring & Logging

• CloudWatch logs for API + ingestion tasks
• Prometheus metrics: request rate, latency, error rate
• Dashboard: daily ingestion count, search volume, cost tracking

Documentation

• API docs (auto-generated by FastAPI at /docs)
• Data schema diagram
• Deployment guide (AWS + Docker)
• Cost breakdown (S3 + RDS + Qdrant)

Post-Launch (Week 5–8)

Data Quality

• Implement feedback loop: mark false positives/negatives
• Periodic metadata audit (sample 100 papers)
• Cross-reference with arXiv/CrossRef to fill gaps

Scale Optimization

• Shard Qdrant by discipline (if vectors exceed 50M)
• Implement S3 object lifecycle (move to Glacier after 1 year)
• Auto-scaling for ingestion workers

Production Hardening

• RDS automated backups (30-day retention)
• S3 cross-region replication (optional, for DR)
• API rate limiting (100 req/min per user)
• Authentication (API keys for external users)

Quick Cost Reference (6 months, 100K papers)

Key Files to Create

papers-storage/
├── docker-compose.yml         # Dev environment
├── docker-compose.prod.yml    # Production
├── requirements.txt           # Python deps
├── .env.example              # Config template
├── ingest/
│   ├── extractor.py          # PDF → metadata
│   ├── embedder.py           # Text → vectors
│   └── schema.py             # Data classes
├── api/
│   ├── main.py               # FastAPI app
│   ├── routes.py             # Endpoints
│   └── models.py             # DB models
├── db/
│   ├── init.sql              # Schema
│   └── migrations/           # Alembic (later)
├── scripts/
│   ├── ingest_batch.py       # Batch import
│   └── health_check.py       # Monitoring
├── tests/
│   ├── test_extraction.py
│   └── test_search.py
└── docs/
    ├── API.md
    ├── DEPLOYMENT.md
    └── ARCHITECTURE.md

Decision Points (For Manager)

1. Paper Source: arXiv dump? CrossRef API? Institutional repo?
2. Metadata Enrichment: Use paid APIs (CrossRef, Semantic Scholar) or heuristics-only?
3. OCR for Scanned Papers: Yes (+$0.02/page) or skip initially?
4. Multi-language Support: English-only or support Chinese/Russian abstracts?
5. Public vs Private: Restrict access or open search API?

Success Metrics (Checkpoint at Week 4)

• Ingestion: 10K papers loaded, <4 hrs
• Search Performance: p99 latency <500ms
• Cost: <$200/mo for 100K papers
• Quality: Metadata extraction >90% accuracy (spot check)
• Deployment: Docker Compose reproduces stack locally