Model Serving

Production serving for recommendation models: an ANN-backed online recommendation engine for high-scale, always-fresh feeds, plus a REST/batch server.

Online serving (recommended for production)

OnlineRecommender serves top-K via ANN (sub-millisecond at million-item scale), supports incremental freshness, and falls back gracefully for unknown users. See the Online Serving guide for the full walkthrough.

from corerec.serving import OnlineRecommender

rec = OnlineRecommender.from_interactions(df, model="lightgcn", device="cuda")
rec.recommend(user_id=42, top_k=10)              # sub-ms ANN
rec.add_items(new_ids, new_vectors)              # freshness, no retrain
rec.fold_in_user("new_user", item_ids=[...])     # new user, no retrain

REST / batch server

from corerec.serving import ModelServer

server = ModelServer(model=my_model, host="0.0.0.0", port=8000)
server.start()

API Endpoints

• POST /predict — Single prediction

• POST /recommend — Single user recommendations

• POST /batch/predict — Batch predictions

• POST /batch/recommend — Batch recommendations

• GET /health — Health check

• GET /info — Model information

API Reference

CoreRec Model Serving Infrastructure

Production-ready model serving with REST API, batch inference, and monitoring.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.serving.ModelServer(model, host: str = '0.0.0.0', port: int = 8000, enable_docs: bool = True, metadata: Dict[str, Any] | None = None, fallback_items: List[Any] | None = None, feedback_log: Any = None, traffic: Dict[str, float] | None = None, reload_fn: Any = None)

Bases: object

Production-ready model serving infrastructure.

Provides REST API endpoints for: - Single predictions - Batch predictions - Recommendations - Batch recommendations - Health checks - Model metadata

Example

from corerec.serving import ModelServer from corerec.engines import DCN

model = DCN.load(‘artifacts/dcn’) server = ModelServer(model, host=”0.0.0.0”, port=8000) server.start() # Server starts at http://0.0.0.0:8000

# API Endpoints: # POST /predict - Single prediction # POST /recommend - Single recommendation # POST /batch/predict - Batch predictions # POST /batch/recommend - Batch recommendations # GET /health - Health check # GET /info - Model info

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(model, host: str = '0.0.0.0', port: int = 8000, enable_docs: bool = True, metadata: Dict[str, Any] | None = None, fallback_items: List[Any] | None = None, feedback_log: Any = None, traffic: Dict[str, float] | None = None, reload_fn: Any = None)

Initialize model server.

Parameters:

• model – Trained recommendation model with predict/recommend methods

• host – Server host address

• port – Server port

• enable_docs – Whether to enable API documentation

• metadata – Extra fields reported by GET /info under “artifact” (for example the training manifest)

• fallback_items – Items, best first, returned to users the model cannot answer (unknown users, empty results). Responses say which path answered in their “source” field.

• feedback_log – path or FeedbackLog. Enables impression logging, POST /feedback and GET /metrics.

• traffic – for A/B tests pass model as {“name”: model, …} and the share of users each gets, e.g. {“control”: 0.9, “treatment”: 0.1}. Users are assigned by a stable hash, so each sees one variant.

• reload_fn – zero-argument callable returning a fresh model; enables POST /reload (e.g. after corerec retrain replaced the artifact).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

start(reload: bool = False)

Start the server.

Parameters:

reload – Enable auto-reload (development mode)

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.serving.PredictionRequest(*, user_id: Any, item_id: Any, context: Dict[str, Any] | None = None)

Bases: BaseModel

Request schema for predictions.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

user_id: Any

item_id: Any

context: Dict[str, Any] | None

class corerec.serving.RecommendationRequest(*, user_id: Any, top_k: int = 10, exclude_items: List[Any] = [], context: Dict[str, Any] | None = None)

Bases: BaseModel

Request schema for recommendations.

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

user_id: Any

top_k: int

exclude_items: List[Any]

context: Dict[str, Any] | None

class corerec.serving.BatchInferenceEngine(model, batch_size: int = 256, num_workers: int = 4)

Bases: object

Optimized batch inference engine.

Provides efficient batch processing for high-throughput scenarios.

Example

from corerec.serving import BatchInferenceEngine

engine = BatchInferenceEngine(model, batch_size=1024)

# Batch predictions pairs = [(1,10), (1,11), (2,10), (2,11)] scores = engine.batch_predict(pairs)

# Batch recommendations user_ids = [1, 2, 3, 4, 5] recs = engine.batch_recommend(user_ids, top_k=10)

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(model, batch_size: int = 256, num_workers: int = 4)

Initialize batch inference engine.

Parameters:

• model – Trained recommendation model

• batch_size – Batch size for inference

• num_workers – Number of parallel workers

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

batch_predict(pairs: List[Tuple[Any, Any]]) → List[float]

Batch predictions for multiple user-item pairs.

Parameters:

pairs – List of (user_id, item_id) tuples

Returns:

List of predicted scores

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

batch_predict_parallel(pairs: List[Tuple[Any, Any]], num_processes: int = 4) → List[float]

Parallel batch predictions using multiprocessing.

Useful for CPU-bound models.

Parameters:

• pairs – List of (user_id, item_id) tuples

• num_processes – Number of processes

Returns:

List of predicted scores

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

batch_recommend(user_ids: List[Any], top_k: int = 10) → Dict[Any, List]

Batch recommendations for multiple users.

Parameters:

• user_ids – List of user IDs

• top_k – Number of recommendations per user

Returns:

Dictionary mapping user_id to recommendations

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.serving.ModelLoader(cache_dir: Path | None = None)

Bases: object

Production model loader with caching.

Loads models efficiently with caching to avoid repeated loading.

Example

loader = ModelLoader() model = loader.load(‘models/ncf_v1.pkl’)

# Subsequent loads use cache model2 = loader.load(‘models/ncf_v1.pkl’) # From cache!

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(cache_dir: Path | None = None)

Initialize model loader.

Parameters:

cache_dir – Directory for model cache

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

clear_cache()

Clear the model cache.

load(model_path: str, use_cache: bool = True, model_class: Any = None) → Any

Load a model saved with model.save(model_path).

Models don’t all write one file at model_path: torch models write <stem>.meta.json (+ weights/arrays) next to it, classic CF models pickle a state dict. Each format carries the class name, so this reads it and hands off to that class’s own load().

Parameters:

• model_path – the same path that was passed to save()

• use_cache – Whether to use cached model

• model_class – skip detection and call model_class.load(model_path)

Returns:

The reconstructed model

class corerec.serving.OnlineRecommender(dim: int, index_type: str = 'hnsw', metric: str = 'cosine')

Bases: object

ANN-backed, incrementally-updatable online recommender.

Parameters:

• dim – embedding dimension.

• index_type – ‘hnsw’ (default, best for serving), ‘ivf’, or ‘flat’ (exact).

• metric – ‘cosine’ (default) or ‘ip’.

add_items(item_ids: Sequence, item_emb: ndarray) → None

Add new items (e.g. fresh content) to the live index without retrain.

fold_in_user(user_id, item_ids: Iterable, update_seen: bool = True) → ndarray

Compute/refresh a user’s embedding from their interactions (ridge fold-in over item factors) – no retraining. Handles new users and keeps returning users fresh.

classmethod from_embeddings(item_ids: Sequence, item_emb: ndarray, user_ids: Sequence | None = None, user_emb: ndarray | None = None, popularity: Sequence | None = None, seen: Dict | None = None, index_type: str = 'hnsw', metric: str = 'cosine') → OnlineRecommender

classmethod from_interactions(interactions, user_col: str = 'user_id', item_col: str = 'item_id', model: str = 'lightgcn', dim: int = 64, epochs: int = 200, n_layers: int = 3, device: str = 'cpu', index_type: str = 'hnsw', metric: str | None = None, verbose: bool = False) → OnlineRecommender

Train native embeddings and build the serving index.

Parameters:

• model – ‘lightgcn’ (default, strongest on graph benchmarks) or ‘bpr’.

• metric – defaults to ‘ip’ (dot product) – the right ranking for these MF/GCN embeddings; pass ‘cosine’ to normalize.

classmethod from_model(model, index_type: str = 'hnsw', metric: str = 'ip')

Serve a trained ALS, Item2Vec, LightGCN or TwoTower model from an ANN index.

metric defaults to ‘ip’ because these models rank by dot product; ‘cosine’ would serve a different ranking than model.recommend().

recommend(user_id, top_k: int = 10, exclude_seen: bool = True) → List

recommend_batch(user_ids: Sequence, top_k: int = 10, exclude_seen: bool = True) → List[List]

stats() → Dict