Evaluation

Evaluation#

Comprehensive metrics and model evaluation for recommendation systems.

Quick Start: one-call `evaluate`#

corerec.evaluation.evaluate scores a fitted model on held-out interactions via the model’s own recommend path and returns the standard top-K ranking metrics:

from corerec.evaluation import evaluate

results = evaluate(
    model,
    test_interactions=test_df,        # DataFrame or (user, item[, rating]) tuples
    train_interactions=train_df,      # excluded from each user's recommendations
    k=10,
    relevance_threshold=4.0,          # rating >= 4 counts as relevant (optional)
)
# {'NDCG@10': ..., 'MAP@10': ..., 'MRR@10': ...,
#  'Precision@10': ..., 'Recall@10': ..., 'HitRate@10': ..., 'n_users': ...}

For finer control (cross-validation, custom metric sets) use the Evaluator class:

from corerec.evaluation import Evaluator

evaluator = Evaluator(metrics=["ndcg@10", "map@10", "recall@20"])
results = evaluator.evaluate(model, test_data)

API Reference#

CoreRec Evaluation Framework

Comprehensive metrics and evaluation tools for recommendation systems.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.RankingMetrics[source]#

Bases: object

Ranking metrics for recommendation evaluation.

Implements standard metrics like NDCG, MAP, MRR, Precision, Recall.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static hit_rate_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Hit Rate @ K (binary: 1 if any hit, 0 otherwise).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static map_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Mean Average Precision @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static mrr_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Mean Reciprocal Rank @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static ndcg_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Normalized Discounted Cumulative Gain @ K.

Parameters:

predictions – Predicted ranked list of items
ground_truth – Ground truth relevant items
k – Cut-off position

Returns:

NDCG @ K score

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static precision_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Precision @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static recall_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Recall @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.ClassificationMetrics[source]#

Bases: object

Classification metrics for binary/multi-class prediction.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static accuracy(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate accuracy.

static f1_score(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate F1 score.

static precision(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate precision.

static recall(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate recall.

class corerec.evaluation.DiversityMetrics[source]#

Bases: object

Diversity and novelty metrics for recommendations.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static coverage(all_recommendations: List[List], total_items: int) → float[source]#

Catalog coverage (percentage of items ever recommended).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static gini_coefficient(item_counts: Dict[Any, int]) → float[source]#

Gini coefficient (measures recommendation concentration).

0 = perfect equality, 1 = perfect inequality

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static intra_list_diversity(recommendations: List[List]) → float[source]#

Intra-list diversity (average pairwise distance within lists).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.Evaluator(metrics: List[str] = None)[source]#

Bases: object

Model evaluator for recommendation systems.

Evaluates models on test data using multiple metrics.

Example:

from corerec.evaluation import Evaluator

evaluator = Evaluator(metrics=['ndcg@10', 'map@10', 'recall@20'])

# Single model evaluation
results = evaluator.evaluate(model, test_data)

# Model comparison
comparison = evaluator.compare_models({
    'NCF': ncf_model,
    'DeepFM': deepfm_model
}, test_data)

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(metrics: List[str] = None)[source]#

Initialize evaluator.

Parameters:: metrics – List of metrics to compute (e.g., [‘ndcg@10’, ‘map@10’])

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

compare_models(models: Dict[str, Any], test_data: Dict[Any, List]) → Dict[str, Dict[str, float]][source]#

Compare multiple models on same test data.

Parameters:

models – Dict mapping model_name to model instance
test_data – Test data (user_id -> relevant items)

Returns:

Dict mapping model_name to evaluation results

Example:

results = evaluator.compare_models({
    'NCF': ncf_model,
    'DeepFM': deepfm_model
}, test_data)

# Results:
# {
#   'NCF': {'ndcg@10': 0.45, 'map@10': 0.38, ...},
#   'DeepFM': {'ndcg@10': 0.48, 'map@10': 0.41, ...}
# }

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

evaluate(model, test_data: Dict[Any, List], strict: bool = False) → Dict[str, float][source]#

Evaluate model on test data.

Parameters:

model – Recommendation model with recommend() method
test_data – Dict mapping user_id to list of relevant items
strict – re-raise the first per-user error instead of skipping the user

Returns:

Dictionary of metric_name -> score, plus n_users (users scored) and n_errors (users skipped because recommend() raised). A metric with no scored users is NaN, not 0.0.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

generate_report(results: Dict[str, Dict[str, float]]) → str[source]#

Generate human-readable evaluation report.

Parameters:: results – Evaluation results from compare_models()
Returns:: Formatted report string

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.CrossValidator(n_folds: int = 5, random_state: int = 42)[source]#

Bases: object

Cross-validation utilities.

Example:

cv = CrossValidator(n_folds=5)
out = cv.cross_validate(lambda: SAR(), df, metric='ndcg@10')
out['mean'], out['folds']

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(n_folds: int = 5, random_state: int = 42)[source]#

Initialize cross-validator.

Parameters:

n_folds – Number of folds
random_state – Random seed

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

cross_validate(model, data: Any, metric: str = 'ndcg@10', user_col: str = 'user_id', item_col: str = 'item_id', rating_col: str = 'rating') → Dict[str, Any][source]#

Fit a fresh model on each fold and score it on the held-out rows.

Parameters:

model – zero-arg factory returning an unfitted model, or an unfitted model instance (deep-copied per fold so folds don’t leak)
data – DataFrame of interactions
metric – any metric Evaluator understands, e.g. ‘ndcg@10’

Returns:

float, ‘std’: float, ‘folds’: [per-fold score]}

Return type:

{‘mean’

split(data: Any, n_folds: int | None = None) → List[tuple][source]#

Split data into folds.

Parameters:

data – Data to split
n_folds – Number of folds (uses self.n_folds if None)

Returns:

List of (train, test) tuples

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

corerec.evaluation.evaluate(model: Any, test_interactions: Iterable, train_interactions: Iterable | None = None, k: int = 10, relevance_threshold: float | None = None, user_col: str = 'user_id', item_col: str = 'item_id', rating_col: str = 'rating', user_subset: Iterable | None = None, verbose: bool = False) → Dict[str, float][source]#

Evaluate a fitted recommender on held-out interactions.

Parameters:

model – a fitted CoreRec model exposing recommend(user_id, top_k=...).
test_interactions – DataFrame or iterable of (user, item[, rating]).
train_interactions – optional; items here are excluded from each user’s recommendations (standard “exclude seen” protocol).
k – cutoff for the @K metrics.
relevance_threshold – if set, a test item counts as relevant only when its rating >= threshold; otherwise every test item is relevant.
user_subset – optional iterable of user ids to restrict evaluation to.

Returns:

dict with NDCG@k, MAP@k, MRR@k, Precision@k, Recall@k, HitRate@k and the number of users evaluated.

Metrics#

Evaluation Metrics

Comprehensive metrics for recommendation system evaluation.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.metrics.RankingMetrics[source]#

Ranking metrics for recommendation evaluation.

Implements standard metrics like NDCG, MAP, MRR, Precision, Recall.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static ndcg_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Normalized Discounted Cumulative Gain @ K.

Parameters:

predictions – Predicted ranked list of items
ground_truth – Ground truth relevant items
k – Cut-off position

Returns:

NDCG @ K score

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static map_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Mean Average Precision @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static mrr_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Mean Reciprocal Rank @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static precision_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Precision @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static recall_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Recall @ K.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static hit_rate_at_k(predictions: List, ground_truth: List, k: int = 10) → float[source]#

Hit Rate @ K (binary: 1 if any hit, 0 otherwise).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.metrics.ClassificationMetrics[source]#

Classification metrics for binary/multi-class prediction.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static accuracy(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate accuracy.

static precision(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate precision.

static recall(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate recall.

static f1_score(predictions: ndarray, ground_truth: ndarray) → float[source]#: Calculate F1 score.

class corerec.evaluation.metrics.DiversityMetrics[source]#

Diversity and novelty metrics for recommendations.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static intra_list_diversity(recommendations: List[List]) → float[source]#

Intra-list diversity (average pairwise distance within lists).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static coverage(all_recommendations: List[List], total_items: int) → float[source]#

Catalog coverage (percentage of items ever recommended).

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

static gini_coefficient(item_counts: Dict[Any, int]) → float[source]#

Gini coefficient (measures recommendation concentration).

0 = perfect equality, 1 = perfect inequality

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

Evaluator#

Model Evaluator

Tools for evaluating and comparing recommendation models.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.evaluator.Evaluator(metrics: List[str] = None)[source]#

Bases: object

Model evaluator for recommendation systems.

Evaluates models on test data using multiple metrics.

Example:

from corerec.evaluation import Evaluator

evaluator = Evaluator(metrics=['ndcg@10', 'map@10', 'recall@20'])

# Single model evaluation
results = evaluator.evaluate(model, test_data)

# Model comparison
comparison = evaluator.compare_models({
    'NCF': ncf_model,
    'DeepFM': deepfm_model
}, test_data)

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(metrics: List[str] = None)[source]#

Initialize evaluator.

Parameters:: metrics – List of metrics to compute (e.g., [‘ndcg@10’, ‘map@10’])

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

evaluate(model, test_data: Dict[Any, List], strict: bool = False) → Dict[str, float][source]#

Evaluate model on test data.

Parameters:

model – Recommendation model with recommend() method
test_data – Dict mapping user_id to list of relevant items
strict – re-raise the first per-user error instead of skipping the user

Returns:

Dictionary of metric_name -> score, plus n_users (users scored) and n_errors (users skipped because recommend() raised). A metric with no scored users is NaN, not 0.0.

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

compare_models(models: Dict[str, Any], test_data: Dict[Any, List]) → Dict[str, Dict[str, float]][source]#

Compare multiple models on same test data.

Parameters:

models – Dict mapping model_name to model instance
test_data – Test data (user_id -> relevant items)

Returns:

Dict mapping model_name to evaluation results

Example:

results = evaluator.compare_models({
    'NCF': ncf_model,
    'DeepFM': deepfm_model
}, test_data)

# Results:
# {
#   'NCF': {'ndcg@10': 0.45, 'map@10': 0.38, ...},
#   'DeepFM': {'ndcg@10': 0.48, 'map@10': 0.41, ...}
# }

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

generate_report(results: Dict[str, Dict[str, float]]) → str[source]#

Generate human-readable evaluation report.

Parameters:: results – Evaluation results from compare_models()
Returns:: Formatted report string

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

class corerec.evaluation.evaluator.CrossValidator(n_folds: int = 5, random_state: int = 42)[source]#

Bases: object

Cross-validation utilities.

Example:

cv = CrossValidator(n_folds=5)
out = cv.cross_validate(lambda: SAR(), df, metric='ndcg@10')
out['mean'], out['folds']

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

__init__(n_folds: int = 5, random_state: int = 42)[source]#

Initialize cross-validator.

Parameters:

n_folds – Number of folds
random_state – Random seed

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

split(data: Any, n_folds: int | None = None) → List[tuple][source]#

Split data into folds.

Parameters:

data – Data to split
n_folds – Number of folds (uses self.n_folds if None)

Returns:

List of (train, test) tuples

Author: Vishesh Yadav (mail: sciencely98@gmail.com)

cross_validate(model, data: Any, metric: str = 'ndcg@10', user_col: str = 'user_id', item_col: str = 'item_id', rating_col: str = 'rating') → Dict[str, Any][source]#

Fit a fresh model on each fold and score it on the held-out rows.

Parameters:

model – zero-arg factory returning an unfitted model, or an unfitted model instance (deep-copied per fold so folds don’t leak)
data – DataFrame of interactions
metric – any metric Evaluator understands, e.g. ‘ndcg@10’

Returns:

float, ‘std’: float, ‘folds’: [per-fold score]}

Return type:

{‘mean’

Evaluation

Contents

Evaluation#

Quick Start: one-call evaluate#

API Reference#

Metrics#

Evaluator#

Quick Start: one-call `evaluate`#