The Production Loop

The Production Loop#

A served model goes stale. CoreRec closes the loop from serving back to training: log what users saw and clicked, read online metrics, A/B test a new model, retrain on fresh data and swap the model without a restart, and get alerted when traffic drifts.

corerec serve --feedback-log  ->  POST /feedback  ->  GET /metrics
        ^                                                 |
        |   POST /reload  <-  corerec retrain  <----------+

1. Log impressions and feedback#

corerec train events.csv -o artifacts/m
corerec serve artifacts/m --feedback-log feedback.jsonl

Every /recommend response now carries a request_id, and the list shown is logged as an impression. When the user acts on an item, send it back with that id so the click is credited to the list (and A/B variant) that showed it:

curl -X POST localhost:8000/recommend -H 'Content-Type: application/json' \
     -d '{"user_id": "u0042", "top_k": 5}'
# {"recommendations": ["i444", "i097", ...], "variant": "default", "request_id": "77e6...", ...}

curl -X POST localhost:8000/feedback -H 'Content-Type: application/json' \
     -d '{"user_id": "u0042", "item_id": "i097", "request_id": "77e6...", "event": "click"}'

event is free text; click and purchase are used for retraining. The log is one JSON object per line, easy to ship to a warehouse.

2. Online metrics#

curl localhost:8000/metrics

Per variant and overall:

Field

Meaning

ctr

clicks on shown items / items shown

requests_with_click

share of lists that got at least one click

mrr

mean of 1 / rank of the first clicked item per list (0 if none)

distinct_items_shown

catalogue coverage

fallback_share

share of requests answered with popular items (unknown users)

These are what the offline NDCG from corerec train is a proxy for. The same numbers are available offline with FeedbackLog("feedback.jsonl").metrics().

3. A/B test a new model#

corerec serve artifacts/m --challenger artifacts/m2 --challenger-share 0.1 \
              --feedback-log feedback.jsonl

Users are assigned by a stable hash of their id, so each one keeps seeing the same variant. /metrics then includes ab_test: both CTRs, the lift and a two-proportion z-test p-value. In Python: ModelServer({"control": m1, "treatment": m2}, traffic={"control": 0.9, "treatment": 0.1}, feedback_log="feedback.jsonl").

4. Retrain on a schedule#

corerec retrain artifacts/m --feedback feedback.jsonl

Retrain re-reads the training file (append new events to it, or pass --data newer.csv) and adds clicks and purchases from the feedback log. Rows newer than the model’s last training are new data, split in time:

  • the candidate trains on everything old plus the earlier half of the new rows

  • both the candidate and the current model are scored on the later half, which neither has seen, with the popular-item fallback the server uses

The candidate replaces the model only if its NDCG@10 is at least the current one’s (--tolerance relaxes that). It is then retrained on all rows; the old artifact stays in artifacts/m/previous/ for rollback. --dry-run compares without changing anything. A random holdout can’t do this job: the deployed model was trained on those rows, and its exclude-seen filter removes exactly the items being tested.

Pick up the new model without a restart:

curl -X POST localhost:8000/reload

Nightly with cron:

0 3 * * *  corerec retrain /srv/artifacts/m --feedback /srv/feedback.jsonl && curl -s -X POST localhost:8000/reload

Retrain needs a timestamp column in the data, to tell new rows from old.

5. Drift alerts#

/metrics also returns drift, comparing the last 1000 requests (?recent=N) with everything before them. It alerts, and logs a warning, when:

  • CTR fell by more than 30%

  • the fallback (unknown-user) share rose by more than 10 points

  • the distribution of clicked items moved (total variation distance > 0.2)

Thresholds are arguments of FeedbackLog.drift(). A rising fallback share usually means many new users the model hasn’t seen: retrain.

Contents