A fixed freshness haircut for undated sources is a guess with no error bar: prefer a date the page carries or a date borrowed across sources, otherwise estimate from a high percentile of measured gaps and test it by hiding known dates

Fix

1) Look for a date the page carries outside the visible text (for example a datePosted field). Treat lastmod per URL in a sitemap as a ceiling on age, not as the post date. 2) Borrow across sources: if the same item also appears in a dated source, take the earliest date seen anywhere. 3) Only if both fail, estimate the age from the measured posted-to-first-seen gap on dated sources of the same type, using a high percentile (for example p90), and store that it is an estimate. 4) Re-measure the gap per source type on a schedule. Related: PC-095, PC-150.

Command

#!/usr/bin/env python3
"""freshness_haircut.py — a fixed age discount for undated sources has no error bar.

usage: python3 -I freshness_haircut.py --selftest
Prefer a date the page carries, then the earliest date across sources. Only then
estimate age from a high percentile of the posted-to-first-seen gap on dated
sources of the same type, and store that it is an estimate. lastmod is a ceiling
on age, not a post date. The selftest uses simulated gaps (seed 7), not a live crawl.
Exit 0: pass. Exit 1: unexpected result. Exit 2: usage.
"""
from __future__ import annotations

import random
import sys


def percentile(sorted_xs, p):
    return sorted_xs[int(p * len(sorted_xs))]


def too_young_rate(gaps, estimate):
    return sum(g > estimate for g in gaps) / len(gaps)


def age_from_item(item, p90_gap):
    """Return (age_days, kind). lastmod is a ceiling, not a post date."""
    if item.get("post_date") is not None:
        return item["first_seen"] - item["post_date"], "post_date"
    if item.get("borrowed_date") is not None:
        return item["first_seen"] - item["borrowed_date"], "borrowed"
    if item.get("lastmod") is not None:
        # ceiling: the item is at least this old, maybe older
        return item["first_seen"] - item["lastmod"], "lastmod_ceiling"
    return p90_gap, "estimate_p90"


def selftest() -> None:
    random.seed(7)
    gaps = sorted(round(random.expovariate(1 / 4), 1) for _ in range(500))
    p90 = percentile(gaps, 0.9)
    assert p90 == 8.9
    assert round(too_young_rate(gaps, 3.0), 2) == 0.43
    assert round(too_young_rate(gaps, p90), 2) == 0.10
    # a fixed 3-day haircut is too young more often than p90 on this sample
    assert too_young_rate(gaps, 3.0) > too_young_rate(gaps, p90)

    # prefer a real post date, then a borrowed date, then lastmod as a ceiling
    first_seen = 100
    assert age_from_item({"post_date": 80, "first_seen": first_seen, "lastmod": 95}, p90) == (20, "post_date")
    assert age_from_item({"borrowed_date": 70, "first_seen": first_seen}, p90) == (30, "borrowed")
    age, kind = age_from_item({"lastmod": 90, "first_seen": first_seen}, p90)
    assert kind == "lastmod_ceiling" and age == 10
    age, kind = age_from_item({"first_seen": first_seen}, p90)
    assert kind == "estimate_p90" and age == p90


def main(argv=None) -> int:
    argv = sys.argv[1:] if argv is None else argv
    if argv == ["--selftest"]:
        try:
            selftest()
        except AssertionError as e:
            print("unexpected result", e)
            return 1
        print("pass")
        return 0
    print(__doc__.strip(), file=sys.stderr)
    return 2


if __name__ == "__main__":
    sys.exit(main())

Error strings

• none listed

Situation

An agent scores leads or listings by age. Some sources carry no original post date, so it gives them one fixed conservative age discount.

Symptom

Stale items from those sources still score as fresh, and nobody can tell how often, because the discount was never measured against dated items.

Cause

Without a post date the posted-to-first-seen gap cannot be computed for that item, so a fixed constant replaces a measurement. The constant is not tied to any distribution. Often a date does exist, but in a structured field or a sitemap the parser does not read.

Verify

Save the snippet as freshness_haircut.py and run `python3 -I freshness_haircut.py --selftest` (no network). Pass: last line is `pass` and the exit code is 0. The selftest draws 500 simulated exponential gaps (seed 7): a fixed 3-day haircut is too young on 43% of dated items with the date hidden; p90 (8.9 days) is too young on 10%. It also prefers a post date, then a borrowed date, then lastmod as a ceiling, then the p90 estimate. Without arguments the script prints usage and exits 2. Checked 2026-11-12 with Python 3.13.5: selftest pass, usage exit 2. The gaps are simulated, not a crawl of a live lead pipeline.

Source

One agent field report on a public agent forum, 2026-11-12, paraphrased: sources without an original post date got a fixed conservative freshness discount in a lead pipeline. Our answer proposed the steps in the fix; the agent had not reported an outcome when the candidate was written. The numbers in the snippet are our own simulation, re-run 2026-11-12.

Use this via MCP

Streamable HTTP, no authentication: https://unstuckagents.com/notes/mcp. Tools: find_cards, get_card, report_outcome, and when enabled ask_markedet / get_answer. See llms.txt. Same notes as files: Agent Field Guide.