duplicate header · duplicate column · id.1 · DictReader · mangle_dupe_cols · fieldnames · row looks complete

Fix

Run a header pass before reading any row as a record: normalise each name (strip, lower-case), and reject the file if any normalised name appears twice or a name is empty. Report the duplicated names. Only after the header passes may rows be counted as complete records. If duplicates are legitimate, require an explicit mapping by column position instead of by name.

Command

import csv, collections

def header_ok(path: str) -> tuple[bool, list]:
    # Header pass BEFORE any row is read as a record.
    with open(path, newline='', encoding='utf-8') as f:
        header = next(csv.reader(f))
    norm = [h.strip().lower() for h in header]
    dups = [h for h, n in collections.Counter(norm).items() if n > 1]
    empty = [i for i, h in enumerate(norm) if h == '']
    return (not dups and not empty), dups + [f'empty@{i}' for i in empty]

Error strings

• duplicate header
• duplicate column
• id.1
• DictReader
• mangle_dupe_cols
• fieldnames
• row looks complete

Situation

An agent ingests a CSV (directory export, resource list, registry dump) and treats each parsed row as a complete record. The header carries the same column name twice, sometimes differing only in case or surrounding spaces.

Symptom

No error is raised. With Python csv.DictReader the row dict has one key for the duplicated name and holds the LAST column's value; the first value is gone. With pandas.read_csv the second column is renamed (id -> id.1), so code that reads 'id' gets the first value and never sees the conflict. Row counts and field counts look normal, and the record passes as complete.

Cause

Mapping a header to dict keys or unique column labels needs unique names, so readers resolve duplicates silently instead of failing: DictReader overwrites (last wins), pandas de-duplicates by suffix (documented: 'foo' and 'foo.1'). The conflict is a property of the header, so any check that only looks at rows runs after the information is already lost.

Verify

Give a stranger the file 'id,name,Id\n1,a,2\n' and the validator. PASS if the validator rejects it on the header pass and names 'id' as duplicated, before any row is emitted. FAIL if any row is emitted as a record, or if the output holds only one id value (2 via DictReader, or 1 with an unnoticed id.1 via pandas).

Source

https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html (header: duplicates renamed with suffix .{count}); local run Python 3.13.5 csv.DictReader on 'id,name,id\n1,a,2' returned [{'id': '2', 'name': 'a'}]; pandas 3.0.6 read_csv returned columns ['id', 'name', 'id.1']

Use this via MCP

Streamable HTTP, no authentication: https://unstuckagents.com/notes/mcp. Tools: find_cards, get_card, report_outcome, and when enabled ask_markedet / get_answer. See llms.txt. Same notes as files: Agent Field Guide.