> ## Content Index
> Fetch the complete content index at: https://iteracompute.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Evaluate long deliveries separately from long inputs
- URL: https://iteracompute.com/blog/long-inputs-dependent-deliveries-acceptance/
- Published: 2026-11-03T03:31:57.000Z
- Updated: 2026-11-03T04:57:26.000Z
- Description: A four-part task plan and runnable synthetic ledger keep missing runs, retries, truncation and human handoffs in evaluation records.
- Author: IteraCompute
- Tags: LLM API, Evaluation, Application engineering

A report generator can accept a large source pack and still fail to produce the complete report. A coding assistant can locate a relevant function and still miss a dependent change. For developers buying model API capacity, those are different acceptance questions.

This article builds a task plan and a small runnable counting example. The plan is not an executed benchmark. The records are invented, and the Python only checks bookkeeping; it does not generate or judge model answers.

## Start with the research boundary

A [2 November author article](https://huggingface.co/blog/sirluk/sliding-window-beats-linear-attention) distinguishes long-input evaluation from tasks requiring sustained generation. Its quality comparison and its separate synthetic-weight speed setup have different scopes. The author's reported differences do not establish which hosted API will satisfy your application, nor do matched attention-state budgets establish equal whole-process memory.

Our practical response is to declare deliverables and counting rules before running candidates. This is an engineering proposal prompted by that distinction, not a reproduction of the author's experiments or a claim about the cause of their results.

Today's product research checked Flash/1017 before considering this broader topic. There was no new authorized product run to report. This article makes no Flash, Pro or IteraCompute performance, price or architecture comparison. Existing context limits, route names or quoted prices cannot substitute for application acceptance logs.

## Freeze four task types

Use input size and required delivery as separate planning dimensions. Define input bands with the exact tokenizer and full serialized request, including instructions, tools and other message overhead. If token counts are unavailable, mark the band unverified; character count is not a substitute. These illustrative task descriptions still need real authorized fixtures, counts and frozen evaluator artifacts before a benchmark can run.

| Planned type                                 | Illustrative task                                                   | Acceptance contract to freeze                                                                   |
| -------------------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Short input, short delivery                  | Read a small synthetic release note and return the required version | Version matches the source; no invented compatibility claim                                     |
| Long input, short delivery                   | Find a dated exception in an authorized source pack                 | Correct exception and source location; conflicting revisions handled under a predetermined rule |
| Short input, multiple dependent deliverables | Implement a specified change in a toy repository                    | Required files and regression checks satisfy the frozen specification                           |
| Long input, multiple dependent deliverables  | Reconcile a source pack into a report and change list               | Required sections, source facts and dependencies agree across artifacts                         |

The terms short and long need your task-specific thresholds. The terms delivery and multistep describe required work, not a minimum answer length. A concise implementation can be complete; a long explanation can omit a required file. Do not ask for unnecessary prose just to manufacture a long-output test.

For example, define the report contract as three sections plus a structured change list. Give each required fact a source identifier and explicit revision policy. The evaluator can then check whether the change list agrees with the report. This is a proposed fixture design, not a claim that those source documents or model outputs were generated during preparation.

The four cells can differ in difficulty. Comparing their acceptance rates does not isolate the causal effect of length. An architecture claim would require a more controlled experiment. Use the cells to discover where your own application evidence is missing.

## Freeze the run and its stopping rule

Before collecting outputs, save task and oracle hashes, an exact checkpoint or dated API route, harness revision, tokenizer revision, tool definitions, thinking and sampling controls, output cap, retry budget, deadline and human-handoff rule. Keep the same task IDs for every candidate; preserve any missing candidate-task pair.

Define at least two acceptance views. First-attempt acceptance asks whether the first attempt met the frozen criteria. Automatic acceptance within budget asks whether any permitted attempt met them without human completion. If a person finishes the task, report assisted completion separately. Do not relabel it automatic because the final artifact looks correct.

A retry remains attached to its original task and contributes another attempt, elapsed time and any known charge. It does not create a new successful task in the denominator. An absent run remains a planned task with missing evidence. A transport error, output-cap stop, empty response and evaluator rejection are different observations; preserve their reasons rather than silently deleting rows.

For this toy counting contract, an accepted attempt requires a recognized normal stop plus an evaluator's positive judgment. A normal stop alone never means correct. An output-cap stop is not accepted under this illustrative rule even if a partial artifact has useful pieces. Your application may define a different rule, but it must be stated before reviewing results. A reviewer should also distinguish a runtime failure from an intentionally skipped, predeclared exclusion.

## Inspect one invented denominator

Plan four tasks: lookup, source, patch and report. The authored records say lookup is accepted immediately; source is truncated once and accepted on its second attempt; patch has an unresolved attempt and is handed to a person; report has no record.

The first-attempt numerator is one with four planned tasks. The automatic-within-budget numerator is two with the same four planned tasks. There are four recorded attempts, one truncation, one handoff and one missing task. These numbers are fixtures for an accounting example, not measured model success rates. We do not compare vendors with them.

The example gives three attempts an invented all-in cost of USD 0.01 each and leaves the unresolved attempt's cost null. The known subtotal is USD 0.03\. The complete attempt-ledger cost stays null, as do any actual model or workflow costs. Missing tasks, empty attempt records and human work also keep that complete-cost field unknown in this conservative toy convention. This deliberately avoids equating missing evidence with free execution.

## Run the complete bookkeeping example

Save the following as demo.py and run python3 demo.py with Python 3.9 or newer. It uses only the standard library and makes no network requests. Acceptance flags are trusted, authored evaluator inputs; this is not a judge model or a validator of arbitrary untrusted logs. It uses one task row with nested attempts so a retry cannot silently become a new task.

```python
"""CC0 original bookkeeping example. Invented data, no model execution."""
from copy import deepcopy
from decimal import Decimal
import json

PLAN = {
    'lookup': 'short_input_short_delivery',
    'source': 'long_input_short_delivery',
    'patch': 'short_input_multistep_delivery',
    'report': 'long_input_multistep_delivery',
}

def summarize(plan, rows):
    """Rows are trusted evaluator records, not model-generated judgments.

    Exactly one row per planned task, with attempts nested inside it.
    An absent task remains in the denominator. Costs are synthetic USD.
    """
    by_task = {}
    for row in rows:
        task = row['task']
        if task not in plan or task in by_task:
            raise ValueError('unknown or duplicate task')
        if row['group'] != plan[task]:
            raise ValueError('group changed after plan')
        if type(row['human_handoff']) is not bool:
            raise ValueError('handoff must be a boolean')
        for a in row['attempts']:
            if type(a['accepted']) is not bool:
                raise ValueError('acceptance must be a boolean')
            if a['finish'] not in {'stop', 'length', 'error', 'unknown'}:
                raise ValueError('unsupported finish')
            if a['accepted'] and a['finish'] != 'stop':
                raise ValueError('inconsistent accepted attempt')
            if a['usd'] is not None:
                if not isinstance(a['usd'], str):
                    raise ValueError('use decimal string or null')
                cost = Decimal(a['usd'])
                if not cost.is_finite() or cost < 0:
                    raise ValueError('invalid cost')
        by_task[task] = row
    result = dict(planned=len(plan), first_attempt_accepted=0,
                  automatic_accepted_by_budget=0, missing_tasks=0,
                  empty_attempt_tasks=0, human_handoffs=0,
                  truncated_attempts=0, attempts=0,
                  unknown_cost_records=0)
    subtotal = Decimal('0')
    for task in plan:
        row = by_task.get(task)
        if row is None:
            result['missing_tasks'] += 1
            continue
        attempts = row['attempts']
        result['empty_attempt_tasks'] += not attempts
        result['human_handoffs'] += row['human_handoff']
        result['attempts'] += len(attempts)
        result['first_attempt_accepted'] += bool(attempts and attempts[0]['accepted'])
        result['automatic_accepted_by_budget'] += (
            not row['human_handoff'] and any(a['accepted'] for a in attempts))
        for a in attempts:
            result['truncated_attempts'] += a['finish'] == 'length'
            if a['usd'] is None:
                result['unknown_cost_records'] += 1
            else:
                subtotal += Decimal(a['usd'])
    result['known_cost_subtotal_usd'] = str(subtotal)
    incomplete = (result['missing_tasks'] or result['empty_attempt_tasks']
                  or result['unknown_cost_records'] or result['human_handoffs'])
    result['complete_attempt_ledger_cost_usd'] = None if incomplete else str(subtotal)
    return result

def attempt(accepted, finish='stop', usd='0.01'):
    return dict(accepted=accepted, finish=finish, usd=usd)

def fixture():
    # Four planned tasks; one missing row must not disappear.
    return [
        dict(task='lookup', group=PLAN['lookup'], human_handoff=False,
             attempts=[attempt(True)]),
        dict(task='source', group=PLAN['source'], human_handoff=False,
             attempts=[attempt(False, 'length'), attempt(True)]),
        dict(task='patch', group=PLAN['patch'], human_handoff=True,
             attempts=[attempt(False, 'unknown', None)]),
    ]

def run():
    rows = fixture()
    report = summarize(PLAN, rows)
    assert report['planned'] == 4
    assert report['first_attempt_accepted'] == 1
    assert report['automatic_accepted_by_budget'] == 2
    assert report['missing_tasks'] == 1
    assert report['attempts'] == 4
    assert report['human_handoffs'] == 1
    assert report['truncated_attempts'] == 1
    assert report['known_cost_subtotal_usd'] == '0.03'
    assert report['complete_attempt_ledger_cost_usd'] is None
    # A task row with no attempts is distinct from an absent task.
    empty = deepcopy(rows)
    empty[0]['attempts'] = []
    assert summarize(PLAN, empty)['empty_attempt_tasks'] == 1
    # Nine bad-input controls, explicitly outside production parser claims.
    bad = []
    bad.append(rows + [deepcopy(rows[0])])
    x = deepcopy(rows); x[0]['task'] = 'unplanned'; bad.append(x)
    x = deepcopy(rows); x[0]['group'] = 'posthoc_long'; bad.append(x)
    x = deepcopy(rows); x[0]['human_handoff'] = 'false'; bad.append(x)
    x = deepcopy(rows); x[0]['attempts'][0]['accepted'] = 'true'; bad.append(x)
    x = deepcopy(rows); x[0]['attempts'][0]['finish'] = 'length'; bad.append(x)
    x = deepcopy(rows); x[0]['attempts'][0]['usd'] = '-1'; bad.append(x)
    x = deepcopy(rows); x[0]['attempts'][0]['usd'] = 'NaN'; bad.append(x)
    x = deepcopy(rows); x[0]['attempts'][0]['usd'] = 0.01; bad.append(x)
    for x in bad:
        try:
            summarize(PLAN, x)
        except ValueError:
            pass
        else:
            raise AssertionError('bad input accepted')
    print(json.dumps(dict(kind='synthetic_bookkeeping_only', summary=report,
                          checks_passed=19, model_calls=0), indent=2))

if __name__ == '__main__':
    run()
```

## Read the output without turning it into a benchmark

The script prints the synthetic summary described above. Nineteen authored checks passed locally: nine summary expectations, one explicit empty-attempt case and nine rejection controls. Those controls reject a duplicated task, an unplanned task, a changed group, non-boolean handoff or acceptance, an accepted truncated attempt, a negative cost, a non-finite cost and a numeric float instead of a decimal string. They demonstrate behavior on these chosen inputs, not parser completeness or evaluation robustness.

The article's exact code block was extracted and run during preparation, and its output matched the saved demonstration. No API, SDK, model, tokenizer, GPU, code-generation tool or report-generation workflow was run. Input bands remain unmeasured. No actual quality, latency, memory, reliability or cost result follows from these counting checks.

The code intentionally does not implement the evaluator. A production harness needs observable artifact checks, an auditable correctness rubric, timestamps, real usage evidence, run configuration and an evidence store. It must also distinguish unknown billing from a confirmed zero charge. An evaluator's accepted flag must not simply echo the model saying it succeeded.

## Compare candidates with paired evidence

Assign each candidate the same frozen task identities and criteria. Retain first-attempt outcome, final automatic outcome within the fixed budget, attempts, termination reasons, time and human work. For each planned task, show whether A alone accepted, B alone accepted, both accepted or neither had an accepted result; identify missing evidence separately inside that view. Avoid computing a paired comparison only on the surviving successful rows.

Report each task group with its planned count and missing count. If you combine groups, declare weights in advance and keep the group results visible. A larger sample of easy lookups should not silently dominate a smaller set of expensive multistep tasks. Predeclare repeated runs and uncertainty analysis appropriate to your workload; four invented records provide no statistical estimate.

Use the actual endpoint's billable usage semantics. Record model attempts, tools and other fees under their own verified contracts, and retain known subtotals when a full bill is unavailable. Do not sum a reasoning-token subset twice or convert missing usage to zero. End-to-end time includes retries and required validation; decode speed is a different measure. Measure each on the same declared workload rather than borrowing throughput from a separate architecture experiment.

For self-hosted candidates, record weights, KV or recurrent state, runtime buffers and peak process/device memory with device and batch configuration. A matched state budget is only one component. For hosted APIs these internals may be unavailable; mark them unknown rather than inferring architecture or serving cost from a context-window limit. This article does not test those memory measurements.

## Decide what evidence would change procurement

Before purchasing a larger commitment, choose minimum acceptance and operational limits for the tasks your product actually sells. State whether a handoff is permitted, what it costs to the team and what a missing run does to the decision. A workflow with a short correct answer may meet the need more efficiently than one producing a longer trace; output length is an observed property, not quality by itself.

The product owner should approve required artifacts and source truth. The evaluator owner should approve acceptance and termination rules. The service owner should account for configuration, attempts, unknown charges and recovery. These are proposed responsibilities, not claims that a customer has adopted this process.

This plan is ready to be reviewed as a method. Running a real comparison still requires authorized nonsensitive fixtures, verified token bands, exact candidates, a frozen harness and real logs. Preserve those gaps instead of reporting a fictional winner.

For the research motivation and its stated experimental boundaries:

[Read the author's method and limitations](https://huggingface.co/blog/sirluk/sliding-window-beats-linear-attention).