Collect and monitor
Create a run from the study plan. As owner, open Execution → Continuous execution → Start / resume. Select the per-batch request budget and text-download parallelism. Closing the browser does not stop processing: the controller schedules bounded worker batches and PostgreSQL preserves progress.
Execution cycle
Collection → normalization/discovery → review report → metrics → group analysis → collection.
Periodic analysis exposes partial results before collection finishes. The group stage advances bounded history/charter collection and synchronizes organizational hypotheses plus a local saved-authorship pass. After final document collection, remaining eligible group work is processed; blocked evidence remains explicit. No stage performs scientific reviews automatically. Independent author intersections are a separate owner-triggered action; they are not implied by the automatic base-driven authorship pass in the group stage.
Pause execution disables continuation; in-progress work reaches a safe checkpoint. Resume uses saved progress. Single stages and terminal continue_run are alternatives for paused/idle runs; do not run competing controllers on the same run.
Sources and coverage
| Input | Used for | Limits |
|---|---|---|
| Datatracker documents, histories and relationships | Identities, affiliations, continuity and references | Current API metadata is not a complete historical snapshot |
| RFC index and preserved texts | Publication metadata, references and technical passages | A collected text still requires review of actual use |
| Group metadata/history and eligible charter revisions | RG/WG screening and organizational clues | Charter mentions enrich the screening universe, not all possible groups/processes |
| Posted draft submissions | Dated author-address observations | No verified person identity or automatic RFC-to-draft authorship inheritance |
BOF/meeting minutes, slides, mailing lists, complete participation histories and aliases are not comprehensively collected/analyzed by this discovery workflow. Do not infer their examination from a nonzero organizational or mediation queue. Source gaps are scientific limitations, not merely pending software tasks.
Inspect progress and saved data
- Execution: current stage, processed requests, retry time, events and completion/blocked state.
- Performance: stage/batch durations and checkpoint/indexing indicators; distinguish downloads, database persistence and analysis before tuning.
- Imported data: run-scoped documents, groups, observations, relationships and files. Use filters and batch selection; original source bytes and SHA-256 remain preserved.
- Results: candidate queues and input dates. Discovery and source collection can be at different points; zero means only zero within that analyzed scope.
Organizational screening is under Results → Organizational → Evidence between groups. Its acquisition statuses are not scientific review states. Candidate inspection uses the associated immutable input snapshot; the latest screening report may be newer.
Saved-data discovery actions do not initiate new downloads. Reanalysis can use newly saved material and retains existing IDs, reviews and investigator notes. A candidate absent from a later snapshot can remain in the run; snapshot counts and run-wide counts therefore differ.
Refresh discovery on an existing run
Deploying compatible code does not add signals to saved candidates by itself. While the run is idle, use the saved-data discovery actions to apply the new rules. The cutoff, group inventory and reviewer assignments remain frozen; source collection and derived discovery are separate operations.
For possible agenda adoption, rerun organizational discovery. To add explicit charter publication/approval events for a saved group, collect those sources first:
docker compose exec -T web python web/manage.py collect_charter_chronology --run RUN_UUID --group dtn --budget 8
docker compose exec -T web python web/manage.py discover_organizational --run RUN_UUID
collect_charter_chronology makes bounded requests and preserves the original event-response bytes. Its --group argument accepts saved group acronyms or IDs and can be repeated. Resume with the same arguments if its request budget is reached; cached pages are reused. The organizational discovery command uses saved evidence only.
For independent author intersections, select Discover independent author intersections in Results or run:
docker compose exec -T web python web/manage.py discover_author_intersections --run RUN_UUID --pages 8000
Repeat while state is indexing or more eligible saved pages remain. After indexing completes, the action verifies saved analytical inputs and projects exploratory candidates. Pages processed, source exclusions, document pairs and address-based leads have different units; inspect the returned status and Results guide before interpreting counts. The command does not download sources or create a base event.
The original discover_human_mediation --run RUN_UUID --pages 8000 command remains available for the documentary-base route. It shares the authorship index but has a separate discovery status and scope. Running one route does not mean the other has searched its universe.
Failures and recovery
Inspect the last Execution event before resuming. Transient failures retry with backoff. Changed pagination totals trigger a bounded revalidation; only an identical preserved prefix can be extended. Unstable or changed prefixes and integrity failures remain errors rather than being silently accepted. Fix the reported cause before retrying.
For manual pagination repair, pause the run and use the task key from the failed event:
docker compose run --rm --no-deps web python web/manage.py recover_pagination --run RUN_UUID --task TASK_KEY
Successful manual repair leaves the run paused. Resume explicitly. See maintenance for logs and service checks.
Historical draft revision texts and RFC texts returning HTTP 404 or 410 are recorded as unavailable, with the URL, timestamp and HTTP error retained in the checkpoint and request log. Collection continues without creating a text artifact or counting that text as downloaded. The progress panel and coverage report expose these gaps; other request failures keep their existing stop/retry behavior.
Manual pagination recovery can use recover_pagination --run <uuid> --task <key> --reconcile-group-metadata while continuous execution is disabled. This accepts only state and time changes in the saved working-group prefix, after full consistent pagination and exact verification of all other fields and object order. Original source bytes and values remain authoritative; refreshed responses and field differences are preserved in the recovery audit. Other differences still block recovery. Resume continuous execution after inspecting the audit.
Pagination recovery stages at most 64 pages per batch in database artifacts and resumes from the saved next URL. Reaching the batch limit is progress, not a failed attempt. Staged pages become collection sources only after the complete snapshot passes consistency and saved-prefix checks. Automatic retries start a new snapshot after a transient or changed-total failure; all staged responses remain preserved. Manual recovery returns pending: true when another invocation is needed.