Prepare a study plan
In the web interface, create a study and choose Create study plan. The form validates the same requirements as the engine. Every saved study plan has its own version; starting a run freezes that version. To change choices, create the next version.
| Field | What to enter |
|---|---|
| Name | An identifiable study-plan name |
| Question | The concrete documentary research question |
| Cutoff | Last instant of a month in UTC, e.g. 2026-09-30T23:59:59Z |
| Publication | Plan for publishing results, evidence and limitations |
| Inventory | CSV rows of active/concluded RG acronyms and numeric Datatracker IDs |
| Inventory verified | Your explicit confirmation of the universe and documented coverage |
| Review team | At least two active accounts; each candidate receives an independent pair |
| Eligible adjudicators | Accounts eligible to resolve disagreement; every pair needs a third person outside that pair |
| Coverage audit | How you check completeness, missing history and retrieval failures |
| Independent review | Universe/sample design, selection, separation and responsibilities |
| Adjudication procedure | How disagreements are resolved and preserved |
| Agreement criterion | What agreement is measured before adjudication and how it is interpreted |
The documentary profile uses all recoverable history up to the cutoff, drafts/RFCs in all states and recoverable IETF documentary destinations. Automated sources are Datatracker and RFC Editor; analysis is documentary summary. These are the supported engine choices, not arbitrary unimplemented options.
Inventory example structure (replace the example with verified real entries):
irtf_acronym,datatracker_acronym,datatracker_id
A blank inventory is intentionally not executable. Names alone are discovery inputs and cannot establish eligibility. New web plans require a complete review team; legacy plans with missing reviewers remain available without claiming independent validation.
From plan to execution
A study is the research project; its plan is a set of choices; each run preserves those choices and the planned team. Operational assignments identify the actual pair for each candidate. Changing a later plan does not change an existing run. HTTP budgets split execution into batches, not scientific samples.
Create reviewer accounts before assigning them. The owner controls execution, export and candidate distribution. Reviewers inspect sources and record their own decisions in the same candidate workspace; there is no exclusive inspector role. A reviewer may adjudicate other candidates, but never their own pair. The owner and review participants can prepare shared evidence and gaps. Frozen snapshots remain unchanged. Existing runs can adopt a larger operational team through an explicit, audited allocation; reviewed legacy pairs and existing explicit assignments are preserved. Start with collection, then independent review. The engine's internal documentary profile also feeds web organizational and mediation discovery; these are not separate scientific transfer labels.
Record discovery-rule updates
Applying a new discovery rule to saved sources can enlarge the queue without changing the frozen cutoff, group inventory or reviewer assignments. Record the rule version, input snapshot and source hashes, and retain earlier candidate IDs and decisions. A changed queue is not evidence that additional transfers were confirmed. If the research question, scientific confirmation criteria or frozen scope change, register a new study-plan version and run rather than silently rewriting the existing snapshot.
Independent human leads may start without a documentary base. Plan how reviewers will identify a contribution and associate an event later, while keeping T6 confirmation dependent on an independently confirmed T1–T5 base and located mediation-role evidence.
Distribute review work
Open a run and select Distribute reviews. Choose the reviewers currently available for new cases, eligible adjudicators and a reproducible seed. Preview distribution displays total, submitted and pending assessments per person, category loads, pair counts and candidate examples. Apply the preview to create assignments and grant the selected accounts study access.
Allocation prioritizes individual evaluation counts, then category balance and varied pairs. Existing assignments and reviewed legacy pairs stay fixed, so exact equality may be impossible. The seed resolves ties; allocation is workload management, not probability sampling. Pending unreviewed cases without explicit assignments can receive different pairs when an owner first adopts allocation on a legacy run.
New candidates remain unassigned until the owner applies another distribution. Previously assigned candidates are not reshuffled. New team choices apply to new assignments; they do not rewrite a frozen study plan or existing reviewers. Assignment and allocation history are included in the run export.
Reviewers see pending counts on their home page and tasks under My reviews. Divergent decisions or label sets create an adjudication task for an eligible third person. These are in-app queues refreshed on page load; the application does not send email notifications.