Skip to main content

Adding New Analysis Types

A step-by-step guide for adding new analysis types to Nquiry's AI analysis system.

Audience: Developers implementing new analysis capabilities Prerequisites: Familiarity with the existing analysis pipeline (see docs/reference/architecture/analysis-system.md)


Overview

Nquiry currently supports 5 analysis types: question_analysis, topic_analysis, overall_summary, gap_analysis, and error_check. Each type has a corresponding output schema, Zod validator, prompt template, test fixture set, and UI rendering component.

Adding a new analysis type touches the following files and systems:

LayerFiles
Type definitionsapp/investigations/[investigation_id]/analysis/types.ts
Output schemalib/ai/analysis-output-schema.ts
Zod validatorslib/ai/analysis-output-validators.ts
Prompt mappinglib/ai/prompt-utils.ts
Prompt templateprompt_template table (database migration)
Generation routeapp/api/analysis/generate/route.ts
Evaluation typeslib/ai/evaluation/types.ts
Test fixtures__tests__/fixtures/prompts/{basic,edge,adversarial}/
UI componentsapp/investigations/[investigation_id]/analysis/analysis-list.tsx
Feature docsdocs/guide/workflow/analysis.md

Step 1: Define the Use Case and User Need

Before writing any code, document:

  1. Who needs this? Which user role and investigation type benefits?
  2. What question does it answer? Frame it as a user question (e.g., "What happened and in what order?")
  3. What does the output look like? Sketch the key sections a user would see.
  4. How is it different from existing types? Ensure it doesn't overlap with question_analysis, topic_analysis, gap_analysis, error_check, or overall_summary.
  5. What evidence types does it need? Some analysis types may only make sense with specific evidence (e.g., timeline analysis needs dated evidence).

Write this up as a design spec in docs/reference/future-analysis-types.md before proceeding.


Step 2: Design the Output Schema (TypeScript Interface)

File: lib/ai/analysis-output-schema.ts

Design the TypeScript interface for the structured JSON output the AI will produce. Follow existing patterns:

Naming Convention

  • Interface name: {PascalCaseName}Output (e.g., TimelineAnalysisOutput)
  • Reuse existing shared types where applicable: ConfidenceLevel, QualityRating, EvidenceType, GapImpact, AnalysisCitation, CitedFinding

Design Principles

  1. Top-level fields represent major sections the user will see in the UI.
  2. Use arrays of typed objects for lists of items (findings, events, risks).
  3. Include confidence/quality indicators at the item level where appropriate.
  4. Include citedFindings?: CitedFinding[] if the output should support inline citations.
  5. Keep fields specific to the analysis purpose. Don't duplicate what existing types already handle.

Example Pattern

/**
* Structured output for timeline_analysis
*/
export interface TimelineAnalysisOutput {
/** Reconstructed timeline of events */
events: Array<{
date: string;
description: string;
evidenceIds: string[];
confidence: ConfidenceLevel;
}>;

/** Gaps in the timeline */
timelineGaps: Array<{
afterEvent: string;
beforeEvent: string;
significance: GapImpact;
description: string;
}>;

/** Conflicting accounts of the same events */
conflicts: Array<{
event: string;
accounts: Array<{ source: string; version: string }>;
resolution?: string;
}>;

/** Overall timeline assessment */
assessment: {
completeness: string;
confidence: ConfidenceLevel;
summary: string;
};

/** Key findings with inline citations */
citedFindings?: CitedFinding[];
}

Checklist

  • Interface defined with JSDoc comments on each field
  • Reuses existing shared types (ConfidenceLevel, GapImpact, etc.) where applicable
  • Exported from lib/ai/analysis-output-schema.ts
  • Added to the StructuredOutput union type in app/investigations/[investigation_id]/analysis/types.ts

Step 3: Create Zod Validation Schema

File: lib/ai/analysis-output-validators.ts

Create a Zod schema that mirrors the TypeScript interface exactly. This schema validates AI output at runtime.

Naming Convention

  • Schema name: {PascalCaseName}OutputSchema (e.g., TimelineAnalysisOutputSchema)
  • Inferred type: Validated{PascalCaseName}Output (e.g., ValidatedTimelineOutput)

What to Add

  1. Define the Zod schema in the "Main Output Schemas" section.
  2. Add an inferred type in the "Type Inference" section.
  3. Add to StructuredOutputType union so it's included in the generic validation.
  4. Create a dedicated validation function following the pattern of validateAnalysisOutput(), validateGapOutput(), etc.
  5. Register in getSchemaForAnalysisType() -- add case entries for both the AnalysisType value and the prompt_type value (these differ; see Step 5).
  6. Register in validateStructuredOutputByType() -- this happens automatically if you update getSchemaForAnalysisType().

Example Pattern

// In Main Output Schemas section:
export const TimelineAnalysisOutputSchema = z.object({
events: z.array(
z.object({
date: z.string(),
description: z.string(),
evidenceIds: z.array(z.string()),
confidence: ConfidenceLevelSchema,
}),
),
timelineGaps: z.array(
z.object({
afterEvent: z.string(),
beforeEvent: z.string(),
significance: GapImpactSchema,
description: z.string(),
}),
),
conflicts: z.array(
z.object({
event: z.string(),
accounts: z.array(z.object({ source: z.string(), version: z.string() })),
resolution: z.string().optional(),
}),
),
assessment: z.object({
completeness: z.string(),
confidence: ConfidenceLevelSchema,
summary: z.string(),
}),
citedFindings: z.array(CitedFindingSchema).optional(),
});

// In Type Inference section:
export type ValidatedTimelineOutput = z.infer<typeof TimelineAnalysisOutputSchema>;

// In StructuredOutputType union:
export type StructuredOutputType =
| ValidatedAnalysisOutput
| ValidatedSummaryOutput
| ValidatedGapOutput
| ValidatedErrorCheckOutput
| ValidatedTimelineOutput; // ADD THIS

// In getSchemaForAnalysisType():
case "timeline_analysis":
case "analysis_timeline":
return TimelineAnalysisOutputSchema as z.ZodSchema<StructuredOutputType>;

Checklist

  • Zod schema mirrors TypeScript interface exactly
  • All enum fields use existing Zod enum schemas (ConfidenceLevelSchema, GapImpactSchema, etc.)
  • Inferred type exported
  • Added to StructuredOutputType union
  • Registered in getSchemaForAnalysisType() with both naming conventions
  • Dedicated validate{Name}Output() function created

Step 4: Write the Prompt Template with Framework Integration

Prompt templates are stored in the prompt_template database table, not in code. You'll create a database migration to insert the new template.

Prompt Type Naming Convention

The prompt_type column uses a different naming convention than AnalysisType:

AnalysisType (API)prompt_type (DB)
question_analysisanalysis_question
topic_analysisanalysis_topic
overall_summaryanalysis_summary
gap_analysisanalysis_gap
error_checkanalysis_error
timeline_analysisanalysis_timeline
comparative_analysisanalysis_comparative

Pattern: The prompt_type uses analysis_ prefix followed by the specific type name.

Creating the Migration

touch supabase/migrations/$(date +%Y%m%d%H%M%S)_add_{type_name}_prompt.sql

Migration Structure

-- Add new analysis prompt template
INSERT INTO prompt_template (
prompt_id,
prompt_type,
version,
system_prompt,
user_prompt_template,
is_active,
created_by,
description
) VALUES (
gen_random_uuid(),
'analysis_timeline', -- prompt_type
1, -- initial version
E'You are an expert investigation analyst...', -- system_prompt
E'## Investigation: {{investigation_title}}\n\n## Focus\n{{focus_statement}}\n\n...', -- user_prompt_template
true,
'system',
'Timeline reconstruction analysis'
);

Prompt Design Guidelines

System Prompt Must Include:

  1. Role definition (expert analyst specializing in this type of analysis)
  2. The evidence evaluation framework reference: {{evaluation_framework}}
  3. Output format instructions (JSON schema description matching your TypeScript interface)
  4. Objectivity and professional tone requirements
  5. Citation requirements (reference evidence by evidence_id)
  6. Confidence level definitions

User Prompt Template Must Include:

These {{variables}} are replaced at runtime by replaceTemplateVariables() in lib/ai/prompt-utils.ts:

VariableContentAvailable For
{{investigation_title}}Investigation titleAll types
{{focus_statement}}Investigation focus statementAll types
{{question_text}}Specific question textQuestion-scoped types
{{topic_title}}Topic titleTopic-scoped types
{{topic_description}}Topic descriptionTopic-scoped types
{{evidence_list}}Formatted evidence with IDsAll types
{{evidence_summary}}Short evidence listError check
{{questions_list}}All questions with statusSummary/gap types
{{topics_list}}All topicsSummary types
{{background_docs_list}}Background documentsAll types
{{investigator_direction}}User-provided directionAll types
{{evaluation_framework}}Evidence evaluation framework textAll types
{{framework_documents}}Uploaded framework documentsAll types
{{work_type_label}}Dynamic work type (Investigation, Audit, etc.)All types

If your analysis type needs new variables, you must also update PromptVariables interface in lib/ai/prompt-utils.ts and add the variable population logic in app/api/analysis/generate/route.ts.

Prompt Versioning

  • Initial version is always 1
  • Future updates create new rows in prompt_template_history (handled automatically by the system)
  • Version naming: Major.Minor.Patch conceptually, integer in DB
  • Always keep the previous version accessible for comparison

Checklist

  • Migration file created with touch supabase/migrations/$(date +%Y%m%d%H%M%S)_add_{type}_prompt.sql
  • System prompt includes framework reference and output format
  • User prompt template uses all required {{variables}}
  • JSON output instructions match the TypeScript interface exactly
  • Run migration with npm run db:migrate

Step 5: Add to AnalysisType Enum and Supporting Mappings

5a. Update AnalysisType

File: app/investigations/[investigation_id]/analysis/types.ts

export type AnalysisType =
| "question_analysis"
| "overall_summary"
| "gap_analysis"
| "error_check"
| "topic_analysis"
| "timeline_analysis"; // ADD THIS

5b. Update ANALYSIS_TYPES Display Array

Same file -- add the display metadata:

{
value: "timeline_analysis",
label: "Timeline Analysis",
description: "Chronological reconstruction of events from evidence",
icon: "timeline",
},

5c. Update StructuredOutput Union Type

Same file -- add to the import and union:

import type {
AnalysisOutput,
GapAnalysisOutput,
ErrorCheckOutput,
SummaryAnalysisOutput,
TimelineAnalysisOutput, // ADD THIS
} from "@/lib/ai/analysis-output-schema";

export type StructuredOutput =
| AnalysisOutput
| GapAnalysisOutput
| ErrorCheckOutput
| SummaryAnalysisOutput
| TimelineAnalysisOutput; // ADD THIS

5d. Update Prompt Type Mapping

File: lib/ai/prompt-utils.ts

In getPromptTypeForAnalysis():

const mapping: Record<string, string> = {
question_analysis: "analysis_question",
overall_summary: "analysis_summary",
gap_analysis: "analysis_gap",
error_check: "analysis_error",
topic_analysis: "analysis_topic",
timeline_analysis: "analysis_timeline", // ADD THIS
};

5e. Update Evaluation Types

File: lib/ai/evaluation/types.ts

Add to EvalPromptType:

export type EvalPromptType =
| "analysis_question"
| "analysis_topic"
| "analysis_summary"
| "gap_analysis"
| "error_check"
| "analysis_timeline"; // ADD THIS

Checklist

  • AnalysisType union updated
  • ANALYSIS_TYPES display array updated with label, description, icon
  • StructuredOutput union updated
  • getPromptTypeForAnalysis() mapping updated
  • EvalPromptType updated

Step 6: Create Test Fixtures (Basic, Edge, Adversarial)

Create at minimum 3 test fixtures covering different scenarios. Follow the existing numbering scheme.

Directory: __tests__/fixtures/prompts/

Fixture File Naming Convention

{NNN}_{descriptive_name}.json
  • Numbers are sequential across all categories (current range: 001-020)
  • Names should describe the scenario, not just the type

Fixture JSON Structure

Every fixture follows this structure (defined in lib/ai/evaluation/types.ts as TestFixture):

{
"test_id": "021",
"prompt_type": "analysis_timeline",
"description": "Clear chronological sequence with multiple dated sources",
"input": {
"investigation_title": "...",
"focus_statement": "...",
"question_text": "...", // Optional: only for question-scoped types
"topic_title": "...", // Optional: only for topic-scoped types
"topic_description": "...", // Optional
"evidence": [
{
"evidence_id": "E001",
"title": "...",
"content_text": "...",
"evidence_type": "testimonial|documentary|physical|digital|expert|quantitative",
"source": "...",
"source_date": "YYYY-MM-DD" // Optional but highly recommended
}
]
},
"expected": {
"valid_json": true,
"schema_valid": true,
"confidence_levels": ["established", "probable"], // Acceptable confidence levels
"must_cite": ["E001", "E002"], // Evidence IDs that must be cited
"quality_ratings_required": true, // Optional
"min_evidence_assessments": 3, // Optional
"should_identify_gaps": false, // Optional
"should_flag_issues": false, // Optional
"must_not_contain": ["phrase to reject"] // Optional: adversarial protection
},
"notes": "Human-readable explanation of what this fixture tests and what correct output looks like"
}

Required Fixture Categories

Basic (happy path):

  • Place in __tests__/fixtures/prompts/basic/
  • Test the core scenario with clean, clear evidence
  • Expected result: high confidence, valid schema, all evidence cited

Edge (challenging but legitimate):

  • Place in __tests__/fixtures/prompts/edge/
  • Test boundary conditions: conflicting dates, partial evidence, ambiguous sources
  • Expected result: lower confidence, gaps identified, issues flagged

Adversarial (things that should NOT fool the AI):

  • Place in __tests__/fixtures/prompts/adversarial/
  • Test: leading framing, irrelevant evidence, injection attempts, biased input
  • Expected result: AI maintains objectivity, doesn't accept flawed premises

Evidence Design Tips

  • Use realistic but fictional data (real names and organizations should never appear)
  • Include enough detail to be meaningful (3-5 evidence items minimum)
  • Vary evidence types (don't make everything documentary)
  • Include dates and sources -- these matter for timeline and comparative analysis
  • For adversarial fixtures, make the trap subtle, not obvious

Checklist

  • At least 1 basic fixture in basic/
  • At least 1 edge fixture in edge/
  • At least 1 adversarial fixture in adversarial/
  • All fixtures follow the TestFixture JSON structure
  • Evidence IDs are unique within each fixture (E001, E002, etc.)
  • Notes explain what correct output looks like

Step 7: Build UI Display Component

The analysis list and detail views need to know how to render the new structured output.

Where to Add

File: app/investigations/[investigation_id]/analysis/analysis-list.tsx

This component renders analysis results and dispatches rendering based on analysis_type and structured_output. You need to add a rendering branch for your new type.

Component Pattern

  1. Create a new component (or section within the existing component) that renders the fields of your output schema.
  2. Use existing UI patterns from how AnalysisOutput, GapAnalysisOutput, etc. are rendered.
  3. Reuse shared components:
    • confidence-badge.tsx for confidence level display
    • quality-metrics-panel.tsx for quality metrics
    • quality-metrics-badge.tsx for compact quality display
    • evidence-assessment.tsx for evidence assessment rendering
    • evidence-retrieval-panel.tsx for retrieval transparency

Design Guidelines

  • Each top-level field in the output schema should have a visible section in the UI
  • Confidence levels should use the ConfidenceBadge component
  • Evidence citations should be clickable (link to evidence detail)
  • Collapsible sections for detailed data (expand/collapse pattern)
  • Mobile-responsive layout

Checklist

  • Rendering component handles the new output type
  • Confidence badges displayed
  • Evidence citations are interactive
  • Graceful fallback if structured_output is null (show analysis_text markdown)
  • Responsive design

Step 8: Add to Evaluation Harness

Integrate the new type into the prompt evaluation framework so quality can be measured systematically.

Files to Update

File: lib/ai/evaluation/types.ts

The EvalPromptType was already updated in Step 5e. Now ensure:

  1. The evaluation harness knows how to load fixtures for the new prompt type.
  2. Type-specific metrics are defined for the new analysis type.

Type-Specific Metrics

Define what quality means specifically for this analysis type. Examples:

Analysis TypeType-Specific Metrics
analysis_questionConfidence appropriate to evidence, alternatives considered
analysis_gapGaps correctly prioritized, actionable recommendations
analysis_errorContradictions identified, materiality assessed
analysis_timelineEvents ordered correctly, date conflicts flagged, gaps noted
analysis_comparativeBoth sides represented, divergences accurate, objectivity held

Checklist

  • EvalPromptType includes the new type
  • Type-specific metrics defined
  • Evaluation harness can discover and run fixtures for the new type
  • Passing criteria documented

Step 9: Test with Real Evidence

Before deploying, test the new analysis type against real-world evidence (from a test investigation).

Testing Process

  1. Create a test investigation using the test account (test@nquiry.ai)
  2. Upload realistic evidence that exercises the analysis type
  3. Generate analysis and review the output
  4. Verify structured output parses correctly (check browser console for validation errors)
  5. Check quality metrics -- faithfulness, coverage, retrieval quality
  6. Iterate on the prompt if output quality is unsatisfactory

What to Look For

  • Does the output match the schema? (No validation errors)
  • Is the confidence level appropriate for the evidence?
  • Are all evidence items cited where relevant?
  • Does the output address the investigation's focus statement?
  • Is the tone professional and objective?
  • Does the framework integration work correctly?

Manual Testing Checklist

  • Generate analysis with full evidence set
  • Generate analysis with minimal evidence (should show low confidence)
  • Generate analysis with conflicting evidence (should flag issues)
  • Generate analysis with investigator direction
  • Regenerate analysis (iteration tracking)
  • Verify structured output renders correctly in UI
  • Verify quality metrics display

Step 10: Deploy with Monitoring

Deployment Steps

  1. Run all tests: npm test -- all must pass
  2. Create migration PR with the prompt template
  3. Deploy code changes through standard CI/CD pipeline
  4. Run migration in production: npm run db:migrate
  5. Monitor for the first 24 hours

Monitoring Checklist

  • Validation pass rate for the new type (target: 99%+)
  • Generation success rate (target: 99%+)
  • Average latency (target: < 25s P95)
  • User regeneration rate (target: < 20%)
  • No error spikes in logs

Post-Deploy

  • Update docs/guide/workflow/analysis.md with the new analysis type
  • Update docs/reference/architecture/analysis-system.md Part 5, Workstream 6 table
  • Add to docs/ROADMAP.md Done section

Common Pitfalls

1. Schema Mismatch Between TypeScript and Zod

The TypeScript interface and Zod schema must match exactly. If you add an optional field to the interface, it must be .optional() in Zod. If you forget, validation will fail silently (the AI may produce correct output that fails validation).

Fix: Always define both in the same session and cross-check field by field.

2. Forgetting the Prompt Type Mapping

The AnalysisType (e.g., timeline_analysis) differs from the prompt_type (e.g., analysis_timeline). If you forget to add the mapping in getPromptTypeForAnalysis(), the system will fail to find the prompt template.

Fix: Always update both getPromptTypeForAnalysis() in lib/ai/prompt-utils.ts and getSchemaForAnalysisType() in lib/ai/analysis-output-validators.ts.

3. Not Registering in the Validator Dispatch

If you add a schema but don't add it to getSchemaForAnalysisType(), validation will fall through to the default AnalysisOutputSchema, causing false validation failures.

Fix: Add case statements for both naming conventions (e.g., timeline_analysis AND analysis_timeline).

4. Prompt Template JSON Instructions Don't Match Schema

The prompt tells Claude what JSON structure to produce. If the instructions in the prompt don't exactly match the Zod schema, the AI will produce output that fails validation.

Fix: Copy the Zod schema field names directly into the prompt instructions. Test with a fixture before deploying.

5. Missing UI Rendering Branch

If the structured output has a new shape but the UI doesn't have a rendering branch for it, the component may crash or show nothing.

Fix: Add a type guard or switch case in the analysis rendering component before deploying.

6. Not Handling the null Case

The structured_output field on analysis can be null (e.g., if the AI produced markdown but not valid JSON). Always render analysis_text as a fallback.

7. Skipping Adversarial Fixtures

Basic fixtures only test the happy path. Without adversarial fixtures, you won't catch cases where the AI produces biased, leading, or structurally broken output.

Fix: Always create at least one adversarial fixture that tests whether the AI maintains objectivity.


Quick Reference: File Change Checklist

When adding a new analysis type {type_name}, touch these files:

lib/ai/analysis-output-schema.ts # TypeScript interface
lib/ai/analysis-output-validators.ts # Zod schema + validation function + dispatch
lib/ai/prompt-utils.ts # getPromptTypeForAnalysis() mapping
lib/ai/evaluation/types.ts # EvalPromptType union
app/investigations/[investigation_id]/analysis/types.ts # AnalysisType + ANALYSIS_TYPES + StructuredOutput
app/api/analysis/generate/route.ts # Generation logic (if new variables needed)
app/investigations/[investigation_id]/analysis/analysis-list.tsx # UI rendering
supabase/migrations/YYYYMMDDHHMMSS_add_{type}_prompt.sql # Prompt template
__tests__/fixtures/prompts/basic/ # Basic test fixture(s)
__tests__/fixtures/prompts/edge/ # Edge case fixture(s)
__tests__/fixtures/prompts/adversarial/ # Adversarial fixture(s)
docs/guide/workflow/analysis.md # User documentation