Adding New Analysis Types
A step-by-step guide for adding new analysis types to Nquiry's AI analysis system.
Audience: Developers implementing new analysis capabilities
Prerequisites: Familiarity with the existing analysis pipeline (see docs/reference/architecture/analysis-system.md)
Overview
Nquiry currently supports 5 analysis types: question_analysis, topic_analysis, overall_summary, gap_analysis, and error_check. Each type has a corresponding output schema, Zod validator, prompt template, test fixture set, and UI rendering component.
Adding a new analysis type touches the following files and systems:
| Layer | Files |
|---|---|
| Type definitions | app/investigations/[investigation_id]/analysis/types.ts |
| Output schema | lib/ai/analysis-output-schema.ts |
| Zod validators | lib/ai/analysis-output-validators.ts |
| Prompt mapping | lib/ai/prompt-utils.ts |
| Prompt template | prompt_template table (database migration) |
| Generation route | app/api/analysis/generate/route.ts |
| Evaluation types | lib/ai/evaluation/types.ts |
| Test fixtures | __tests__/fixtures/prompts/{basic,edge,adversarial}/ |
| UI components | app/investigations/[investigation_id]/analysis/analysis-list.tsx |
| Feature docs | docs/guide/workflow/analysis.md |
Step 1: Define the Use Case and User Need
Before writing any code, document:
- Who needs this? Which user role and investigation type benefits?
- What question does it answer? Frame it as a user question (e.g., "What happened and in what order?")
- What does the output look like? Sketch the key sections a user would see.
- How is it different from existing types? Ensure it doesn't overlap with
question_analysis,topic_analysis,gap_analysis,error_check, oroverall_summary. - What evidence types does it need? Some analysis types may only make sense with specific evidence (e.g., timeline analysis needs dated evidence).
Write this up as a design spec in docs/reference/future-analysis-types.md before proceeding.
Step 2: Design the Output Schema (TypeScript Interface)
File: lib/ai/analysis-output-schema.ts
Design the TypeScript interface for the structured JSON output the AI will produce. Follow existing patterns:
Naming Convention
- Interface name:
{PascalCaseName}Output(e.g.,TimelineAnalysisOutput) - Reuse existing shared types where applicable:
ConfidenceLevel,QualityRating,EvidenceType,GapImpact,AnalysisCitation,CitedFinding
Design Principles
- Top-level fields represent major sections the user will see in the UI.
- Use arrays of typed objects for lists of items (findings, events, risks).
- Include confidence/quality indicators at the item level where appropriate.
- Include
citedFindings?: CitedFinding[]if the output should support inline citations. - Keep fields specific to the analysis purpose. Don't duplicate what existing types already handle.
Example Pattern
/**
* Structured output for timeline_analysis
*/
export interface TimelineAnalysisOutput {
/** Reconstructed timeline of events */
events: Array<{
date: string;
description: string;
evidenceIds: string[];
confidence: ConfidenceLevel;
}>;
/** Gaps in the timeline */
timelineGaps: Array<{
afterEvent: string;
beforeEvent: string;
significance: GapImpact;
description: string;
}>;
/** Conflicting accounts of the same events */
conflicts: Array<{
event: string;
accounts: Array<{ source: string; version: string }>;
resolution?: string;
}>;
/** Overall timeline assessment */
assessment: {
completeness: string;
confidence: ConfidenceLevel;
summary: string;
};
/** Key findings with inline citations */
citedFindings?: CitedFinding[];
}
Checklist
- Interface defined with JSDoc comments on each field
- Reuses existing shared types (ConfidenceLevel, GapImpact, etc.) where applicable
- Exported from
lib/ai/analysis-output-schema.ts - Added to the
StructuredOutputunion type inapp/investigations/[investigation_id]/analysis/types.ts
Step 3: Create Zod Validation Schema
File: lib/ai/analysis-output-validators.ts
Create a Zod schema that mirrors the TypeScript interface exactly. This schema validates AI output at runtime.
Naming Convention
- Schema name:
{PascalCaseName}OutputSchema(e.g.,TimelineAnalysisOutputSchema) - Inferred type:
Validated{PascalCaseName}Output(e.g.,ValidatedTimelineOutput)
What to Add
- Define the Zod schema in the "Main Output Schemas" section.
- Add an inferred type in the "Type Inference" section.
- Add to
StructuredOutputTypeunion so it's included in the generic validation. - Create a dedicated validation function following the pattern of
validateAnalysisOutput(),validateGapOutput(), etc. - Register in
getSchemaForAnalysisType()-- add case entries for both theAnalysisTypevalue and theprompt_typevalue (these differ; see Step 5). - Register in
validateStructuredOutputByType()-- this happens automatically if you updategetSchemaForAnalysisType().
Example Pattern
// In Main Output Schemas section:
export const TimelineAnalysisOutputSchema = z.object({
events: z.array(
z.object({
date: z.string(),
description: z.string(),
evidenceIds: z.array(z.string()),
confidence: ConfidenceLevelSchema,
}),
),
timelineGaps: z.array(
z.object({
afterEvent: z.string(),
beforeEvent: z.string(),
significance: GapImpactSchema,
description: z.string(),
}),
),
conflicts: z.array(
z.object({
event: z.string(),
accounts: z.array(z.object({ source: z.string(), version: z.string() })),
resolution: z.string().optional(),
}),
),
assessment: z.object({
completeness: z.string(),
confidence: ConfidenceLevelSchema,
summary: z.string(),
}),
citedFindings: z.array(CitedFindingSchema).optional(),
});
// In Type Inference section:
export type ValidatedTimelineOutput = z.infer<typeof TimelineAnalysisOutputSchema>;
// In StructuredOutputType union:
export type StructuredOutputType =
| ValidatedAnalysisOutput
| ValidatedSummaryOutput
| ValidatedGapOutput
| ValidatedErrorCheckOutput
| ValidatedTimelineOutput; // ADD THIS
// In getSchemaForAnalysisType():
case "timeline_analysis":
case "analysis_timeline":
return TimelineAnalysisOutputSchema as z.ZodSchema<StructuredOutputType>;
Checklist
- Zod schema mirrors TypeScript interface exactly
- All enum fields use existing Zod enum schemas (ConfidenceLevelSchema, GapImpactSchema, etc.)
- Inferred type exported
- Added to StructuredOutputType union
- Registered in
getSchemaForAnalysisType()with both naming conventions - Dedicated
validate{Name}Output()function created
Step 4: Write the Prompt Template with Framework Integration
Prompt templates are stored in the prompt_template database table, not in code. You'll create a database migration to insert the new template.
Prompt Type Naming Convention
The prompt_type column uses a different naming convention than AnalysisType:
| AnalysisType (API) | prompt_type (DB) |
|---|---|
question_analysis | analysis_question |
topic_analysis | analysis_topic |
overall_summary | analysis_summary |
gap_analysis | analysis_gap |
error_check | analysis_error |
timeline_analysis | analysis_timeline |
comparative_analysis | analysis_comparative |
Pattern: The prompt_type uses analysis_ prefix followed by the specific type name.
Creating the Migration
touch supabase/migrations/$(date +%Y%m%d%H%M%S)_add_{type_name}_prompt.sql
Migration Structure
-- Add new analysis prompt template
INSERT INTO prompt_template (
prompt_id,
prompt_type,
version,
system_prompt,
user_prompt_template,
is_active,
created_by,
description
) VALUES (
gen_random_uuid(),
'analysis_timeline', -- prompt_type
1, -- initial version
E'You are an expert investigation analyst...', -- system_prompt
E'## Investigation: {{investigation_title}}\n\n## Focus\n{{focus_statement}}\n\n...', -- user_prompt_template
true,
'system',
'Timeline reconstruction analysis'
);
Prompt Design Guidelines
System Prompt Must Include:
- Role definition (expert analyst specializing in this type of analysis)
- The evidence evaluation framework reference:
{{evaluation_framework}} - Output format instructions (JSON schema description matching your TypeScript interface)
- Objectivity and professional tone requirements
- Citation requirements (reference evidence by
evidence_id) - Confidence level definitions
User Prompt Template Must Include:
These {{variables}} are replaced at runtime by replaceTemplateVariables() in lib/ai/prompt-utils.ts:
| Variable | Content | Available For |
|---|---|---|
{{investigation_title}} | Investigation title | All types |
{{focus_statement}} | Investigation focus statement | All types |
{{question_text}} | Specific question text | Question-scoped types |
{{topic_title}} | Topic title | Topic-scoped types |
{{topic_description}} | Topic description | Topic-scoped types |
{{evidence_list}} | Formatted evidence with IDs | All types |
{{evidence_summary}} | Short evidence list | Error check |
{{questions_list}} | All questions with status | Summary/gap types |
{{topics_list}} | All topics | Summary types |
{{background_docs_list}} | Background documents | All types |
{{investigator_direction}} | User-provided direction | All types |
{{evaluation_framework}} | Evidence evaluation framework text | All types |
{{framework_documents}} | Uploaded framework documents | All types |
{{work_type_label}} | Dynamic work type (Investigation, Audit, etc.) | All types |
If your analysis type needs new variables, you must also update PromptVariables interface in lib/ai/prompt-utils.ts and add the variable population logic in app/api/analysis/generate/route.ts.
Prompt Versioning
- Initial version is always
1 - Future updates create new rows in
prompt_template_history(handled automatically by the system) - Version naming: Major.Minor.Patch conceptually, integer in DB
- Always keep the previous version accessible for comparison
Checklist
- Migration file created with
touch supabase/migrations/$(date +%Y%m%d%H%M%S)_add_{type}_prompt.sql - System prompt includes framework reference and output format
- User prompt template uses all required
{{variables}} - JSON output instructions match the TypeScript interface exactly
- Run migration with
npm run db:migrate
Step 5: Add to AnalysisType Enum and Supporting Mappings
5a. Update AnalysisType
File: app/investigations/[investigation_id]/analysis/types.ts
export type AnalysisType =
| "question_analysis"
| "overall_summary"
| "gap_analysis"
| "error_check"
| "topic_analysis"
| "timeline_analysis"; // ADD THIS
5b. Update ANALYSIS_TYPES Display Array
Same file -- add the display metadata:
{
value: "timeline_analysis",
label: "Timeline Analysis",
description: "Chronological reconstruction of events from evidence",
icon: "timeline",
},
5c. Update StructuredOutput Union Type
Same file -- add to the import and union:
import type {
AnalysisOutput,
GapAnalysisOutput,
ErrorCheckOutput,
SummaryAnalysisOutput,
TimelineAnalysisOutput, // ADD THIS
} from "@/lib/ai/analysis-output-schema";
export type StructuredOutput =
| AnalysisOutput
| GapAnalysisOutput
| ErrorCheckOutput
| SummaryAnalysisOutput
| TimelineAnalysisOutput; // ADD THIS
5d. Update Prompt Type Mapping
File: lib/ai/prompt-utils.ts
In getPromptTypeForAnalysis():
const mapping: Record<string, string> = {
question_analysis: "analysis_question",
overall_summary: "analysis_summary",
gap_analysis: "analysis_gap",
error_check: "analysis_error",
topic_analysis: "analysis_topic",
timeline_analysis: "analysis_timeline", // ADD THIS
};
5e. Update Evaluation Types
File: lib/ai/evaluation/types.ts
Add to EvalPromptType:
export type EvalPromptType =
| "analysis_question"
| "analysis_topic"
| "analysis_summary"
| "gap_analysis"
| "error_check"
| "analysis_timeline"; // ADD THIS
Checklist
-
AnalysisTypeunion updated -
ANALYSIS_TYPESdisplay array updated with label, description, icon -
StructuredOutputunion updated -
getPromptTypeForAnalysis()mapping updated -
EvalPromptTypeupdated
Step 6: Create Test Fixtures (Basic, Edge, Adversarial)
Create at minimum 3 test fixtures covering different scenarios. Follow the existing numbering scheme.
Directory: __tests__/fixtures/prompts/
Fixture File Naming Convention
{NNN}_{descriptive_name}.json
- Numbers are sequential across all categories (current range: 001-020)
- Names should describe the scenario, not just the type
Fixture JSON Structure
Every fixture follows this structure (defined in lib/ai/evaluation/types.ts as TestFixture):
{
"test_id": "021",
"prompt_type": "analysis_timeline",
"description": "Clear chronological sequence with multiple dated sources",
"input": {
"investigation_title": "...",
"focus_statement": "...",
"question_text": "...", // Optional: only for question-scoped types
"topic_title": "...", // Optional: only for topic-scoped types
"topic_description": "...", // Optional
"evidence": [
{
"evidence_id": "E001",
"title": "...",
"content_text": "...",
"evidence_type": "testimonial|documentary|physical|digital|expert|quantitative",
"source": "...",
"source_date": "YYYY-MM-DD" // Optional but highly recommended
}
]
},
"expected": {
"valid_json": true,
"schema_valid": true,
"confidence_levels": ["established", "probable"], // Acceptable confidence levels
"must_cite": ["E001", "E002"], // Evidence IDs that must be cited
"quality_ratings_required": true, // Optional
"min_evidence_assessments": 3, // Optional
"should_identify_gaps": false, // Optional
"should_flag_issues": false, // Optional
"must_not_contain": ["phrase to reject"] // Optional: adversarial protection
},
"notes": "Human-readable explanation of what this fixture tests and what correct output looks like"
}
Required Fixture Categories
Basic (happy path):
- Place in
__tests__/fixtures/prompts/basic/ - Test the core scenario with clean, clear evidence
- Expected result: high confidence, valid schema, all evidence cited
Edge (challenging but legitimate):
- Place in
__tests__/fixtures/prompts/edge/ - Test boundary conditions: conflicting dates, partial evidence, ambiguous sources
- Expected result: lower confidence, gaps identified, issues flagged
Adversarial (things that should NOT fool the AI):
- Place in
__tests__/fixtures/prompts/adversarial/ - Test: leading framing, irrelevant evidence, injection attempts, biased input
- Expected result: AI maintains objectivity, doesn't accept flawed premises
Evidence Design Tips
- Use realistic but fictional data (real names and organizations should never appear)
- Include enough detail to be meaningful (3-5 evidence items minimum)
- Vary evidence types (don't make everything documentary)
- Include dates and sources -- these matter for timeline and comparative analysis
- For adversarial fixtures, make the trap subtle, not obvious
Checklist
- At least 1 basic fixture in
basic/ - At least 1 edge fixture in
edge/ - At least 1 adversarial fixture in
adversarial/ - All fixtures follow the
TestFixtureJSON structure - Evidence IDs are unique within each fixture (E001, E002, etc.)
- Notes explain what correct output looks like
Step 7: Build UI Display Component
The analysis list and detail views need to know how to render the new structured output.
Where to Add
File: app/investigations/[investigation_id]/analysis/analysis-list.tsx
This component renders analysis results and dispatches rendering based on analysis_type and structured_output. You need to add a rendering branch for your new type.
Component Pattern
- Create a new component (or section within the existing component) that renders the fields of your output schema.
- Use existing UI patterns from how
AnalysisOutput,GapAnalysisOutput, etc. are rendered. - Reuse shared components:
confidence-badge.tsxfor confidence level displayquality-metrics-panel.tsxfor quality metricsquality-metrics-badge.tsxfor compact quality displayevidence-assessment.tsxfor evidence assessment renderingevidence-retrieval-panel.tsxfor retrieval transparency
Design Guidelines
- Each top-level field in the output schema should have a visible section in the UI
- Confidence levels should use the
ConfidenceBadgecomponent - Evidence citations should be clickable (link to evidence detail)
- Collapsible sections for detailed data (expand/collapse pattern)
- Mobile-responsive layout
Checklist
- Rendering component handles the new output type
- Confidence badges displayed
- Evidence citations are interactive
- Graceful fallback if
structured_outputis null (showanalysis_textmarkdown) - Responsive design
Step 8: Add to Evaluation Harness
Integrate the new type into the prompt evaluation framework so quality can be measured systematically.
Files to Update
File: lib/ai/evaluation/types.ts
The EvalPromptType was already updated in Step 5e. Now ensure:
- The evaluation harness knows how to load fixtures for the new prompt type.
- Type-specific metrics are defined for the new analysis type.
Type-Specific Metrics
Define what quality means specifically for this analysis type. Examples:
| Analysis Type | Type-Specific Metrics |
|---|---|
analysis_question | Confidence appropriate to evidence, alternatives considered |
analysis_gap | Gaps correctly prioritized, actionable recommendations |
analysis_error | Contradictions identified, materiality assessed |
analysis_timeline | Events ordered correctly, date conflicts flagged, gaps noted |
analysis_comparative | Both sides represented, divergences accurate, objectivity held |
Checklist
-
EvalPromptTypeincludes the new type - Type-specific metrics defined
- Evaluation harness can discover and run fixtures for the new type
- Passing criteria documented
Step 9: Test with Real Evidence
Before deploying, test the new analysis type against real-world evidence (from a test investigation).
Testing Process
- Create a test investigation using the test account (
test@nquiry.ai) - Upload realistic evidence that exercises the analysis type
- Generate analysis and review the output
- Verify structured output parses correctly (check browser console for validation errors)
- Check quality metrics -- faithfulness, coverage, retrieval quality
- Iterate on the prompt if output quality is unsatisfactory
What to Look For
- Does the output match the schema? (No validation errors)
- Is the confidence level appropriate for the evidence?
- Are all evidence items cited where relevant?
- Does the output address the investigation's focus statement?
- Is the tone professional and objective?
- Does the framework integration work correctly?
Manual Testing Checklist
- Generate analysis with full evidence set
- Generate analysis with minimal evidence (should show low confidence)
- Generate analysis with conflicting evidence (should flag issues)
- Generate analysis with investigator direction
- Regenerate analysis (iteration tracking)
- Verify structured output renders correctly in UI
- Verify quality metrics display
Step 10: Deploy with Monitoring
Deployment Steps
- Run all tests:
npm test-- all must pass - Create migration PR with the prompt template
- Deploy code changes through standard CI/CD pipeline
- Run migration in production:
npm run db:migrate - Monitor for the first 24 hours
Monitoring Checklist
- Validation pass rate for the new type (target: 99%+)
- Generation success rate (target: 99%+)
- Average latency (target: < 25s P95)
- User regeneration rate (target: < 20%)
- No error spikes in logs
Post-Deploy
- Update
docs/guide/workflow/analysis.mdwith the new analysis type - Update
docs/reference/architecture/analysis-system.mdPart 5, Workstream 6 table - Add to
docs/ROADMAP.mdDone section
Common Pitfalls
1. Schema Mismatch Between TypeScript and Zod
The TypeScript interface and Zod schema must match exactly. If you add an optional field to the interface, it must be .optional() in Zod. If you forget, validation will fail silently (the AI may produce correct output that fails validation).
Fix: Always define both in the same session and cross-check field by field.
2. Forgetting the Prompt Type Mapping
The AnalysisType (e.g., timeline_analysis) differs from the prompt_type (e.g., analysis_timeline). If you forget to add the mapping in getPromptTypeForAnalysis(), the system will fail to find the prompt template.
Fix: Always update both getPromptTypeForAnalysis() in lib/ai/prompt-utils.ts and getSchemaForAnalysisType() in lib/ai/analysis-output-validators.ts.
3. Not Registering in the Validator Dispatch
If you add a schema but don't add it to getSchemaForAnalysisType(), validation will fall through to the default AnalysisOutputSchema, causing false validation failures.
Fix: Add case statements for both naming conventions (e.g., timeline_analysis AND analysis_timeline).
4. Prompt Template JSON Instructions Don't Match Schema
The prompt tells Claude what JSON structure to produce. If the instructions in the prompt don't exactly match the Zod schema, the AI will produce output that fails validation.
Fix: Copy the Zod schema field names directly into the prompt instructions. Test with a fixture before deploying.
5. Missing UI Rendering Branch
If the structured output has a new shape but the UI doesn't have a rendering branch for it, the component may crash or show nothing.
Fix: Add a type guard or switch case in the analysis rendering component before deploying.
6. Not Handling the null Case
The structured_output field on analysis can be null (e.g., if the AI produced markdown but not valid JSON). Always render analysis_text as a fallback.
7. Skipping Adversarial Fixtures
Basic fixtures only test the happy path. Without adversarial fixtures, you won't catch cases where the AI produces biased, leading, or structurally broken output.
Fix: Always create at least one adversarial fixture that tests whether the AI maintains objectivity.
Quick Reference: File Change Checklist
When adding a new analysis type {type_name}, touch these files:
lib/ai/analysis-output-schema.ts # TypeScript interface
lib/ai/analysis-output-validators.ts # Zod schema + validation function + dispatch
lib/ai/prompt-utils.ts # getPromptTypeForAnalysis() mapping
lib/ai/evaluation/types.ts # EvalPromptType union
app/investigations/[investigation_id]/analysis/types.ts # AnalysisType + ANALYSIS_TYPES + StructuredOutput
app/api/analysis/generate/route.ts # Generation logic (if new variables needed)
app/investigations/[investigation_id]/analysis/analysis-list.tsx # UI rendering
supabase/migrations/YYYYMMDDHHMMSS_add_{type}_prompt.sql # Prompt template
__tests__/fixtures/prompts/basic/ # Basic test fixture(s)
__tests__/fixtures/prompts/edge/ # Edge case fixture(s)
__tests__/fixtures/prompts/adversarial/ # Adversarial fixture(s)
docs/guide/workflow/analysis.md # User documentation
Related Documentation
- Analysis System Master Plan - Strategic overview
- AI Analysis - User-facing feature documentation
- AI Quality Metrics - Quality measurement details
- Future Analysis Types - Design specs for planned types