novelty-seeking/experiments/docs/experiment_design_2026-01-19.md

# Experiment Design: 5-Condition Idea Generation Study

**Date:** January 19, 2026
**Version:** 1.0
**Status:** Pilot Implementation

## Overview

This experiment tests whether the novelty-seeking system's two key mechanisms—**attribute decomposition** and **expert transformation**—independently and jointly improve creative ideation quality compared to direct LLM generation.

## Research Questions

1. Does decomposing a query into structured attributes improve idea diversity?
2. Do expert perspectives improve idea novelty?
3. Do these mechanisms have synergistic effects when combined?
4. Is the benefit from experts due to domain knowledge, or simply perspective-shifting?

## Experimental Design

### 2×2 Factorial Design + Control

|                    | No Attributes | With Attributes |
|--------------------|---------------|-----------------|
| **No Experts**     | C1: Direct    | C3: Attr-Only   |
| **With Experts**   | C2: Expert-Only | C4: Full Pipeline |

**Plus:** C5: Random-Perspective (tests perspective-shifting without domain knowledge)

### Condition Descriptions

#### C1: Direct Generation (Baseline)
- Single LLM call: "Generate 20 creative ideas for [query]"
- No attribute decomposition
- No expert perspectives
- Purpose: Baseline for standard LLM ideation

#### C2: Expert-Only
- 4 experts from curated occupations
- Each expert generates 5 ideas directly for the query
- No attribute decomposition
- Purpose: Isolate expert contribution

#### C3: Attribute-Only
- Decompose query into 4 fixed categories
- Generate attributes per category
- Direct idea generation per attribute (no expert framing)
- Purpose: Isolate attribute decomposition contribution

#### C4: Full Pipeline
- Full attribute decomposition (4 categories)
- Expert transformation (4 experts × 1 keyword per attribute)
- Purpose: Test combined mechanism (main system)

#### C5: Random-Perspective
- 4 random words per query (from curated pool)
- Each word used as a "perspective" to generate 5 ideas
- Purpose: Control for perspective-shifting vs. expert knowledge

---

## Key Design Decisions & Rationale

### 1. Why 5 Conditions?

C1-C4 form a 2×2 factorial design that isolates the independent contributions of:
- **Attribute decomposition** (C1 vs C3, C2 vs C4)
- **Expert perspectives** (C1 vs C2, C3 vs C4)

C5 addresses a critical confound: if experts improve ideation, is it because of their **domain knowledge** or simply because any **perspective shift** helps? By using random words instead of domain experts, C5 tests whether the perspective-taking mechanism alone provides benefits.

### 2. Why Random Words in C5 (Not Fixed)?

**Decision:** Use randomly sampled words (with seed) rather than a fixed set.

**Rationale:**
- Stronger generalization: results hold across many word combinations
- Avoids cherry-picking accusation ("you just picked easy words")
- Reproducible via random seed (seed=42)
- Each query gets different random words, increasing robustness

### 3. Why Apply Deduplication Uniformly?

**Decision:** Apply embedding-based deduplication (threshold=0.85) to ALL conditions after generation.

**Rationale:**
- Fair comparison: all conditions normalized to unique ideas
- Creates "dedup survival rate" as an additional metric
- Hypothesis: Full Pipeline ideas are diverse (low redundancy), not just numerous
- Direct generation may produce many similar ideas that collapse after dedup

### 4. Why FIXED_ONLY Categories?

**Decision:** Use 4 fixed categories: Functions, Usages, User Groups, Characteristics

**Rationale:**
- Best for proof power: isolates "attribute decomposition" effect
- No confound from dynamic category selection variability
- Universal applicability: these 4 categories apply to objects, technology, and services
- Dropped "Materials" category as it doesn't apply well to services

### 5. Why Curated Expert Source?

**Decision:** Use curated occupations (210 professions) rather than LLM-generated experts.

**Rationale:**
- Reproducibility: same occupation pool across runs
- Consistency: no variance from LLM expert generation
- Control: we know exactly which experts are available
- Validation: occupations were manually curated for diversity

### 6. Why Temperature 0.9?

**Decision:** Use temperature=0.9 for all conditions.

**Rationale:**
- Higher temperature encourages more diverse/creative outputs
- Matches typical creative task settings
- Consistent across conditions for fair comparison
- Lower temperatures (0.7) showed more repetitive outputs in testing

### 7. Why 10 Pilot Queries?

**Decision:** Start with 10 queries before scaling to full 30.

**Rationale:**
- Validate pipeline works before full investment
- Catch implementation bugs early
- Balanced across categories (3 everyday, 3 technology, 4 services)
- Sufficient for initial pattern detection

---

## Configuration Summary

| Setting | Value | Rationale |
|---------|-------|-----------|
| **LLM Model** | qwen3:8b | Local, fast, consistent |
| **Temperature** | 0.9 | Encourages creativity |
| **Expert Count** | 4 | Balance diversity vs. cost |
| **Expert Source** | Curated | Reproducibility |
| **Keywords/Expert** | 1 | Simplifies analysis |
| **Language** | English | Consistency |
| **Categories** | Functions, Usages, User Groups, Characteristics | Universal applicability |
| **Dedup Threshold** | 0.85 | Standard similarity cutoff |
| **Random Seed** | 42 | Reproducibility |
| **Pilot Queries** | 10 | Validation before scaling |

---

## Query Selection

### Pilot Queries (10)

| ID | Query | Category |
|----|-------|----------|
| A1 | Chair | Everyday |
| A5 | Bicycle | Everyday |
| A7 | Smartphone | Everyday |
| B1 | Solar panel | Technology |
| B3 | 3D printer | Technology |
| B4 | Drone | Technology |
| C1 | Food delivery service | Services |
| C2 | Online education platform | Services |
| C4 | Public transportation | Services |
| C9 | Elderly care service | Services |

### Selection Criteria
- Balanced across 3 domains (everyday objects, technology, services)
- Varying complexity levels
- Different user familiarity levels
- Subset from full 30-query experimental protocol

---

## Random Word Pool (C5)

35 words selected across 7 conceptual categories:

| Category | Words |
|----------|-------|
| Nature | ocean, mountain, forest, desert, cave |
| Optics | microscope, telescope, kaleidoscope, prism, lens |
| Animals | butterfly, elephant, octopus, eagle, ant |
| Weather | sunrise, thunderstorm, rainbow, fog, aurora |
| Art | clockwork, origami, mosaic, symphony, ballet |
| Temporal | ancient, futuristic, organic, crystalline, liquid |
| Sensory | whisper, explosion, rhythm, silence, echo |

**Selection Criteria:**
- Concrete and evocative (easy to generate associations)
- Diverse domains (no overlap with typical expert knowledge)
- No obvious connection to test queries
- Equal representation across categories

---

## Expected Outputs

### Per Condition Per Query

| Condition | Expected Ideas (pre-dedup) | Mechanism |
|-----------|---------------------------|-----------|
| C1 | 20 | Direct request |
| C2 | 20 | 4 experts × 5 ideas |
| C3 | ~20 | Varies by attribute count |
| C4 | ~20 | 4 experts × ~5 keywords × 1 description |
| C5 | 20 | 4 words × 5 ideas |

### Metrics to Collect

1. **Pre-deduplication count**: Raw ideas generated
2. **Post-deduplication count**: Unique ideas after similarity filtering
3. **Dedup survival rate**: post/pre ratio
4. **Generation metadata**: Experts/words used, attributes generated

---

## File Structure

```
experiments/
├── __init__.py
├── config.py               # Experiment configuration
├── docs/
│   └── experiment_design_2026-01-19.md  # This file
├── conditions/
│   ├── __init__.py
│   ├── c1_direct.py
│   ├── c2_expert_only.py
│   ├── c3_attribute_only.py
│   ├── c4_full_pipeline.py
│   └── c5_random_perspective.py
├── data/
│   ├── queries.json        # 10 pilot queries
│   └── random_words.json   # Word pool for C5
├── generate_ideas.py       # Main runner
├── deduplication.py        # Post-processing
└── results/                # Output (gitignored)
```

---

## Verification Checklist

- [ ] Each condition produces expected number of ideas
- [ ] Deduplication reduces count meaningfully
- [ ] Results JSON contains all required metadata
- [ ] Random seed produces reproducible C5 results
- [ ] No runtime errors on all 10 pilot queries

---

## Next Steps After Pilot

1. Analyze pilot results for obvious issues
2. Adjust parameters if needed (idea count normalization, etc.)
3. Scale to full 30 queries
4. Human evaluation of idea quality (novelty, usefulness, feasibility)
5. Statistical analysis of condition differences