chore: save local changes
This commit is contained in:
555
research/experimental_protocol.md
Normal file
555
research/experimental_protocol.md
Normal file
@@ -0,0 +1,555 @@
|
||||
# Experimental Protocol: Expert-Augmented LLM Ideation
|
||||
|
||||
## Executive Summary
|
||||
|
||||
This document outlines a comprehensive experimental design to test the hypothesis that multi-expert LLM-based ideation produces more diverse and novel ideas than direct LLM generation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Research Questions
|
||||
|
||||
| ID | Research Question |
|
||||
|----|-------------------|
|
||||
| **RQ1** | Does multi-expert generation produce higher semantic diversity than direct LLM generation? |
|
||||
| **RQ2** | Does multi-expert generation produce ideas with lower patent overlap (higher novelty)? |
|
||||
| **RQ3** | What is the optimal number of experts for maximizing diversity? |
|
||||
| **RQ4** | How do different expert sources (LLM vs Curated vs DBpedia) affect idea quality? |
|
||||
| **RQ5** | Does structured attribute decomposition enhance the multi-expert effect? |
|
||||
|
||||
---
|
||||
|
||||
## 2. Experimental Design Overview
|
||||
|
||||
### 2.1 Design Type
|
||||
**Mixed Design**: Between-subjects for main conditions × Within-subjects for queries
|
||||
|
||||
### 2.2 Variables
|
||||
|
||||
#### Independent Variables (Manipulated)
|
||||
|
||||
| Variable | Levels | Your System Parameter |
|
||||
|----------|--------|----------------------|
|
||||
| **Generation Method** | 5 levels (see conditions) | Condition-dependent |
|
||||
| **Expert Count** | 1, 2, 4, 6, 8 | `expert_count` |
|
||||
| **Expert Source** | LLM, Curated, DBpedia | `expert_source` |
|
||||
| **Attribute Structure** | With/Without decomposition | Pipeline inclusion |
|
||||
|
||||
#### Dependent Variables (Measured)
|
||||
|
||||
| Variable | Measurement Method |
|
||||
|----------|-------------------|
|
||||
| **Semantic Diversity** | Mean pairwise cosine distance (embeddings) |
|
||||
| **Cluster Spread** | Number of clusters, silhouette score |
|
||||
| **Patent Novelty** | 1 - (ideas with patent match / total ideas) |
|
||||
| **Semantic Distance** | Distance from query centroid |
|
||||
| **Human Novelty Rating** | 7-point Likert scale |
|
||||
| **Human Usefulness Rating** | 7-point Likert scale |
|
||||
| **Human Creativity Rating** | 7-point Likert scale |
|
||||
|
||||
#### Control Variables (Held Constant)
|
||||
|
||||
| Variable | Fixed Value |
|
||||
|----------|-------------|
|
||||
| LLM Model | Qwen3:8b (or specify) |
|
||||
| Temperature | 0.7 |
|
||||
| Total Ideas per Query | 20 |
|
||||
| Keywords per Expert | 1 |
|
||||
| Deduplication | Disabled for raw comparison |
|
||||
| Language | English (for patent search) |
|
||||
|
||||
---
|
||||
|
||||
## 3. Experimental Conditions
|
||||
|
||||
### 3.1 Main Study: Generation Method Comparison
|
||||
|
||||
| Condition | Description | Implementation |
|
||||
|-----------|-------------|----------------|
|
||||
| **C1: Direct** | Direct LLM generation | Prompt: "Generate 20 creative ideas for [query]" |
|
||||
| **C2: Single-Expert** | 1 expert × 20 ideas | `expert_count=1`, `keywords_per_expert=20` |
|
||||
| **C3: Multi-Expert-4** | 4 experts × 5 ideas each | `expert_count=4`, `keywords_per_expert=5` |
|
||||
| **C4: Multi-Expert-8** | 8 experts × 2-3 ideas each | `expert_count=8`, `keywords_per_expert=2-3` |
|
||||
| **C5: Random-Perspective** | 4 random words as "perspectives" | Custom prompt with random nouns |
|
||||
|
||||
### 3.2 Expert Count Study
|
||||
|
||||
| Condition | Expert Count | Ideas per Expert |
|
||||
|-----------|--------------|------------------|
|
||||
| **E1** | 1 | 20 |
|
||||
| **E2** | 2 | 10 |
|
||||
| **E4** | 4 | 5 |
|
||||
| **E6** | 6 | 3-4 |
|
||||
| **E8** | 8 | 2-3 |
|
||||
|
||||
### 3.3 Expert Source Study
|
||||
|
||||
| Condition | Source | Implementation |
|
||||
|-----------|--------|----------------|
|
||||
| **S-LLM** | LLM-generated | `expert_source=ExpertSource.LLM` |
|
||||
| **S-Curated** | Curated 210 occupations | `expert_source=ExpertSource.CURATED` |
|
||||
| **S-DBpedia** | DBpedia 2164 occupations | `expert_source=ExpertSource.DBPEDIA` |
|
||||
| **S-Random** | Random word "experts" | Custom implementation |
|
||||
|
||||
---
|
||||
|
||||
## 4. Query Dataset
|
||||
|
||||
### 4.1 Design Principles
|
||||
- **Diversity**: Cover multiple domains (consumer products, technology, services, abstract concepts)
|
||||
- **Complexity Variation**: Simple objects to complex systems
|
||||
- **Familiarity Variation**: Common items to specialized equipment
|
||||
- **Cultural Neutrality**: Concepts understandable across cultures
|
||||
|
||||
### 4.2 Query Set (30 Queries)
|
||||
|
||||
#### Category A: Everyday Objects (10)
|
||||
| ID | Query | Complexity |
|
||||
|----|-------|------------|
|
||||
| A1 | Chair | Low |
|
||||
| A2 | Umbrella | Low |
|
||||
| A3 | Backpack | Low |
|
||||
| A4 | Coffee mug | Low |
|
||||
| A5 | Bicycle | Medium |
|
||||
| A6 | Refrigerator | Medium |
|
||||
| A7 | Smartphone | Medium |
|
||||
| A8 | Running shoes | Medium |
|
||||
| A9 | Kitchen knife | Low |
|
||||
| A10 | Desk lamp | Low |
|
||||
|
||||
#### Category B: Technology & Tools (10)
|
||||
| ID | Query | Complexity |
|
||||
|----|-------|------------|
|
||||
| B1 | Solar panel | Medium |
|
||||
| B2 | Electric vehicle | High |
|
||||
| B3 | 3D printer | High |
|
||||
| B4 | Drone | Medium |
|
||||
| B5 | Smart thermostat | Medium |
|
||||
| B6 | Noise-canceling headphones | Medium |
|
||||
| B7 | Water purifier | Medium |
|
||||
| B8 | Wind turbine | High |
|
||||
| B9 | Robotic vacuum | Medium |
|
||||
| B10 | Wearable fitness tracker | Medium |
|
||||
|
||||
#### Category C: Services & Systems (10)
|
||||
| ID | Query | Complexity |
|
||||
|----|-------|------------|
|
||||
| C1 | Food delivery service | Medium |
|
||||
| C2 | Online education platform | High |
|
||||
| C3 | Healthcare appointment system | High |
|
||||
| C4 | Public transportation | High |
|
||||
| C5 | Hotel booking system | Medium |
|
||||
| C6 | Personal finance app | Medium |
|
||||
| C7 | Grocery shopping experience | Medium |
|
||||
| C8 | Parking solution | Medium |
|
||||
| C9 | Elderly care service | High |
|
||||
| C10 | Waste management system | High |
|
||||
|
||||
### 4.3 Sample Size Justification
|
||||
|
||||
Based on [CHI meta-study on effect sizes](https://dl.acm.org/doi/10.1145/3706598.3713671):
|
||||
|
||||
- **Queries**: 30 (crossed with conditions)
|
||||
- **Expected effect size**: d = 0.5 (medium)
|
||||
- **Power target**: 80%
|
||||
- **For automatic metrics**: 30 queries × 5 conditions × 20 ideas = 3,000 ideas
|
||||
- **For human evaluation**: Subset of 10 queries × 3 conditions × 20 ideas = 600 ideas
|
||||
|
||||
---
|
||||
|
||||
## 5. Automatic Metrics Collection
|
||||
|
||||
### 5.1 Semantic Diversity Metrics
|
||||
|
||||
#### 5.1.1 Mean Pairwise Distance (Primary)
|
||||
```python
|
||||
def compute_mean_pairwise_distance(ideas: List[str], embedding_model: str) -> float:
|
||||
"""
|
||||
Compute mean cosine distance between all idea pairs.
|
||||
Higher = more diverse.
|
||||
"""
|
||||
embeddings = get_embeddings(ideas, model=embedding_model)
|
||||
n = len(embeddings)
|
||||
distances = []
|
||||
for i in range(n):
|
||||
for j in range(i+1, n):
|
||||
dist = 1 - cosine_similarity(embeddings[i], embeddings[j])
|
||||
distances.append(dist)
|
||||
return np.mean(distances), np.std(distances)
|
||||
```
|
||||
|
||||
#### 5.1.2 Cluster Analysis
|
||||
```python
|
||||
def compute_cluster_metrics(ideas: List[str], embedding_model: str) -> dict:
|
||||
"""
|
||||
Analyze idea clustering patterns.
|
||||
"""
|
||||
embeddings = get_embeddings(ideas, model=embedding_model)
|
||||
|
||||
# Find optimal k using silhouette score
|
||||
silhouette_scores = []
|
||||
for k in range(2, min(len(ideas), 10)):
|
||||
kmeans = KMeans(n_clusters=k)
|
||||
labels = kmeans.fit_predict(embeddings)
|
||||
score = silhouette_score(embeddings, labels)
|
||||
silhouette_scores.append((k, score))
|
||||
|
||||
best_k = max(silhouette_scores, key=lambda x: x[1])[0]
|
||||
|
||||
return {
|
||||
'optimal_clusters': best_k,
|
||||
'silhouette_score': max(silhouette_scores, key=lambda x: x[1])[1],
|
||||
'cluster_distribution': compute_cluster_sizes(embeddings, best_k)
|
||||
}
|
||||
```
|
||||
|
||||
#### 5.1.3 Semantic Distance from Query
|
||||
```python
|
||||
def compute_query_distance(query: str, ideas: List[str], embedding_model: str) -> dict:
|
||||
"""
|
||||
Measure how far ideas are from the original query.
|
||||
Higher = more novel/distant.
|
||||
"""
|
||||
query_emb = get_embedding(query, model=embedding_model)
|
||||
idea_embs = get_embeddings(ideas, model=embedding_model)
|
||||
|
||||
distances = [1 - cosine_similarity(query_emb, e) for e in idea_embs]
|
||||
|
||||
return {
|
||||
'mean_distance': np.mean(distances),
|
||||
'max_distance': np.max(distances),
|
||||
'min_distance': np.min(distances),
|
||||
'std_distance': np.std(distances)
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 Patent Novelty Metrics
|
||||
|
||||
#### 5.2.1 Patent Overlap Rate
|
||||
```python
|
||||
def compute_patent_novelty(ideas: List[str], query: str) -> dict:
|
||||
"""
|
||||
Search patents for each idea and compute overlap rate.
|
||||
Uses existing patent_search_service.
|
||||
"""
|
||||
matches = 0
|
||||
match_details = []
|
||||
|
||||
for idea in ideas:
|
||||
result = patent_search_service.search(idea)
|
||||
if result.has_match:
|
||||
matches += 1
|
||||
match_details.append({
|
||||
'idea': idea,
|
||||
'patent': result.best_match
|
||||
})
|
||||
|
||||
return {
|
||||
'novelty_rate': 1 - (matches / len(ideas)),
|
||||
'match_count': matches,
|
||||
'total_ideas': len(ideas),
|
||||
'match_details': match_details
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 Metrics Summary Table
|
||||
|
||||
| Metric | Formula | Interpretation |
|
||||
|--------|---------|----------------|
|
||||
| **Mean Pairwise Distance** | avg(1 - cos_sim(i, j)) for all pairs | Higher = more diverse |
|
||||
| **Silhouette Score** | Cluster cohesion vs separation | Higher = clearer clusters |
|
||||
| **Optimal Cluster Count** | argmax(silhouette) | More clusters = more themes |
|
||||
| **Query Distance** | 1 - cos_sim(query, idea) | Higher = farther from original |
|
||||
| **Patent Novelty Rate** | 1 - (matches / total) | Higher = more novel |
|
||||
|
||||
---
|
||||
|
||||
## 6. Human Evaluation Protocol
|
||||
|
||||
### 6.1 Participants
|
||||
|
||||
#### 6.1.1 Recruitment
|
||||
- **Platform**: Prolific, MTurk, or domain experts
|
||||
- **Sample Size**: 60 evaluators (20 per condition group)
|
||||
- **Criteria**:
|
||||
- Native English speakers
|
||||
- Bachelor's degree or higher
|
||||
- Attention check pass rate > 80%
|
||||
|
||||
#### 6.1.2 Compensation
|
||||
- $15/hour equivalent
|
||||
- ~30 minutes per session
|
||||
- Bonus for high-quality ratings
|
||||
|
||||
### 6.2 Rating Scales
|
||||
|
||||
#### 6.2.1 Novelty (7-point Likert)
|
||||
```
|
||||
How novel/surprising is this idea?
|
||||
1 = Not at all novel (very common/obvious)
|
||||
4 = Moderately novel
|
||||
7 = Extremely novel (never seen before)
|
||||
```
|
||||
|
||||
#### 6.2.2 Usefulness (7-point Likert)
|
||||
```
|
||||
How useful/practical is this idea?
|
||||
1 = Not at all useful (impractical)
|
||||
4 = Moderately useful
|
||||
7 = Extremely useful (highly practical)
|
||||
```
|
||||
|
||||
#### 6.2.3 Creativity (7-point Likert)
|
||||
```
|
||||
How creative is this idea overall?
|
||||
1 = Not at all creative
|
||||
4 = Moderately creative
|
||||
7 = Extremely creative
|
||||
```
|
||||
|
||||
### 6.3 Procedure
|
||||
|
||||
1. **Introduction** (5 min)
|
||||
- Study purpose (without revealing hypotheses)
|
||||
- Rating scale explanation
|
||||
- Practice with 3 example ideas
|
||||
|
||||
2. **Training** (5 min)
|
||||
- Rate 5 calibration ideas with feedback
|
||||
- Discuss edge cases
|
||||
|
||||
3. **Main Evaluation** (20 min)
|
||||
- Rate 30 ideas (randomized order)
|
||||
- 3 attention check items embedded
|
||||
- Break after 15 ideas
|
||||
|
||||
4. **Debriefing** (2 min)
|
||||
- Demographics
|
||||
- Open-ended feedback
|
||||
|
||||
### 6.4 Quality Control
|
||||
|
||||
| Check | Threshold | Action |
|
||||
|-------|-----------|--------|
|
||||
| Attention checks | < 2/3 correct | Exclude |
|
||||
| Completion time | < 10 min | Flag for review |
|
||||
| Variance in ratings | All same score | Exclude |
|
||||
| Inter-rater reliability | Cronbach's α < 0.7 | Review ratings |
|
||||
|
||||
### 6.5 Analysis Plan
|
||||
|
||||
#### 6.5.1 Reliability
|
||||
- Cronbach's alpha for each scale
|
||||
- ICC (Intraclass Correlation) for inter-rater agreement
|
||||
|
||||
#### 6.5.2 Main Analysis
|
||||
- Mixed-effects ANOVA: Condition × Query
|
||||
- Post-hoc: Tukey HSD for pairwise comparisons
|
||||
- Effect sizes: Cohen's d
|
||||
|
||||
#### 6.5.3 Correlation with Automatic Metrics
|
||||
- Pearson correlation: Human ratings vs semantic diversity
|
||||
- Regression: Predict human ratings from automatic metrics
|
||||
|
||||
---
|
||||
|
||||
## 7. Experimental Procedure
|
||||
|
||||
### 7.1 Phase 1: Idea Generation
|
||||
|
||||
```
|
||||
For each query Q in QuerySet:
|
||||
For each condition C in Conditions:
|
||||
|
||||
If C == "Direct":
|
||||
ideas = direct_llm_generation(Q, n=20)
|
||||
|
||||
Elif C == "Single-Expert":
|
||||
expert = generate_expert(Q, n=1)
|
||||
ideas = expert_transformation(Q, expert, ideas_per_expert=20)
|
||||
|
||||
Elif C == "Multi-Expert-4":
|
||||
experts = generate_experts(Q, n=4)
|
||||
ideas = expert_transformation(Q, experts, ideas_per_expert=5)
|
||||
|
||||
Elif C == "Multi-Expert-8":
|
||||
experts = generate_experts(Q, n=8)
|
||||
ideas = expert_transformation(Q, experts, ideas_per_expert=2-3)
|
||||
|
||||
Elif C == "Random-Perspective":
|
||||
perspectives = random.sample(RANDOM_WORDS, 4)
|
||||
ideas = perspective_generation(Q, perspectives, ideas_per=5)
|
||||
|
||||
Store(Q, C, ideas)
|
||||
```
|
||||
|
||||
### 7.2 Phase 2: Automatic Metrics
|
||||
|
||||
```
|
||||
For each (Q, C, ideas) in Results:
|
||||
metrics = {
|
||||
'diversity': compute_mean_pairwise_distance(ideas),
|
||||
'clusters': compute_cluster_metrics(ideas),
|
||||
'query_distance': compute_query_distance(Q, ideas),
|
||||
'patent_novelty': compute_patent_novelty(ideas, Q)
|
||||
}
|
||||
Store(Q, C, metrics)
|
||||
```
|
||||
|
||||
### 7.3 Phase 3: Human Evaluation
|
||||
|
||||
```
|
||||
# Sample selection
|
||||
selected_queries = random.sample(QuerySet, 10)
|
||||
selected_conditions = ["Direct", "Multi-Expert-4", "Multi-Expert-8"]
|
||||
|
||||
# Create evaluation set
|
||||
evaluation_items = []
|
||||
For each Q in selected_queries:
|
||||
For each C in selected_conditions:
|
||||
ideas = Get(Q, C)
|
||||
For each idea in ideas:
|
||||
evaluation_items.append((Q, C, idea))
|
||||
|
||||
# Randomize and assign to evaluators
|
||||
random.shuffle(evaluation_items)
|
||||
assignments = assign_to_evaluators(evaluation_items, n_evaluators=60)
|
||||
|
||||
# Collect ratings
|
||||
ratings = collect_human_ratings(assignments)
|
||||
```
|
||||
|
||||
### 7.4 Phase 4: Analysis
|
||||
|
||||
```
|
||||
# Automatic metrics analysis
|
||||
Run ANOVA: diversity ~ condition + query + condition:query
|
||||
Run post-hoc: Tukey HSD for condition pairs
|
||||
Compute effect sizes
|
||||
|
||||
# Human ratings analysis
|
||||
Check reliability: Cronbach's alpha, ICC
|
||||
Run mixed-effects model: rating ~ condition + (1|evaluator) + (1|query)
|
||||
Compute correlations: human vs automatic metrics
|
||||
|
||||
# Visualization
|
||||
Plot: Diversity by condition (box plots)
|
||||
Plot: t-SNE of idea embeddings colored by condition
|
||||
Plot: Expert count vs diversity curve
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation Checklist
|
||||
|
||||
### 8.1 Code to Implement
|
||||
|
||||
- [ ] `experiments/generate_ideas.py` - Idea generation for all conditions
|
||||
- [ ] `experiments/compute_metrics.py` - Automatic metric computation
|
||||
- [ ] `experiments/export_for_evaluation.py` - Prepare human evaluation set
|
||||
- [ ] `experiments/analyze_results.py` - Statistical analysis
|
||||
- [ ] `experiments/visualize.py` - Generate figures
|
||||
|
||||
### 8.2 Data Files to Create
|
||||
|
||||
- [ ] `data/queries.json` - 30 queries with metadata
|
||||
- [ ] `data/random_words.json` - Random perspective words
|
||||
- [ ] `data/generated_ideas/` - Raw idea outputs
|
||||
- [ ] `data/metrics/` - Computed metric results
|
||||
- [ ] `data/human_ratings/` - Collected ratings
|
||||
|
||||
### 8.3 Analysis Outputs
|
||||
|
||||
- [ ] `results/diversity_by_condition.csv`
|
||||
- [ ] `results/patent_novelty_by_condition.csv`
|
||||
- [ ] `results/human_ratings_summary.csv`
|
||||
- [ ] `results/statistical_tests.txt`
|
||||
- [ ] `figures/` - All visualizations
|
||||
|
||||
---
|
||||
|
||||
## 9. Expected Results & Hypotheses
|
||||
|
||||
### 9.1 Primary Hypotheses
|
||||
|
||||
| Hypothesis | Prediction | Metric |
|
||||
|------------|------------|--------|
|
||||
| **H1** | Multi-Expert-4 > Single-Expert > Direct | Semantic diversity |
|
||||
| **H2** | Multi-Expert-8 ≈ Multi-Expert-4 (diminishing returns) | Semantic diversity |
|
||||
| **H3** | Multi-Expert > Direct | Patent novelty rate |
|
||||
| **H4** | LLM experts > Curated > DBpedia | Unconventionality |
|
||||
| **H5** | With attributes > Without attributes | Overall diversity |
|
||||
|
||||
### 9.2 Expected Effect Sizes
|
||||
|
||||
Based on related work:
|
||||
- Diversity increase: d = 0.5-0.8 (medium to large)
|
||||
- Patent novelty increase: 20-40% improvement
|
||||
- Human creativity rating: d = 0.3-0.5 (small to medium)
|
||||
|
||||
### 9.3 Potential Confounds
|
||||
|
||||
| Confound | Mitigation |
|
||||
|----------|-----------|
|
||||
| Query difficulty | Crossed design (all queries × all conditions) |
|
||||
| LLM variability | Multiple runs, fixed seed where possible |
|
||||
| Evaluator bias | Randomized presentation, blinding |
|
||||
| Order effects | Counterbalancing in human evaluation |
|
||||
|
||||
---
|
||||
|
||||
## 10. Timeline
|
||||
|
||||
| Week | Activity |
|
||||
|------|----------|
|
||||
| 1-2 | Implement idea generation scripts |
|
||||
| 3 | Generate all ideas (5 conditions × 30 queries) |
|
||||
| 4 | Compute automatic metrics |
|
||||
| 5 | Design and pilot human evaluation |
|
||||
| 6-7 | Run human evaluation (60 participants) |
|
||||
| 8 | Analyze results |
|
||||
| 9-10 | Write paper |
|
||||
| 11 | Internal review |
|
||||
| 12 | Submit |
|
||||
|
||||
---
|
||||
|
||||
## 11. Appendix: Direct Generation Prompt
|
||||
|
||||
For baseline condition C1 (Direct LLM generation):
|
||||
|
||||
```
|
||||
You are a creative innovation consultant. Generate 20 unique and creative ideas
|
||||
for improving or reimagining a [QUERY].
|
||||
|
||||
Requirements:
|
||||
- Each idea should be distinct and novel
|
||||
- Ideas should range from incremental improvements to radical innovations
|
||||
- Consider different aspects: materials, functions, user experiences, contexts
|
||||
- Provide a brief (15-30 word) description for each idea
|
||||
|
||||
Output format:
|
||||
1. [Idea keyword]: [Description]
|
||||
2. [Idea keyword]: [Description]
|
||||
...
|
||||
20. [Idea keyword]: [Description]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Appendix: Random Perspective Words
|
||||
|
||||
For condition C5 (Random-Perspective), sample from:
|
||||
|
||||
```json
|
||||
[
|
||||
"ocean", "mountain", "forest", "desert", "cave",
|
||||
"microscope", "telescope", "kaleidoscope", "prism", "lens",
|
||||
"butterfly", "elephant", "octopus", "eagle", "ant",
|
||||
"sunrise", "thunderstorm", "rainbow", "fog", "aurora",
|
||||
"clockwork", "origami", "mosaic", "symphony", "ballet",
|
||||
"ancient", "futuristic", "organic", "crystalline", "liquid",
|
||||
"whisper", "explosion", "rhythm", "silence", "echo"
|
||||
]
|
||||
```
|
||||
|
||||
This tests whether ANY perspective shift helps, or if EXPERT perspectives specifically matter.
|
||||
Reference in New Issue
Block a user