THE PORTFOLIO ATLAS — SOURCE & METHOD A fixed-data graph using pretrained text embeddings, facet-weighted cosine similarity, relevance gates, and diversity selection. The vectors are generated locally ahead of time. This document contains the implementation, its provenance, and reproducible diagnostics. No held-out human relevance evaluation has been performed. MODEL & RANKING METADATA { "model": "sentence-transformers/all-MiniLM-L6-v2", "revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41", "modelUrl": "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/tree/1110a243fdf4706b3f48f1d95db1a4f5529b4d41", "dimension": 384, "pooling": "attention-mask mean pooling, then L2 normalization", "precision": "float32 inference; decimal values rounded to 8 places", "device": "cpu", "transformersVersion": "4.57.3", "torchVersion": "2.11.0", "pythonVersion": "3.14.5", "tokenizersVersion": "0.22.2", "safetensorsVersion": "0.8.0", "numpyVersion": "2.4.2", "maxSequenceLength": 256, "longestInputTokens": 79, "contentSha256": "bf4d9278d9f8596a62e09fd00f3243b1f2aa4fd28973165ffeaba850414b6f71", "nodeCount": 16, "vectorCount": 64, "facets": [ "question", "method", "constraint" ], "baseline": "One embedding of question, method and constraint joined by newlines", "modelName": "MiniLM-L6-v2", "license": "Apache-2.0", "algorithm": "Facet-weighted cosine with relevance-gated diversity selection", "weights": { "balanced": { "question": 0.4, "method": 0.4, "constraint": 0.2 }, "questions": { "question": 0.7, "method": 0.2, "constraint": 0.1 }, "methods": { "question": 0.2, "method": 0.7, "constraint": 0.1 } }, "minimumRelevance": 0.22, "minimumFacetSimilarity": 0.26, "diversityPenalty": 0.18, "baselineName": "Whole-description cosine", "layout": "Editorial positions; the algorithm selects connections, not coordinates", "qualityClaim": "Design heuristic; no held-out human relevance evaluation has been performed" } ──────────────────────────────────────────────────────────────────────── GUIDE docs/atlas-algorithm.md SHA-256: ef8d97cb599a51b86e15f3ecdd28086f0816079406b253887d0ff0e2f98d062b # How the map finds connections The map compares the questions, methods, and constraints behind each entry. It uses a pretrained text encoder and a small custom ranking layer. The positions and branch labels are editorial; the selected connections are computed. The embedding model, cosine similarity, and diversity reranking are established techniques, not inventions claimed by this project. ## The input Each of the 16 entries in `lib/atlas-content.json` has three short, curated fields: - **Question:** what is the work trying to understand or accomplish? - **Method:** how does it approach that question? - **Constraint:** what limits the work or makes it difficult? This separates the reason two entries might be related. A single paragraph can mix subject matter, technique, and context; separate vectors let a visitor emphasize a shared question or a shared method. The descriptions themselves are editorial choices and affect the outcome. ## Real embeddings, generated locally The generator uses [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2), an Apache-2.0 pretrained model. It creates 384-dimensional vectors with attention-mask mean pooling and L2 normalization, following the model's documented pooling recipe. Model files are pinned to revision `1110a243fdf4706b3f48f1d95db1a4f5529b4d41`. There are **64 vectors**: three facet vectors plus one whole-description baseline vector per entry. The whole description joins the question, method, and constraint with newlines. Generation runs on the local CPU; it downloads public model files but sends no portfolio descriptions to a model service. The website loads the checked-in vectors and makes no inference requests. Every vector is stored with eight decimal places. The artifact records the model revision, library versions, input file SHA-256, and a SHA-256 for each node's ID and facet text. Runtime normalization removes rounding drift. Inputs longer than 256 tokens fail generation rather than being silently truncated; the current longest input is 79 tokens. ## Scoring a pair For unit vectors, a dot product gives cosine similarity. The system computes this separately for the question (`q`), method (`m`), and constraint (`c`), then combines them: ```text balanced: relevance(a,b) = .40 cos(q_a,q_b) + .40 cos(m_a,m_b) + .20 cos(c_a,c_b) questions: relevance(a,b) = .70 cos(q_a,q_b) + .20 cos(m_a,m_b) + .10 cos(c_a,c_b) methods: relevance(a,b) = .20 cos(q_a,q_b) + .70 cos(m_a,m_b) + .10 cos(c_a,c_b) ``` Weights are declared design choices, not learned parameters. Scores are similarities, **not confidence percentages**. The explanatory facet is the field with the largest weighted contribution. Its source and target descriptions are shown as evidence for a visitor to inspect; they are not a causal explanation extracted from the neural network. Invalid, zero-length, mismatched, or nonfinite vectors cannot create a connection. If a pair lacks one field, the remaining field weights are renormalized. At least two valid shared fields are required. Self-links are excluded. ## Choosing two useful neighbors Candidates must pass both gates: ```text weighted relevance >= .22 at least one facet cosine >= .26 ``` These thresholds were calibrated by inspecting this small collection. They have not been validated as general relevance thresholds. A high semantic score still does not establish a real-world relationship. Among eligible candidates, the first neighbor has the highest relevance. Each subsequent choice balances relevance with redundancy: ```text next = argmax(candidate in eligible, unselected) relevance(source, candidate) - .18 * max(0, relevance(candidate, each already-selected neighbor)) ``` This is an application of [maximal marginal relevance / diversity reranking](https://doi.org/10.1145/290941.291025). Its purpose is to avoid spending both links on almost identical tangents. It cannot promote a candidate through the relevance gate. There is no cross-domain novelty bonus and no forced spanning tree. The map returns fewer than two links when the evidence is weak; restoration currently has no suggested connection in balanced mode. Ties resolve by score and then stable node ID. Pair scores are cached. With this small collection, exact comparisons are simpler than approximate nearest-neighbor infrastructure. Computing the cosine table is O(N² × F × D), with N entries, F = 3 fields, and D = 384 dimensions; greedy selection operates on the resulting scalar scores. Per-node selections are directed: A can recommend B without B spending one of its two recommendations on A. ## What was actually measured Run `node scripts/evaluate-atlas.mjs` to reproduce `docs/atlas-evaluation.json`. For the current 16 entries / 120 unordered pairs, at a two-neighbor limit: | Diagnostic | Observed result | | --- | --- | | Balanced directed connections | 28 | | Cross-branch connections among those | 6 | | Entries with no balanced suggestions | Restoration | | Whole-description cosine baseline connections | 32 | | Baseline connections retained by the balanced system | 21 | | Sources whose choices change due to diversity reranking | USACO, Diagnostics | | Sources whose ranking changes in questions mode | 7 | | Sources whose ranking changes in methods mode | 8 | The baseline simply selects the top two whole-description cosine matches; it has no gate or diversity step. These results describe how the mechanisms differ, **not which one is better**. The recorded rankings show exactly which suggestions change under diversity reranking for USACO and Diagnostics. Those are inspectable effects, not measured accuracy improvements. There is no held-out human relevance evaluation yet. Before claiming improved recommendation quality, collect pairwise relevance judgments independently, reserve some pairs or nodes before tuning, compare accepted links and redundancy against the baseline, and disclose the small sample. It would be misleading to present training-set editorial preferences as a general benchmark. ## Reproduction Generation dependencies and model cache belong outside the website directory. A generation environment can be created with: ```sh python3 -m venv ../atlas-model-env ../atlas-model-env/bin/python -m pip install -r scripts/atlas-model-requirements.txt ../atlas-model-env/bin/python scripts/generate-atlas-embeddings.py \ --revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41 \ --cache-dir ../atlas-model-cache ``` After the first download, add `--offline` to use only cached files. `--output` can point at a temporary artifact for comparison. Offline regeneration in the recorded local environment produced the exact same artifact bytes. Other hardware or library versions can produce slightly different floating-point output, so the checked-in artifact is the canonical runtime input. Verification on Node 25: ```sh node --test tests/atlas-graph.test.mjs node scripts/evaluate-atlas.mjs node scripts/export-atlas-source.mjs ``` Tests check source hashes and model provenance, normalized finite vectors, deterministic rankings, self exclusion, limit handling, relevance gates, invalid/missing vectors, mode changes, diversity behavior, tie breaking, and the independent baseline. These are numerical and integrity checks, not substitutes for human relevance judgments. The source export uses an explicit file allowlist and records each included file's hash. The `/fellowship/algorithm` page displays this checked-in bundle as escaped text, with a raw text version at `/fellowship/algorithm/source`, so production does not depend on reading the source filesystem. It includes the guide, ranker, generator, evaluation script, metadata, and recorded diagnostics; model weights, installed packages, email records, and workspace files are excluded. ## Scope The site intentionally uses a fixed, inspectable set of entries. It does not accept arbitrary visitor nodes. That avoids collecting visitor text and makes every displayed comparison reproducible. Nothing here trains a new embedding model or makes claims about the value, quality, or publication status of a project; those remain separately sourced editorial facts. ──────────────────────────────────────────────────────────────────────── RANKING IMPLEMENTATION lib/atlas-graph.ts SHA-256: 7ddb43c075c6e44cea98a22c8050ae20a63ec3957330cadbc0ca0bb6b4bb7484 import content from './atlas-content.json' with { type: 'json' }; import embeddingArtifact from './atlas-embeddings.json' with { type: 'json' }; export type AtlasMode = 'balanced' | 'questions' | 'methods'; export type AtlasFacet = 'question' | 'method' | 'constraint'; type FacetScores = Record; type FacetVectors = Partial>; export interface GraphNodeInput { id: string; title: string; branch: string; facets: Partial>; } export interface EmbeddingNode { id: string; textSha256?: string; facets: FacetVectors; whole?: readonly number[]; } export interface AtlasPairScore { score: number; facets: FacetScores; sharedFacet: AtlasFacet; reason: string; evidence: { source: string; target: string }; eligible: boolean; } export interface AtlasConnection extends AtlasPairScore { id: string; } const FACETS: readonly AtlasFacet[] = ['question', 'method', 'constraint']; export const ATLAS_WEIGHTS: Record> = { balanced: { question: 0.4, method: 0.4, constraint: 0.2 }, questions: { question: 0.7, method: 0.2, constraint: 0.1 }, methods: { question: 0.2, method: 0.7, constraint: 0.1 }, }; const DEFAULTS = { dimension: 384, minimumRelevance: 0.22, minimumFacetSimilarity: 0.26, diversityPenalty: 0.18, }; function normalize(values: readonly number[] | undefined, dimension: number): number[] | null { if (!values || values.length !== dimension || !values.every(Number.isFinite)) return null; const norm = Math.hypot(...values); if (!Number.isFinite(norm) || norm < 1e-12) return null; return values.map(value => value / norm); } function cosine(a: readonly number[] | null | undefined, b: readonly number[] | null | undefined): number | null { if (!a || !b || a.length !== b.length) return null; // Clamp numerical roundoff; these are cosine values, not confidence probabilities. return Math.max(-1, Math.min(1, a.reduce((sum, value, index) => sum + value * b[index], 0))); } function compareIds(a: string, b: string): number { return a < b ? -1 : a > b ? 1 : 0; } function validLimit(limit: number, available: number): number { return Number.isFinite(limit) ? Math.min(available, Math.max(0, Math.floor(limit))) : 0; } /** Pure graph constructor also permits small synthetic fixtures in numerical tests. */ export function createAtlasGraph( nodes: readonly GraphNodeInput[], embeddings: readonly EmbeddingNode[], options: Partial = {}, ) { const config = { ...DEFAULTS, ...options }; const nodeMap = new Map(nodes.map(node => [node.id, node])); if (nodeMap.size !== nodes.length) throw new Error('Atlas node IDs must be unique'); const vectors = new Map(embeddings.map(record => [record.id, { facets: Object.fromEntries(FACETS.map(facet => [facet, normalize(record.facets[facet], config.dimension)])) as Record, whole: normalize(record.whole, config.dimension), }])); if (vectors.size !== embeddings.length) throw new Error('Atlas embedding IDs must be unique'); const ids = [...nodeMap.keys()].sort(compareIds); const pairCache = new Map(); function getPairScore(aId: string, bId: string, mode: AtlasMode = 'balanced'): AtlasPairScore | null { if (aId === bId) return null; const a = nodeMap.get(aId); const b = nodeMap.get(bId); const aVectors = vectors.get(aId); const bVectors = vectors.get(bId); if (!a || !b || !aVectors || !bVectors) return null; const cacheKey = JSON.stringify([aId, bId, mode]); const cached = pairCache.get(cacheKey); if (cached) return cached; const weights = ATLAS_WEIGHTS[mode] ?? ATLAS_WEIGHTS.balanced; const facets = Object.fromEntries(FACETS.map(facet => [facet, a.facets[facet]?.trim() && b.facets[facet]?.trim() ? cosine(aVectors.facets[facet], bVectors.facets[facet]) : null, ])) as FacetScores; const available = FACETS.filter(facet => facets[facet] !== null); // One accidental matching field cannot stand in for a well-described relationship. if (available.length < 2) return null; const weightSum = available.reduce((sum, facet) => sum + weights[facet], 0); const score = available.reduce((sum, facet) => sum + weights[facet] * (facets[facet] ?? 0), 0) / weightSum; const sharedFacet = [...available].sort((aFacet, bFacet) => weights[bFacet] * (facets[bFacet] ?? 0) - weights[aFacet] * (facets[aFacet] ?? 0) || FACETS.indexOf(aFacet) - FACETS.indexOf(bFacet), )[0]; const reason = { question: 'Connected through the questions these entries explore.', method: 'Connected through the methods these entries describe.', constraint: 'Connected through the constraints these entries work within.', }[sharedFacet]; const result = { score, facets, sharedFacet, reason, evidence: { source: a.facets[sharedFacet] ?? '', target: b.facets[sharedFacet] ?? '' }, eligible: score >= config.minimumRelevance && Math.max(...available.map(facet => facets[facet] ?? -1)) >= config.minimumFacetSimilarity, }; pairCache.set(cacheKey, result); return result; } function getConnections(sourceId: string, mode: AtlasMode = 'balanced', limit = 2): AtlasConnection[] { const count = validLimit(limit, Math.max(0, nodes.length - 1)); if (!nodeMap.has(sourceId) || !count) return []; const candidates = ids.flatMap(id => { const result = getPairScore(sourceId, id, mode); return result?.eligible ? [{ id, ...result }] : []; }); const selected: AtlasConnection[] = []; // MMR-style selection diversifies only candidates already through the relevance gate. while (selected.length < count && candidates.length) { const objective = (candidate: AtlasConnection) => { const redundancy = selected.reduce((max, previous) => Math.max(max, getPairScore(candidate.id, previous.id, mode)?.score ?? 0), 0); return candidate.score - config.diversityPenalty * redundancy; }; const ranked = candidates.map(candidate => ({ candidate, value: objective(candidate) })) .sort((a, b) => b.value - a.value || b.candidate.score - a.candidate.score || compareIds(a.candidate.id, b.candidate.id)); const next = ranked[0].candidate; selected.push(next); candidates.splice(candidates.indexOf(next), 1); } return selected; } function getBaselineConnections(sourceId: string, limit = 2): { id: string; score: number }[] { if (!nodeMap.has(sourceId)) return []; const sourceVector = vectors.get(sourceId)?.whole; return ids.filter(id => id !== sourceId).flatMap(id => { const score = cosine(sourceVector, vectors.get(id)?.whole); return score === null ? [] : [{ id, score }]; }).sort((a, b) => b.score - a.score || compareIds(a.id, b.id)) .slice(0, validLimit(limit, Math.max(0, nodes.length - 1))); } return { getPairScore, getConnections, getBaselineConnections }; } const graph = createAtlasGraph(content, embeddingArtifact.nodes); export const getPairScore = graph.getPairScore; export const getConnections = graph.getConnections; export const getBaselineConnections = graph.getBaselineConnections; export const graphMetadata = { ...embeddingArtifact.metadata, modelName: 'MiniLM-L6-v2', license: 'Apache-2.0', algorithm: 'Facet-weighted cosine with relevance-gated diversity selection', weights: ATLAS_WEIGHTS, minimumRelevance: DEFAULTS.minimumRelevance, minimumFacetSimilarity: DEFAULTS.minimumFacetSimilarity, diversityPenalty: DEFAULTS.diversityPenalty, baselineName: 'Whole-description cosine', layout: 'Editorial positions; the algorithm selects connections, not coordinates', qualityClaim: 'Design heuristic; no held-out human relevance evaluation has been performed', }; ──────────────────────────────────────────────────────────────────────── LOCAL EMBEDDING GENERATOR scripts/generate-atlas-embeddings.py SHA-256: 3c03b72056be73d91cfe01208255a712c45565f39e55c94c0e01b874fcac172d #!/usr/bin/env python3 """Generate checked-in local embeddings; no portfolio text leaves this process. Install transformers==4.57.3 and torch in an isolated environment outside the site. The first run downloads public model files; subsequent runs can use --offline. """ import argparse import hashlib import importlib.metadata import json import platform from pathlib import Path import torch from transformers import AutoModel, AutoTokenizer MODEL = "sentence-transformers/all-MiniLM-L6-v2" FACETS = ("question", "method", "constraint") ROOT = Path(__file__).resolve().parents[1] def digest(value: bytes) -> str: return hashlib.sha256(value).hexdigest() def node_hash(node: dict) -> str: return digest("\0".join([node["id"], *[node["facets"].get(f, "").strip() for f in FACETS]]).encode()) def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--revision", required=True, help="Immutable 40-character Hugging Face model commit") parser.add_argument("--cache-dir", required=True, type=Path, help="Model cache outside the site bundle") parser.add_argument("--offline", action="store_true") parser.add_argument("--output", type=Path, default=ROOT / "lib/atlas-embeddings.json") args = parser.parse_args() if len(args.revision) != 40 or any(c not in "0123456789abcdef" for c in args.revision): parser.error("--revision must be an immutable lowercase Git commit hash") source_path = ROOT / "lib/atlas-content.json" source = source_path.read_bytes() nodes = json.loads(source) if len({node["id"] for node in nodes}) != len(nodes): raise ValueError("Duplicate node IDs") kwargs = dict(revision=args.revision, cache_dir=str(args.cache_dir), local_files_only=args.offline, trust_remote_code=False) tokenizer = AutoTokenizer.from_pretrained(MODEL, **kwargs) model = AutoModel.from_pretrained(MODEL, use_safetensors=True, **kwargs).to("cpu").eval() torch.set_num_threads(1) torch.manual_seed(0) torch.use_deterministic_algorithms(True) records, texts = [], [] for node in nodes: facets = {facet: node["facets"].get(facet, "").strip() for facet in FACETS} if not all(facets.values()): raise ValueError(f"Missing facet in {node['id']}; do not silently encode an empty sentence") record = {"id": node["id"], "textSha256": node_hash(node), "facets": {}} for facet, text in facets.items(): texts.append((record, facet, text)) texts.append((record, "whole", "\n".join(facets.values()))) records.append(record) max_tokens = 0 # Encode each input separately so changing batch composition cannot alter rounding. for record, facet, text in texts: encoded = tokenizer(text, return_tensors="pt", truncation=False) length = encoded["input_ids"].shape[1] max_tokens = max(max_tokens, length) if length > 256: raise ValueError(f"{record['id']}/{facet} has {length} tokens; shorten it instead of silently truncating") with torch.inference_mode(): states = model(**encoded).last_hidden_state mask = encoded["attention_mask"].unsqueeze(-1).expand(states.size()).float() pooled = (states * mask).sum(1) / mask.sum(1).clamp(min=1e-9) vector = torch.nn.functional.normalize(pooled, p=2, dim=1)[0] values = [round(float(value), 8) for value in vector] if facet == "whole": record["whole"] = values else: record["facets"][facet] = values output = { "schemaVersion": 1, "metadata": { "model": MODEL, "revision": args.revision, "modelUrl": f"https://huggingface.co/{MODEL}/tree/{args.revision}", "dimension": 384, "pooling": "attention-mask mean pooling, then L2 normalization", "precision": "float32 inference; decimal values rounded to 8 places", "device": "cpu", "transformersVersion": importlib.metadata.version("transformers"), "torchVersion": torch.__version__, "pythonVersion": platform.python_version(), "tokenizersVersion": importlib.metadata.version("tokenizers"), "safetensorsVersion": importlib.metadata.version("safetensors"), "numpyVersion": importlib.metadata.version("numpy"), "maxSequenceLength": 256, "longestInputTokens": max_tokens, "contentSha256": digest(source), "nodeCount": len(nodes), "vectorCount": len(texts), "facets": list(FACETS), "baseline": "One embedding of question, method and constraint joined by newlines", }, "nodes": records, } args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(output, ensure_ascii=False, separators=(",", ":")) + "\n") print(json.dumps({"output": str(args.output), "nodeCount": len(nodes), "vectorCount": len(texts), "maxTokens": max_tokens, "sha256": digest(args.output.read_bytes())})) if __name__ == "__main__": main() ──────────────────────────────────────────────────────────────────────── EVALUATION SCRIPT scripts/evaluate-atlas.mjs SHA-256: c15fe1a6bc448304eb4753e8688b8d16fb1fd69c0128e4e0192188a4ddee8ef2 #!/usr/bin/env node // Structural diagnostics, not a relevance/accuracy benchmark: no human labels exist yet. import { writeFile, mkdir, readFile } from 'node:fs/promises'; import { createHash } from 'node:crypto'; import content from '../lib/atlas-content.json' with { type: 'json' }; import artifact from '../lib/atlas-embeddings.json' with { type: 'json' }; import { createAtlasGraph, getConnections, getBaselineConnections, graphMetadata } from '../lib/atlas-graph.ts'; const modes = ['balanced', 'questions', 'methods']; const noDiversity = createAtlasGraph(content, artifact.nodes, { diversityPenalty: 0 }); const round = number => Number(number.toFixed(6)); const pack = edge => ({ id: edge.id, score: round(edge.score) }); const nodeMap = new Map(content.map(node => [node.id, node])); const rows = content.map(node => ({ id: node.id, baseline: getBaselineConnections(node.id).map(pack), modes: Object.fromEntries(modes.map(mode => [mode, getConnections(node.id, mode).map(pack)])), relevanceWithoutDiversity: noDiversity.getConnections(node.id).map(pack), })); const selected = rows.flatMap(row => row.modes.balanced.map(edge => ({ source: row.id, ...edge }))); const retained = rows.reduce((sum, row) => sum + row.modes.balanced.filter(edge => row.baseline.some(item => item.id === edge.id)).length, 0); const report = { model: graphMetadata.model, revision: graphMetadata.revision, contentSha256: graphMetadata.contentSha256, artifactSha256: createHash('sha256').update(await readFile(new URL('../lib/atlas-embeddings.json', import.meta.url))).digest('hex'), interpretation: 'These are reproducible graph diagnostics, not accuracy, superiority, or held-out relevance results. Baseline returns two neighbors without a gate; the proposed ranking may return fewer.', humanRelevanceEvaluation: 'Not performed', configuration: { limit: 2, weights: graphMetadata.weights, minimumRelevance: graphMetadata.minimumRelevance, minimumFacetSimilarity: graphMetadata.minimumFacetSimilarity, diversityPenalty: graphMetadata.diversityPenalty, }, diagnostics: { nodes: content.length, unorderedPairs: content.length * (content.length - 1) / 2, balancedDirectedConnections: selected.length, balancedNodesWithNoConnections: rows.filter(row => !row.modes.balanced.length).map(row => row.id), balancedCrossBranchConnections: selected.filter(edge => nodeMap.get(edge.source).branch !== nodeMap.get(edge.id).branch).length, wholeDescriptionBaseline: { directedConnections: rows.reduce((sum, row) => sum + row.baseline.length, 0), retainedByBalanced: retained }, sourcesChangedByDiversity: rows.filter(row => JSON.stringify(row.modes.balanced.map(edge => edge.id)) !== JSON.stringify(row.relevanceWithoutDiversity.map(edge => edge.id))).map(row => row.id), sourcesChangedInQuestionsMode: rows.filter(row => JSON.stringify(row.modes.balanced.map(edge => edge.id)) !== JSON.stringify(row.modes.questions.map(edge => edge.id))).map(row => row.id), sourcesChangedInMethodsMode: rows.filter(row => JSON.stringify(row.modes.balanced.map(edge => edge.id)) !== JSON.stringify(row.modes.methods.map(edge => edge.id))).map(row => row.id), }, rankings: rows, }; const directory = new URL('../docs/', import.meta.url); await mkdir(directory, { recursive: true }); await writeFile(new URL('atlas-evaluation.json', directory), JSON.stringify(report, null, 2) + '\n'); console.log(JSON.stringify(report.diagnostics, null, 2)); ──────────────────────────────────────────────────────────────────────── RECORDED EVALUATION docs/atlas-evaluation.json SHA-256: 89aa82d3fc0d6f588986ca5e4c44858dc830dff1b4e9e4df782aa855f9e4bc1b { "model": "sentence-transformers/all-MiniLM-L6-v2", "revision": "1110a243fdf4706b3f48f1d95db1a4f5529b4d41", "contentSha256": "bf4d9278d9f8596a62e09fd00f3243b1f2aa4fd28973165ffeaba850414b6f71", "artifactSha256": "120c2967647171cdd39c01c385da266be590f5ad4995c080b0e788b852a06e11", "interpretation": "These are reproducible graph diagnostics, not accuracy, superiority, or held-out relevance results. Baseline returns two neighbors without a gate; the proposed ranking may return fewer.", "humanRelevanceEvaluation": "Not performed", "configuration": { "limit": 2, "weights": { "balanced": { "question": 0.4, "method": 0.4, "constraint": 0.2 }, "questions": { "question": 0.7, "method": 0.2, "constraint": 0.1 }, "methods": { "question": 0.2, "method": 0.7, "constraint": 0.1 } }, "minimumRelevance": 0.22, "minimumFacetSimilarity": 0.26, "diversityPenalty": 0.18 }, "diagnostics": { "nodes": 16, "unorderedPairs": 120, "balancedDirectedConnections": 28, "balancedNodesWithNoConnections": [ "restoration" ], "balancedCrossBranchConnections": 6, "wholeDescriptionBaseline": { "directedConnections": 32, "retainedByBalanced": 21 }, "sourcesChangedByDiversity": [ "usaco", "diagnostics" ], "sourcesChangedInQuestionsMode": [ "usaco", "usapho", "rook", "jaxphys", "diagnostics", "infiltr", "writing" ], "sourcesChangedInMethodsMode": [ "usapho", "sage", "structure", "diagnostics", "infiltr", "alphasig", "greensight", "tie-cyber" ] }, "rankings": [ { "id": "usaco", "baseline": [ { "id": "jaxphys", "score": 0.328909 }, { "id": "usapho", "score": 0.279926 } ], "modes": { "balanced": [ { "id": "usapho", "score": 0.308474 }, { "id": "rook", "score": 0.241823 } ], "questions": [ { "id": "jaxphys", "score": 0.318755 }, { "id": "usapho", "score": 0.282514 } ], "methods": [ { "id": "usapho", "score": 0.354089 }, { "id": "rook", "score": 0.256194 } ] }, "relevanceWithoutDiversity": [ { "id": "usapho", "score": 0.308474 }, { "id": "jaxphys", "score": 0.283366 } ] }, { "id": "usapho", "baseline": [ { "id": "sage", "score": 0.407002 }, { "id": "jaxphys", "score": 0.380258 } ], "modes": { "balanced": [ { "id": "sage", "score": 0.417871 }, { "id": "structure", "score": 0.36169 } ], "questions": [ { "id": "structure", "score": 0.386219 }, { "id": "sage", "score": 0.384247 } ], "methods": [ { "id": "sage", "score": 0.452286 }, { "id": "usaco", "score": 0.354089 } ] }, "relevanceWithoutDiversity": [ { "id": "sage", "score": 0.417871 }, { "id": "structure", "score": 0.36169 } ] }, { "id": "rook", "baseline": [ { "id": "sage", "score": 0.305069 }, { "id": "usaco", "score": 0.243426 } ], "modes": { "balanced": [ { "id": "sage", "score": 0.258193 }, { "id": "usaco", "score": 0.241823 } ], "questions": [ { "id": "sage", "score": 0.268351 } ], "methods": [ { "id": "sage", "score": 0.263541 }, { "id": "usaco", "score": 0.256194 } ] }, "relevanceWithoutDiversity": [ { "id": "sage", "score": 0.258193 }, { "id": "usaco", "score": 0.241823 } ] }, { "id": "sage", "baseline": [ { "id": "usapho", "score": 0.407002 }, { "id": "jaxphys", "score": 0.349979 } ], "modes": { "balanced": [ { "id": "usapho", "score": 0.417871 }, { "id": "structure", "score": 0.332386 } ], "questions": [ { "id": "usapho", "score": 0.384247 }, { "id": "structure", "score": 0.377223 } ], "methods": [ { "id": "usapho", "score": 0.452286 }, { "id": "jaxphys", "score": 0.317894 } ] }, "relevanceWithoutDiversity": [ { "id": "usapho", "score": 0.417871 }, { "id": "structure", "score": 0.332386 } ] }, { "id": "structure", "baseline": [ { "id": "usapho", "score": 0.363995 }, { "id": "sage", "score": 0.342338 } ], "modes": { "balanced": [ { "id": "usapho", "score": 0.36169 }, { "id": "sage", "score": 0.332386 } ], "questions": [ { "id": "usapho", "score": 0.386219 }, { "id": "sage", "score": 0.377223 } ], "methods": [ { "id": "usapho", "score": 0.328819 }, { "id": "diagnostics", "score": 0.258435 } ] }, "relevanceWithoutDiversity": [ { "id": "usapho", "score": 0.36169 }, { "id": "sage", "score": 0.332386 } ] }, { "id": "jaxphys", "baseline": [ { "id": "usapho", "score": 0.380258 }, { "id": "sage", "score": 0.349979 } ], "modes": { "balanced": [ { "id": "usapho", "score": 0.354022 }, { "id": "sage", "score": 0.312832 } ], "questions": [ { "id": "usapho", "score": 0.357199 }, { "id": "usaco", "score": 0.318755 } ], "methods": [ { "id": "usapho", "score": 0.356701 }, { "id": "sage", "score": 0.317894 } ] }, "relevanceWithoutDiversity": [ { "id": "usapho", "score": 0.354022 }, { "id": "sage", "score": 0.312832 } ] }, { "id": "diagnostics", "baseline": [ { "id": "sage", "score": 0.301474 }, { "id": "usapho", "score": 0.26854 } ], "modes": { "balanced": [ { "id": "tie-cyber", "score": 0.277067 }, { "id": "usapho", "score": 0.251341 } ], "questions": [ { "id": "usapho", "score": 0.244948 }, { "id": "tie-cyber", "score": 0.227617 } ], "methods": [ { "id": "tie-cyber", "score": 0.320386 }, { "id": "infiltr", "score": 0.315843 } ] }, "relevanceWithoutDiversity": [ { "id": "tie-cyber", "score": 0.277067 }, { "id": "infiltr", "score": 0.256668 } ] }, { "id": "extra", "baseline": [ { "id": "alphasig", "score": 0.219457 }, { "id": "infiltr", "score": 0.170475 } ], "modes": { "balanced": [ { "id": "alphasig", "score": 0.252871 } ], "questions": [ { "id": "alphasig", "score": 0.286931 } ], "methods": [ { "id": "alphasig", "score": 0.239374 } ] }, "relevanceWithoutDiversity": [ { "id": "alphasig", "score": 0.252871 } ] }, { "id": "infiltr", "baseline": [ { "id": "tie-cyber", "score": 0.240259 }, { "id": "jaxphys", "score": 0.229147 } ], "modes": { "balanced": [ { "id": "diagnostics", "score": 0.256668 }, { "id": "tie-cyber", "score": 0.239943 } ], "questions": [ { "id": "diagnostics", "score": 0.226552 } ], "methods": [ { "id": "tie-cyber", "score": 0.319837 }, { "id": "diagnostics", "score": 0.315843 } ] }, "relevanceWithoutDiversity": [ { "id": "diagnostics", "score": 0.256668 }, { "id": "tie-cyber", "score": 0.239943 } ] }, { "id": "alphasig", "baseline": [ { "id": "tie-cyber", "score": 0.320636 }, { "id": "fai", "score": 0.249024 } ], "modes": { "balanced": [ { "id": "tie-cyber", "score": 0.355783 }, { "id": "extra", "score": 0.252871 } ], "questions": [ { "id": "tie-cyber", "score": 0.354192 }, { "id": "extra", "score": 0.286931 } ], "methods": [ { "id": "tie-cyber", "score": 0.391856 }, { "id": "writing", "score": 0.33624 } ] }, "relevanceWithoutDiversity": [ { "id": "tie-cyber", "score": 0.355783 }, { "id": "extra", "score": 0.252871 } ] }, { "id": "code4cause", "baseline": [ { "id": "fai", "score": 0.377433 }, { "id": "greensight", "score": 0.317593 } ], "modes": { "balanced": [ { "id": "fai", "score": 0.391042 }, { "id": "greensight", "score": 0.294984 } ], "questions": [ { "id": "fai", "score": 0.387792 }, { "id": "greensight", "score": 0.25864 } ], "methods": [ { "id": "fai", "score": 0.408973 }, { "id": "greensight", "score": 0.332339 } ] }, "relevanceWithoutDiversity": [ { "id": "fai", "score": 0.391042 }, { "id": "greensight", "score": 0.294984 } ] }, { "id": "fai", "baseline": [ { "id": "code4cause", "score": 0.377433 }, { "id": "greensight", "score": 0.281686 } ], "modes": { "balanced": [ { "id": "code4cause", "score": 0.391042 }, { "id": "greensight", "score": 0.229118 } ], "questions": [ { "id": "code4cause", "score": 0.387792 }, { "id": "greensight", "score": 0.232473 } ], "methods": [ { "id": "code4cause", "score": 0.408973 }, { "id": "greensight", "score": 0.255649 } ] }, "relevanceWithoutDiversity": [ { "id": "code4cause", "score": 0.391042 }, { "id": "greensight", "score": 0.229118 } ] }, { "id": "greensight", "baseline": [ { "id": "code4cause", "score": 0.317593 }, { "id": "fai", "score": 0.281686 } ], "modes": { "balanced": [ { "id": "code4cause", "score": 0.294984 }, { "id": "fai", "score": 0.229118 } ], "questions": [ { "id": "code4cause", "score": 0.25864 }, { "id": "fai", "score": 0.232473 } ], "methods": [ { "id": "code4cause", "score": 0.332339 }, { "id": "usaco", "score": 0.235428 } ] }, "relevanceWithoutDiversity": [ { "id": "code4cause", "score": 0.294984 }, { "id": "fai", "score": 0.229118 } ] }, { "id": "restoration", "baseline": [ { "id": "fai", "score": 0.187212 }, { "id": "usaco", "score": 0.157191 } ], "modes": { "balanced": [], "questions": [], "methods": [] }, "relevanceWithoutDiversity": [] }, { "id": "writing", "baseline": [ { "id": "alphasig", "score": 0.19753 }, { "id": "usapho", "score": 0.135653 } ], "modes": { "balanced": [ { "id": "alphasig", "score": 0.244178 } ], "questions": [], "methods": [ { "id": "alphasig", "score": 0.33624 } ] }, "relevanceWithoutDiversity": [ { "id": "alphasig", "score": 0.244178 } ] }, { "id": "tie-cyber", "baseline": [ { "id": "alphasig", "score": 0.320636 }, { "id": "infiltr", "score": 0.240259 } ], "modes": { "balanced": [ { "id": "alphasig", "score": 0.355783 }, { "id": "diagnostics", "score": 0.277067 } ], "questions": [ { "id": "alphasig", "score": 0.354192 }, { "id": "diagnostics", "score": 0.227617 } ], "methods": [ { "id": "alphasig", "score": 0.391856 }, { "id": "infiltr", "score": 0.319837 } ] }, "relevanceWithoutDiversity": [ { "id": "alphasig", "score": 0.355783 }, { "id": "diagnostics", "score": 0.277067 } ] } ] }