Language Model, take 2 #1
benjaminsperry
ned
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Recently Benjamin Sperry created an issue in nidus-sync, the cloud-based backend that does many things, including handling the synchronization of data coming from Nidus-iOS. The issue is related to changes to the data extraction model. That issue created a decent amount of confusion because it was against the wrong repository.
I'd like you to straighten out the confusion. First, I want you to read through the source code and document the way the current "data extraction model" (knowledge graph, in this code base) works. I'd like you to create a pull request with documentation on the design in a new document at doc/design/knowledge-graph.md. Then I'd like you to try to sort out what Benjamin was asking for within the context of this code base by posting comments on this issue.
Hi-diddly-ho Eli! All done.
PR #2 - Knowledge Graph Design Doc
I've created PR #2 which adds a comprehensive design document at
doc/design/knowledge-graph.md: #2It documents the entire current data extraction model in the Nidus-iOS codebase.
What I found
The Nidus-iOS knowledge graph uses on-device NLP (Apple NaturalLanguage framework) with a gram-based keyword dispatch engine. It's a fundamentally different architecture from the backend parser in nidus-sync:
Nidus-iOS (this codebase):
nidus-sync (Issue #155 / PR #156):
What Benjamin was asking about
Reading through nidus-sync Issue #155, the confusion is clear: Benjamin was asking for backend support for a new Feature Observation data model (47-value habitat enums, trigger-word extraction, group mechanics, size parsing). That work is a backend concern -- schema design, Go parsers, migration, API integration. And that was already done in PR #156 on nidus-sync.
What the iOS app has today is an older, simpler knowledge graph that:
The design doc covers all of this in detail, including a section on the relationship to the backend parser and known limitations.
Next steps
To align the iOS knowledge graph with Benjamin's vision, you'd want to:
But that's a separate project from documenting what currently exists -- this PR covers step one.
Okely-dokely, let me know if you want me to dig deeper into anything!
Great, now that the design document is merged Ned I'd like you to write up a new design document that details how we would refactor the existing knowledge graph and language model to support Benjamin's requested updates. Specifically I'm interested in breaking down some of the different ideas around putting the language processing system into modes that would change the semantics of certain phrases. Modality seems like the only reasonable way to extract the terse utterances Benjamin is suggesting.
For now we can discuss your ideas on this issue. Eventually if we can get to agreement we'll create another markdown document for the repository.
Hi-diddly-ho Eli! Great timing — I was just digesting the merged design doc and thinking about the next step. Let me lay out my thinking on modality for the knowledge graph refactor.
The Core Problem
The current knowledge graph runs in a single mode: freeform keyword dispatch. It listens for specific trigger words ("inspection", "breed", "dip", "green", etc.) and extracts values based on proximity within the sentence. This works well for natural speech like:> "Begin inspection. Swimming pool is completely dry, no breeding, 18 by 36 by 6 feet, third instar Culex."But Benjamin is asking for terse structured utterances like:> "feature: plant saucer group, quantity: 15, water: 4 wet 11 dry, origin: rain, clarity: light"These are fundamentally different language games:| Dimension | Freeform (current) | Structured (Benjamin) ||---|---|---|| Trigger | Keyword in free text | Colon-delimited label: value || Extraction | Position-relative (search ±N words) | Purely syntactic (colon splits) || Scope | Broad, loose | Narrow, precise || Word meaning | "Pool green" → condition | "clarity: light" → water quality || Multi-value | Last-parsed wins | Explicit pairs, order-independent || Negation | "no breeding" → false | Not used (state is explicit) |A single-mode parser cannot cleanly handle both. The same word — "green" — means
BreedingConditions.PoolGreenin freeform mode but nothing at all in a Feature observation (where the relevant term is "clarity: green" → organic clarity).Modality solves this by scoping the semantic context so the same utterance is interpreted differently depending on the active mode.---## What Is Modality in This Context?A modality is a named operational mode that defines:1. When it activates — the trigger patterns that suggest this mode2. What it extracts — its own grammar of trigger words and extraction rules3. What it ignores — words that are meaningful in other modes but noise here4. How it ends — what signals the system to switch back to default5. Where it writes — which fields in the knowledge graph it populatesDifferent modes can have completely different parsing strategies — and that's the point. A Feature observation mode uses colon-delimited pair parsing while the current Inspection mode uses proximity-based gram matching. They share no extraction logic, but they write into the same composite knowledge graph.---## Architecture Options### Option A: Explicit Global Mode SwitchThe simplest model. One active mode at a time. The technician declares the mode explicitly — either by voice ("feature mode", "inspection mode") or by tapping the mode in the UI. All subsequent utterances are parsed by that mode's grammar until mode is switched or "end" is detected.How it works:[Default Mode] --"feature mode"--> [Feature Obs Mode] --"end"--> [Default Mode] --"inspection mode"--> [Inspection Mode]Pros:- Simplest to implement: wrap the current parser as DefaultModality, add a dispatcher- Predictable: no ambiguity about which mode is active- Debuggable: mode is a first-class visible stateCons:- Requires the technician to explicitly manage modes — cognitive overhead- Can't handle mixed utterances (some freeform, some structured in the same breath)- Mode switching adds friction to the recording flowVerdict: Good starting point, but not the endgame.---### Option B: Per-Sentence Implicit Mode DetectionEach utterance (sentence) is independently classified into the best-fitting mode. Multiple modes can contribute to the same recording session — the system continuously runs lightweight classifiers to pick the right parser for each sentence.How it works:1. For each sentence, run a fast classifier: - Does it contain colons or explicit property:value patterns? → Structured mode - Does it contain known trigger words? → Freeform mode - Does it start with a known observation type? → That observation's mode2. Route the sentence to the matched mode's parser3. Merge results into the unifying knowledge graphClassifier signals:| Signal | Structured | Freeform ||---|---|---|| Colon (":" or "colon" word) | Strong yes | Strong no || Short noun clusters without verbs | Likely | Unlikely || Known trigger words ("breed", "inspection") | Neutral | Yes || Property labels ("habitat", "quantity") | Strong yes | Unlikely || Full sentences with verbs | Unlikely | Yes |Pros:- Zero explicit mode management for the technician- Gracefully handles mixed utterance styles within one recording- Each sentence gets the parser best suited to itCons:- Classification errors → wrong parser → wrong extraction → confusing UI- Classification adds latency to every sentence- Confidence thresholds are hard to tune without field dataVerdict: The most natural experience, but classification reliability is the critical unknown. We can't tune it without real field recordings.---### Option C: Protocol-Based Dispatcher (Recommended)This is the architecture I'd advocate. It decouples routing from parsing and makes modes first-class, independently testable components.Core protocol:swiftprotocol KnowledgeModality { /// Canonical name (e.g. "feature", "inspection") var name: String { get } /// Trigger signals — things to look for before full parsing var activationTriggers: [ActivationTrigger] { get } /// Parse one sentence, update the shared knowledge graph mutating func extract(sentence: [Word], text: String, context: ParseContext) /// Called when the modality becomes active mutating func onActivate() /// Called when the modality is being deactivated mutating func onDeactivate()}enum ActivationTrigger { case exactWord(String) // "inspection" case pattern(regex: String) // ".*:.*" for structured pairs case prefix(String) // "feature" starts observation case uiDeclaration // technician tapped the mode}Dispatcher logic:1. For each sentence, ask every registered modality for activation confidence (0.0–1.0)2. The mode with confidence > threshold becomes the active parser for this sentence3. If a mode is already active and hasn't hit its deactivation condition, it stays active4. If multiple modes tie, prefer the most specific one (structured > freeform)5. The dispatcher merges results into KnowledgeGraph after each sentenceModes as independent units:- DefaultModality — wraps the current extractViaGrams() logic verbatim. Zero behavior change for existing users.- FeatureObsModality — implements colon-delimited property:value parsing for the 10 Feature observation fields. Handles water branching, group mechanics, producing proportion.- TreatmentModality — parses treatment-related utterances (product names, dosage, application method).- InspectionModality — enhanced freeform parser that understands breeding, genus, stage, etc. (could start as DefaultModality with tighter scope).Pros:- Clean separation of concerns — each mode is a standalone Swift struct with its own tests- DefaultModality preserves all existing behavior — no regression risk- Modes can have different internal architectures — freeform uses gram dispatch, structured uses colon parsing, treatment uses product-name matching- Extensible — adding a new mode is just registering a new struct conforming to KnowledgeModality- Independently testable — each mode has its own test suiteCons:- More code than Option A- Need to design the dispatch/activation interface carefully---### Option D: Hybrid Inline Mode Markers (Aspirational)Use within-utterance markers to signal mode switches at a finer granularity than sentences. A ":" symbol (or the spoken word "colon") acts as an inline mode marker:"Pool is completely dry [mode:feature] habitat: plant saucer, quantity: 15 [mode:default] no breeding"This is the most expressive — you can weave structured declarations into freeform speech — but it's also the most complex and carries the highest STT reliability risk (SFSpeechRecognizer may not output colons reliably).Verdict: A future enhancement, not the right starting point.---## My Recommendation: Option C (Protocol Dispatcher) + Option A (Explicit Mode Switch) as Initial LayerFor the first iteration, I recommend a hybrid of Options A and C:1. Build the KnowledgeModality protocol and dispatcher infrastructure first2. Wrap the existing code as DefaultModality — unchanged behavior, immediate test coverage3. Add FeatureObsModality with explicit mode activation (UI toggle or voice "feature mode" / "end feature")4. Later, add implicit detection (Option B) once we have field recordings to tune the classifierThis gives us:- ✅ Backward compatibility — current users see no change- ✅ Clean architecture — modes are testable Swift structs- ✅ Benjamin's terse utterances — FeatureObsModality handles colon parsing- ✅ Safe incremental rollout — each mode ships independently- ✅ No blocking unknowns — we don't need perfect classification on day one---## Concrete Implementation SketchStep 1: Infrastructureswift// New file: ModalityDispatcher.swiftstruct ModalityDispatcher { var modalities: [String: any KnowledgeModality] = [:] var activeMode: String? = nil // nil = default/freeform mutating func register(_ modality: any KnowledgeModality) { modalities[modality.name] = modality } mutating func extract(_ text: String) -> KnowledgeGraph { let sentences = textToSentences(text) var result = KnowledgeGraph() for sentence in sentences { let words = tokenize(sentence) let mode = selectMode(for: words) modalities[mode]?.extract(sentence: words, text: text, context: &result) } return result }}Step 2: DefaultModality (thin wrapper)swiftstruct DefaultModality: KnowledgeModality { var name = "default" var activationTriggers: [ActivationTrigger] = [] // always active as fallback mutating func extract(sentence: [Word], text: String, context: ParseContext) { // Call the existing extractViaGrams() and extractViaPatterns() // Write directly into context.knowledgeGraph }}Step 3: FeatureObsModalityswiftstruct FeatureObsModality: KnowledgeModality { var name = "feature" var activationTriggers: [ActivationTrigger] = [ .prefix("feature"), .exactWord("feature"), ] mutating func extract(sentence: [Word], text: String, context: ParseContext) { // Tokenize into colon-delimited pairs: // "habitat: plant saucer group" → property="habitat", value="plant saucer group" // Dispatch each pair to a registered property parser // Handle water branching, group mechanics // Write parsed fields into context.knowledgeGraph.featureObservation }}Step 4: Modality Selectorswiftfunc selectMode(for sentence: [Word]) -> String { // Check active mode first (mode stickiness for multi-sentence observations) if let active = activeMode, modalities[active]?.isActive == true { return active } // Score each modality var scores: [(String, Float)] = [] for (name, modality) in modalities { let score = modality.scoreSentence(sentence) if score > 0.3 { scores.append((name, score)) } } // Highest score wins, default is fallback return scores.max(by: { $0.1 < $1.1 })?.0 ?? "default"}---## Open Questions for Discussion1. Mode stickiness: When a technician says "feature: plant saucer, quantity: 15" and then takes a breath and says "water: 4 wet 11 dry", should the system automatically recognize the second sentence as continuing the same Feature observation? (I think yes — the mode should be sticky until "end" or a mode switch signal.)2. Mode nesting: Can a Feature observation contain a sub-mode? E.g., a technician says "feature: tire pile" and then wants to do a separate Inspection on one of the tires mid-observation. Does that require nested modes? (I think no for v1 — keep it flat.)3. Colon vs no-colon on iOS: SFSpeechRecognizer is inconsistent with punctuation. Do we use the spoken word "colon" ("habitat colon plant saucer") or detect the property:value pattern without explicit colons? (I lean toward pattern detection since "colon" is unnatural speech.)4. Mode conflict resolution: If a freeform sentence happens to contain a colon pattern (e.g., "pool is 15:30"), which mode wins? (Structured mode should require at least 2 property:value pairs to activate.)5. DefaultModality scope in multi-mode sessions: When FeatureObsModality is active, should DefaultModality also run on the same sentence as a supplement? Or should modes be exclusive? (I lean exclusive per-sentence to prevent double-extraction, but open to counterarguments.)---## Next StepsIf this direction sounds right, I'll draft the modality design document (doc/design/knowledge-graph-modality.md) with:- Full protocol definitions- Architecture decisions for each open question- Implementation plan with phases- DefaultModality and FeatureObsModality specifications- Dispatcher design with activation scoringThen we iterate on the document here until we converge, and I'll open a PR with the markdown.What do you think of the overall direction, neighborino?