feat: recruit library agents into project conversations and scan project folders (#206 )

* media: check ffmpeg filter support before subtitle operations

* sandbox: handle synchronous spawn errors safely in desktop sandbox

* Voice-first streaming, non-blocking follow-ups, quiet routines and chat UX

Why: questions blocked the chat, follow-up messages had no acknowledgement, message actions sat beside the bubble, and voice turn-taking added latency.

What: waiting runs accept steering messages and the bot marks each claimed follow-up as seen; unclaimed follow-ups show as queued. Reaction, copy and reply controls sit under each message. Streaming voice (VAD endpointing, barge-in that stops the run, clause-based TTS), an optimistic input queue, procedural earcons and delta-based quiet routines were added. Onboarding can be skipped straight to the main conversation. Dual-surface layout and project pickers are included. Status colors use semantic tokens.

Tested: unit suites for core, adapters, api and web pass, typecheck and Biome are clean. Browser end-to-end tests were not run locally.

* Update translation catalogs

Re-extracted catalogs for the new and moved strings; the catalog sync test passes.

* Add project plan and test readiness notes

Notes from the milestone work: feature inventory, test infrastructure and test readiness. No code changes.

* fix(ci): align scale guards, space deletion, computer toggle, and mobile expo deps

* fix(ci): restore space deletion guard, browser toggle, and run-steps contracts

* fix(ci): restore model resolution errors, group stage, and onboarding flow

* fix(ci): restore in-app-browser close button, disable stop button while sending, and sync lingui catalogs

* fix(ci): address web e2e failures for bot-crud, choice cards, message quote, and voice onboarding

* test(web): wait for avatar and rig visibility in oo-avatar motion spec

* test(web): wait for persisted message IDs in message-hover-actions pagination spec

* feat: recruit library agents into project conversations and scan project folders

* test(web): expect repo description in GithubRepoPicker onChange

* Onboarding: always drop the local voice when the voice step is skipped

Skipping waited only for the latest voice load and disconnected only
when the client saw the connect succeed. An earlier or client-failed
connect could still persist the local credential, so a user who
skipped voice was left with one. Wait for every load and always
disconnect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy1md9QE1ayJbDCwJ4vAQs

* Shell: hide the optimistic copy once the transcript shows the send

A message sent while the bot works streamed into the transcript before
its send request resolved, so the same text showed twice until the
optimistic bubble was removed. Drop an optimistic message as soon as a
persisted user message with the same text appears.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wy1md9QE1ayJbDCwJ4vAQs

---------

Co-authored-by: Claude <noreply@anthropic.com>

2026-10-15 14:44:53 +02:00

9.1 KiB

Raw Blame History

evOo — Product rules (Gesamterfahrung)

Design mantra (apply ruthlessly)

EINFACHHEIT und Durchgängigkeit. Dennoch super flexibel. Weniger ist Mehr.

Rule	Product
One model	Unterhaltungen (1:1 / Gruppe), Ordner/Projekte, specialist Oos
One language	Same stage, sound, and transcript grammar everywhere — no one-off chrome
Hide power	Depth in specialists and settings — not every control on first screen
Cut clutter	Fewer, stronger affordances; stub/TODO lists stay in docs, not UI

When a slice gets feature-heavy: prune to the simplest coherent path that still delivers Stage + sound + calm onboarding.

Conversations, not agent rows

The product is organized as Unterhaltungen (see VOICE.md). Team lists your Oos; opening one starts a conversation, not “the agent app”.

Type	Who
1:1 Unterhaltung	One Oo
Gruppenunterhaltung	Multiple Oos (+ humans later) — fun factor essential (see `docs/GROUP-CONVERSATIONS.md`)
Projekte	One group conversation per project, led by its Projektkoordinator

Projects recruit from the library

Oos are created once, in the home space: that is the library. Each has one job.
A project is a space with a fixed coordinator (Bot.kind="main") and one main group conversation. A linked folder's README or manifest, or the GitHub repo's description, tells the coordinator what the project is (PROJECT_GOAL.md) when the user gave no goal.
The coordinator recruits the Oos a request needs with delegate_task, briefing them on the task card. A recruited Oo gets a project member (spawnKey recruit:<libraryBotId>, packages/db/src/project-recruits.ts) that joins the conversation. The member keeps the Oo's persona and picks up later edits to it. It works with the project's files, memory, computer and connections. The Oo's home memory stays home.

Deepest personalization

evOo must adapt to each user more individually than any other app. Persona/crew fit is lifestyle-agnostic — office worker, club/scene person, hardcore nerd: same mechanism, simplest path to what they want.

Companion tone	When
Geek-peer / Fachsimpeln	User wants depth — match enthusiasm, stay in lane
Quiet-doer	User wants outcomes — never act like we know better

Onboarding harvest infers help & vibe (user-help-vibe.ts → MEMORY.md), then spawn/select specialist Oos and Unterhaltungen — not one personality for everyone. See docs/PERSONA-SELECTION.md.

Accessibility of ambition

Power-user literacy must not be required. Same product, radically different crews/Unterhaltungen — domain excellence in each case.

User	Gets
Mom, Thermomix	That — simply, excellently
Hard math	Specialist Oo at very high level
Software development	Specialist Oo at very high level

Axis	Rule
Skill floor	Grandma-simple entry — no jargon gate, no setup maze
Skill ceiling	Expert-grade per specialty — depth when wanted
Anti-pattern	Wunderwuzzi mediocre at everything

Simple entry, deep ceiling. UI and onboarding copy stay calm and plain; specialist lanes carry the depth.

Emergent ensemble

Das Ganze ist mehr als die Summe seiner Teile.

Power comes from more individualists and differences — a crew of distinct specialists with chemistry — not from one mega-agent. Capability emerges when diverse Oos meet in Unterhaltungen and Gruppen.

Pillar	Why
No Wunderwuzzi	Depth per lane beats one mediocre generalist
Specialty ceiling	Each Oo expert-grade in its domain
Fun friction	Banter and pushback between personalities — useful spark
Ensemble > solo	Group stage + handoffs make the crew feel stronger than any single Oo

Chief coordinates; the crew delivers. Product, prompts, and stage should never imply one bot could replace the group.

No Wunderwuzzi

Every Oo has a clear specialty. There is no do-everything generalist personality.

Persona archetypes map to templates (CTO, Fixer, quant, engineer, …).
Unmatched bots fall back to specialist, not “companion/helper”.
Chief (first onboarding bot) is seeded as Chief of Staff — coordinates the crew, does not replace specialists.
Group encounters show active speaker + ensemble; collaboration is the model — banter & chemistry, not corporate-dull committee.

When adding templates or copy: narrow scope, explicit role line, defer out-of-lane work to other Oos.

First-run emotional north star

The app sets itself up. It discovers what it can (profile, tools, crew) and shows the user:

Don't stress. We're here now. Trust us — there are many of us. We've got this.

Principle	Product
Calm collective crew	Reassurance copy + discovery progress (`OnboardingDiscoveryShell`), not a long wizard
Honest discovery	Progress bar + one active step in onboarding — future sources stay in docs, not UI laundry lists
Specialists	Chief of Staff + crew framing; no Wunderwuzzi

Surfaces: Welcome.tsx (spirit), Onboarding.tsx (progress + minimal model/tools steps).

Onboarding → user profile

During onboarding, harvest respectful local signals into the existing user-scoped MEMORY.md (space memory — not a parallel brain):

Signal	Source
Name, email	`rpc.me()`
Timezone	`Intl` / `localTimezone()`
Locale, UI locale	`navigator.language`, saved UI pref
Languages, platform	`navigator`
Default model, computer host	bootstrap `Me`

Implementation:

Layer	Path
Profile harvest	`apps/web/src/lib/user-profile-harvest.ts`
Help & vibe inference	`apps/web/src/lib/user-help-vibe.ts`
Persona/crew selection notes	`docs/PERSONA-SELECTION.md`
Discovery plan + progress	`apps/web/src/lib/onboarding-discovery.ts`
Reassurance UI	`apps/web/src/components/onboarding/OnboardingDiscoveryShell.tsx`

Called from Onboarding.tsx after first bot ensure.

Oo relationship over time

Principle	Meaning
Always on	Presence when needed — 24/7 feeling without theater
Compounding	Stronger/more attuned the more the user works with their Oos (memory + threads)
Never creepy	Warm competence; no surveillance vibes
Not yes-men	Respectful pushback — don't just echo

Seeded in user MEMORY.md (user-relationship-contract.ts). Persona copy and prompts follow. Details: docs/OO-RELATIONSHIP.md.

Memory and the user model

Store	Holds	Written by
`UserProfile` + `UserFact` (Settings → About you)	The user model: who the user is, across spaces; a fact can be limited to chosen spaces	`update_user_profile`, onboarding, chat consolidation, the user
User-scoped `MEMORY.md` (Settings → Memory)	Space notes shared by the bots of one space	The user, onboarding harvest
Bot-scoped `MEMORY.md` (bot → Knowledge)	One bot's notes in one space	`remember` appends dated notes; `mode: "replace"` rewrites

Space notes are not migrated into facts: facts are short, consented statements about the user, space notes are free-form context for one space. A memory document holds at most 16 KB, so a bot's notes and the space notes both reach the prompt; when it is full, remember asks the bot to condense it.

Capability vision — ambient & video (future roadmap)

This PR does not build surveillance. Current Gesamterfahrung stays Stage + sound + calm onboarding (VOICE.md). The following is Ingo's long-term direction — optional layers only.

Capability	Rule
Video calls	Optional — user-initiated; audio-only remains the simpler default
Ambient help	Oo handles practical presence in the background; user need not sit in front of the app
"It's just an Oo"	Presence without performative attention-seeking — compounding via use, not constant face time
Home metaphors (far future)	Could replace alarm system or surveillance camera only with strict opt-in — never creepy, never default-on

No always-on camera/mic without explicit, revocable consent UX
Any sensing feature must pass earned or creepy? (docs/OO-RELATIONSHIP.md)
Specialist lanes for home/security — not one mega-agent watching everything
Respectful pushback when a setup would be over-broad or unsafe

Details and encounter grammar: VOICE.md § Ambient & video presence.

TODO (future passes)

Merge focus choice from onboarding.chooseFocus into help/vibe (mergeHelpVibeHints — mapping exists, wire on answer)
Merge connector authorizations into profile
Wire projects-on-disk / habits-routines / pain-points discovery sources (currently shown as planned stubs)
Suggested Unterhaltungen + specialist Oos + project folders from discovery results
Optional explicit name confirmation step in onboarding UI

9.1 KiB Raw Blame History