Matt Herzog
EmailLinkedInX

Voice & Video AI Assistant – Multiplatform

Designer, prototyper2025
AndroidDesign systemUI/UXWebiOS

Production-grade multimodal assistant supporting voice, text, screen sharing, video avatars, and transcription across web, iOS, Android, and Vision Pro.

Overview

As part of the Client SDK team, we designed and built a production-grade multimodal assistant experience powered by LiveKit. The application supported voice, text, screen sharing, video avatars, and transcription — everything needed to interact with an AI assistant in real time.

From a design perspective, the initiative aimed to establish a base design system that could extend across all future starter kits and demos, ensuring consistency while fitting seamlessly into each platform.

Voice & Video AI Assistant for web
Voice & Video AI Assistant for web

Establishing a Cross-Platform Design System

One of the first challenges was defining a design system spanning multiple platforms — web, iOS, Android, Vision Pro, tablet, and macOS.

  • Foundations: Established typography, iconography, and layout primitives tailored for each platform.
  • Brand Extension: Leveraged LiveKit's core identity while extending into a product-ready UI library.
  • Consistency: Created reusable patterns to serve as the baseline for all future open source experiences.
Semantic tokens for light/dark mode
Semantic tokens for light/dark mode
Icon and typography
Icon and typography
Typography and icon pairing for OSS design system on web
Typography / icon pairing for our OSS design system on web

Designing the Multimodal Experience

We designed and built the core interaction components for the assistant:

  • Core components: Buttons, multimodal bars, video tiles, voice visualizers.
  • Platform-specific patterns: Toolbar treatments and layout adjustments for mobile vs. desktop.
  • Competitive analysis: Studied leading video/audio chat apps and AI assistants to guide interaction decisions.
High-level view of key iOS states
High-level view of key iOS states
Voice visualizer states
Voice visualizer states: Idle → connecting → listening → thinking → speaking
Loading sequence for avatar mode on iOS
Loading sequence for avatar mode on iOS

Fitting into the Ecosystem

The assistant was built to live within the broader LiveKit ecosystem:

  • Explored integration with Sandbox for hands-on experimentation.
  • Included in documentation and GitHub READMEs for ready-to-use examples.
  • Positioned as an on-ramp for adoption, showcasing best practices while lowering the barrier to entry.
Docs integration with browser chrome / spotlight component
Docs integration with browser chrome / spotlight component
Sandbox tile for the voice assistant
Sandbox tile for the voice assistant
Landing page for the voice assistant
Landing page for the voice assistant
OG image designed to be customized per customer
OG image — designed to be customized per customer with logos/colors

Outcome

The project delivered both a flagship demo and a design foundation. By codifying these components and styles, we created a consistent, extensible system that:

  • Accelerates the creation of new demos and starter kits.
  • Ensures consistency across web, mobile, and emerging platforms.
  • Provides developers with production-ready UI out of the box.
  • Seamlessly distributes examples across the LiveKit ecosystem.
Experience in action with various interactions