<< BACK_TO_PROJECTS

AI Virtual Companion Box

Gemini APIPythonRaspberry PiReact NativeSupabaseIoT
ACCESS_LINKS
PROPRIETARY_CODE_UNAVAILABLE

AI Virtual Companion Box

An interactive virtual companion device merging IoT and LLMs with real-time emotion detection.

AI Virtual Companion Box is an AI assistant project developed as a virtual companion device that integrates Internet of Things (IoT) technology with Large Language Models (LLM). The device can listen, converse, interact, and provide advice as if it were a real friend. Additionally, it features a facial emotion detection system to analyze the user's emotions and adjust the conversation context, making the interaction more appropriate and personalized.

Project Objectives

  • Study and develop an AI assistant system in the form of a virtual companion device capable of expressing facial emotions on an LCD screen and interacting with users.
  • Integrate Artificial Intelligence (AI) with Internet of Things (IoT) systems.
  • Study approaches for applying Large Language Models (LLM) in daily life.

Key Features

Device Features

  • Voice Interaction: Users can talk to the device via a microphone, and it will respond with a synthesized voice through speakers, accompanied by synchronized lip movements.
  • Emotion and Face Detection: Utilizes a 5MP CSI camera to capture the user's face and predict their emotional state in real-time (such as happy, sad, angry, or fear) to adjust the LLM's conversation context.
  • Virtual Companion Display: A 7-inch touch screen displays a 2.5D virtual companion capable of expressing various facial emotions.
  • Customization: Users can freely customize the virtual companion's appearance (face, hair, clothes), change the background, and adjust voice characteristics (pitch, speed).
  • Idle Mode: When not in use, the device functions as a digital clock to reduce energy consumption.

Mobile Application Features

  • Text Chat: Supports text-based conversations with the virtual companion at any time.
  • Daily Journal and Mood Tracking: The system summarizes daily conversations into a journal and displays the user's overall emotional statistics in a monthly format.
  • Store and Reward System: Users can collect credits from usage to exchange for additional appearance parts or backgrounds.
  • Usage Statistics: Displays user engagement volume with the virtual companion and emotional statistics for the last 7 days.

Tech Stack & Hardware

AI & APIs

  • Gemini API (Gemini 1.5 Flash & 1.5 Flash 8B): Serves as the main brain for processing, generating conversations, converting text to SSML format, and summarizing daily conversations.
  • Google Text-to-Speech API: Converts SSML formatted text into natural-sounding speech.
  • Deepface: A library for processing and predicting emotional states from user facial images.

Software

  • Python: Used for device-side development, utilizing libraries such as Picamera2 (camera), OpenCV, PyAudio & Webrtcvad (audio input), and Pygame (UI creation on display).
  • React Native: Used for developing the mobile application that works alongside the device.
  • Supabase & PostgreSQL: Used as the backend database management system to store user accounts, chat history, and companion data.

Hardware

  • Raspberry Pi 5 (8GB) microcomputer board.
  • 7-inch IPS LCD Touch Screen (1024 x 600).
  • 5MP CSI Camera.
  • USB Microphone and 8-ohm 2W dual speakers.