ElevenLabs - Mem0

Create voice-based conversational AI agents with memory capabilities

Integrate ElevenLabs and Mem0 to enable persistent, context-aware voice interactions that remember past conversations.

Overview

In this guide, we’ll build a voice agent that:

  1. Uses ElevenLabs Conversational AI for voice interaction
  2. Leverages Mem0 to store and retrieve memories from past conversations
  3. Provides personalized responses based on user history

Setup and Configuration

Install necessary libraries:

pip install elevenlabs mem0ai python-dotenv

Configure your environment variables:

You’ll need both an ElevenLabs API key and a Mem0 API key to use this integration.

# Create a .env file with these variables
AGENT_ID=your-agent-id
USER_ID=unique-user-identifier
ELEVENLABS_API_KEY=your-elevenlabs-api-key
MEM0_API_KEY=your-mem0-api-key

Integration Code Breakdown

1. Imports and Environment Setup

First, we import required libraries and set up the environment:

import os
import signal
import sys
from mem0 import AsyncMemoryClient

from elevenlabs.client import ElevenLabs
from elevenlabs.conversational_ai.conversation import Conversation
from elevenlabs.conversational_ai.default_audio_interface import DefaultAudioInterface
from elevenlabs.conversational_ai.conversation import ClientTools

These imports provide:

2. Environment Variables and Validation

Next, we validate the required environment variables:

def main():
    # Required environment variables
    AGENT_ID = os.environ.get('AGENT_ID')
    USER_ID = os.environ.get('USER_ID')
    API_KEY = os.environ.get('ELEVENLABS_API_KEY')
    MEM0_API_KEY = os.environ.get('MEM0_API_KEY')

# Validate required environment variables
    if not AGENT_ID:
        sys.stderr.write("AGENT_ID environment variable must be set\n")
        sys.exit(1)

if not USER_ID:
        sys.stderr.write("USER_ID environment variable must be set\n")
        sys.exit(1)

if not API_KEY:
        sys.stderr.write("ELEVENLABS_API_KEY not set, assuming the agent is public\n")

if not MEM0_API_KEY:
        sys.stderr.write("MEM0_API_KEY environment variable must be set\n")
        sys.exit(1)

# Set up Mem0 API key in the environment
    os.environ['MEM0_API_KEY'] = MEM0_API_KEY

This section:

3. Client Initialization

Initialize both the ElevenLabs and Mem0 clients:

# Initialize ElevenLabs client
client = ElevenLabs(api_key=API_KEY)

# Initialize memory client and tools
client_tools = ClientTools()
mem0_client = AsyncMemoryClient()

Here we:

4. Memory Function Definitions

Define the two key memory functions that will be registered as tools:

# Define memory-related functions for the agent
async def add_memories(parameters):
    """Add a message to the memory store"""
    message = parameters.get("message")
    await mem0_client.add(
        messages=message,
        user_id=USER_ID
    )
    return "Memory added successfully"

async def retrieve_memories(parameters):
    """Retrieve relevant memories based on the input message"""
    message = parameters.get("message")

filters = {"user_id": USER_ID}

results = await mem0_client.search(
        query=message,
        filters=filters
    )

memories = ' '.join([result["memory"] for result in results.get('results', [])])
    print("[ Memories ]", memories)

if memories:
        return memories
    return "No memories found"

These functions:

add_memories:

retrieve_memories:

5. Registering Memory Functions as Tools

Register the memory functions with the ElevenLabs ClientTools system:

# Register the memory functions as tools for the agent
client_tools.register("addMemories", add_memories, is_async=True)
client_tools.register("retrieveMemories", retrieve_memories, is_async=True)

This allows the ElevenLabs agent to:

6. Conversation Setup

Configure the conversation with ElevenLabs:

# Initialize the conversation
conversation = Conversation(
    client,
    AGENT_ID,
    requires_auth=bool(API_KEY),
    audio_interface=DefaultAudioInterface(),
    client_tools=client_tools,
    callback_agent_response=lambda response: print(f"Agent: {response}"),
    callback_agent_response_correction=lambda original, corrected: print(f"Agent: {original} -> {corrected}"),
    callback_user_transcript=lambda transcript: print(f"User: {transcript}"),
)

This sets up the conversation with:

7. Conversation Management

Start and manage the conversation:

# Start the conversation
print(f"Starting conversation with user_id: {USER_ID}")
conversation.start_session()

# Handle Ctrl+C to gracefully end the session
signal.signal(signal.SIGINT, lambda sig, frame: conversation.end_session())

# Wait for the conversation to end and get the conversation ID
conversation_id = conversation.wait_for_session_end()
print(f"Conversation ID: {conversation_id}")

if __name__ == '__main__':
    main()

This final section:

Memory Tools Overview

This integration provides two key memory functions to your conversational AI agent:

1. Adding Memories (addMemories)

The addMemories tool allows your agent to store important information during a conversation, including:

When the agent identifies information worth remembering, it calls this function to store it in the Mem0 database with the appropriate user ID.

2. Retrieving Memories (retrieveMemories)

The retrieveMemories tool allows your agent to search for and retrieve relevant memories from previous conversations. The agent can:

Configuring Your ElevenLabs Agent

To enable your agent to effectively use memory:

  1. Add function calling capabilities to your agent in the ElevenLabs platform:
    • Go to your agent settings in the ElevenLabs platform
    • Navigate to the “Tools” section
    • Enable function calling for your agent
    • Add the memory tools as described below
  2. Add the addMemories and retrieveMemories tools to your agent with the provided specifications.
  3. Update your agent’s prompt to instruct it to use these memory functions.

Example Conversation Flow

Here’s how a typical conversation with memory might flow:

  1. User speaks: “Hi, do you remember my favorite color?”
  2. Agent retrieves memories:
# Agent calls retrieve_memories
memories = retrieve_memories({"message": "user's favorite color"})
# If found: "The user's favorite color is blue"
  1. Agent processes with context:
    • If memories found: Prepares a personalized response
    • If no memories: Prepares to ask and store the information
  2. Agent responds:
    • With memory: “Yes, your favorite color is blue!”
    • Without memory: “I don’t think you’ve told me your favorite color before. What is it?”
  3. User responds: “It’s actually green.”
  4. Agent stores new information:
# Agent calls add_memories
add_memories({"message": "The user's favorite color is green"})
  1. Agent confirms: “Thanks, I’ll remember that your favorite color is green.”

Conclusion

By integrating ElevenLabs Conversational AI with Mem0, you can create voice agents that maintain context across conversations and provide personalized responses based on user history. This powerful combination enables: