Listen for Instructions

Overview

The Butler can listen to users for voice instructions. Say what you want (e.g., "log an expense", "add milk to the list") and The Butler routes it to the right Butler via action delegation—hands-free when you need it. Voice commands are processed through The Butler's intent resolution system and routed to the appropriate Butler based on likelihood scores.

Voice Command Processing

Voice commands are automatically processed by The Butler and routed through the action delegation system. Your Butler participates by registering actions and responding to intent queries.

How It Works

1. User speaks a command (e.g., "log an expense of 25 dollars")
2. The Butler processes the voice input and converts it to an intent
3. The Butler queries all Butlers to determine which can handle the intent
4. Your Butler responds with a likelihood score if it can handle the command
5. The Butler routes the command to the Butler with the highest likelihood
6. Your Butler's action handler is called with the parsed parameters

Register Voice-Activated Actions

Register actions that can be triggered via voice commands. Include keywords and natural language patterns that users might say.

Action Registration with Voice Keywords

// Register action with voice keywords
await window.lifeButler.registerActions([
  {
    id: 'log-expense',
    name: 'Log Expense',
    description: 'Log a new expense transaction',
    keywords: [
      'log expense',
      'add expense',
      'record spending',
      'track expense',
      'spent money'
    ],
    voicePatterns: [
      'log (an|a) expense (of|for) {amount}',
      'add (an|a) {amount} expense',
      'I spent {amount} (on|for) {category}'
    ],
    handler: async (params) => {
      const { amount, category, notes } = params;
      const expense = await createExpense({
        amount: parseFloat(amount),
        category: category || 'other',
        notes: notes,
        date: new Date()
      });
      return { success: true, expenseId: expense.id };
    }
  }
]);

Intent Query Handling

Respond to intent queries from The Butler when it processes voice commands. Return a likelihood score indicating how well your Butler can handle the command.

Handle Intent Queries

// Respond to intent queries from voice commands
window.lifeButler.onIntentQuery(async (intent) => {
  const { action, params, context } = intent;
  
  // Parse voice command
  const command = intent.voiceCommand.toLowerCase();
  
  // Check if command matches your Butler's capabilities
  let likelihood = 0.0;
  
  // High likelihood for direct matches
  if (command.includes('expense') || command.includes('spend')) {
    likelihood = 0.9;
  }
  
  // Medium likelihood for related terms
  if (command.includes('transaction') || command.includes('payment')) {
    likelihood = 0.6;
  }
  
  // Extract parameters from voice command
  const amountMatch = command.match(/\$?(\d+(?:\.\d{2})?)/);
  const amount = amountMatch ? parseFloat(amountMatch[1]) : null;
  
  if (likelihood > 0.5) {
    return {
      butlerId: 'my-butler',
      actionId: 'log-expense',
      likelihood: likelihood,
      handler: async () => {
        return await handleLogExpense({
          amount: amount || params.amount,
          category: params.category,
          notes: params.notes
        });
      }
    };
  }
  
  return null; // Don't handle if not applicable
});

Natural Language Processing

The Butler uses natural language processing to extract parameters from voice commands. Your Butler receives parsed parameters, but you can also implement custom parsing for Butler-specific commands.

Parameter Extraction

// The Butler extracts common parameters automatically
// Your handler receives:
{
  action: 'log-expense',
  params: {
    amount: 25.50,           // Extracted from "25 dollars" or "$25.50"
    category: 'groceries',     // Extracted from context or keywords
    date: '2024-01-15',       // Extracted from "today" or date mentions
    merchant: 'Grocery Store'   // Extracted from merchant name if mentioned
  },
  voiceCommand: 'log an expense of 25 dollars for groceries'
}

// You can also implement custom parsing
function parseVoiceCommand(command) {
  const lowerCommand = command.toLowerCase();
  
  // Extract amount
  const amountMatch = lowerCommand.match(/\$?(\d+(?:\.\d{2})?)/);
  const amount = amountMatch ? parseFloat(amountMatch[1]) : null;
  
  // Extract category
  const categories = ['groceries', 'dining', 'gas', 'entertainment'];
  const category = categories.find(cat => lowerCommand.includes(cat)) || 'other';
  
  // Extract merchant (if mentioned)
  const merchantMatch = lowerCommand.match(/(?:at|from) ([A-Za-z ]+)/);
  const merchant = merchantMatch ? merchantMatch[1].trim() : null;
  
  return { amount, category, merchant };
}

Voice Command Examples

Examples of voice commands that users might say and how they're routed to your Butler.

Requirements

• Register voice keywords: Include keywords and natural language patterns in your action registrations.
• Handle intent queries: Respond to intent queries with accurate likelihood scores for voice commands.
• Parse parameters: Handle parsed parameters from voice commands, and implement custom parsing if needed.
• Provide feedback: Provide audio or visual feedback when voice commands are processed successfully.
• Handle errors: Handle errors gracefully and provide helpful error messages if voice command processing fails.