This might be the least “design-y” project I’ve ever noodled on, it’s basically a pure API product. It started innocently enough, I bought a pair of Meta Rayban sunglasses. But I quickly I got frustrated with the speed of Meta’s feature development for them, at the same time realizing I could send messages to WhatsApp. From there I created an API endpoint that accepted any audio/text, meaning a WhatsApp bot, SMS, email, chat/audio interface from a web page, etc. are all possible.

It’s a pretty simple system, it starts with parsing the provided text/audio through a master “context router” that determines what tasks can be completed from the tasks the “monkey” knows how to complete (calling it a monkey was probably funnier at the time, e.g. dance monkey dance). Once the first monkey thinks it knows what the task requested was it passes that task on to a monkey that’s specifically knowledgeable in that task. This new monkey looks for additional information in the text/audio that it needs to complete the task and either completes the task or tells the first monkey to try a different service. Ultimately if a service can’t be found that can complete the tasks found by the first monkey it sends the text to an LLM for a more generic answer.

So far I’ve given it an email address, a phone number (for texting, but voice would work too!), a WhatsApp account (for use with my Meta glasses) and a web interface with a microphone icon to capture my voice. I’ve also given it access to my lights, speakers (& music selection), heaters, calendar, and email. Of course, in the spirt of making “Future Andrew’s” life easier I also set it up to have the ability to have anything that has a webhook added to it in a matter of minutes.

Of course, me being me, I’ve set up as a ‘product’ from the start (e.g. accounts, users, login, etc.) but I haven’t decided if I should go to the effort of making it available to others or not (if you’re interested send me a message, I’d love to chat see if you can use it too!), until I do I’ll just keep making it better for me as it’s incredibly fulfilling to have my own “Jarvis” with me.