Autonomous embodied agents with TypeSafe's Jev

Roughly 2 hours ago, I saw this absolute banger of a post from @CompleteSkeptic announcing the existence of something called Jev.

What in the world? Is this another LLM model? Wait... "reads further" ... hold on ... "scrolls back up and watches the video in full" ... and yeah, by the end of the video, my jaw was pretty much on the floor.

Oh look, there's a waitlist. I sign up immediately. Join the discord, listen to the man himself speak about what exactly Jev is, and by now I'm itching to get my hands on it.

~10m later, I get an email from the team. I've got access.

Okay, enough yap. Here's what I did with Jev and what I think it could unlock for humanity.

Experiment "Control Orin"

First off, Jev isn't a typical chat-based model.

It is essentially a probabilistic, structured decision model. Give it data and then ask closed-ended questions to get probable answers.

You get back a set of probable results, each with a confidence score. Here's a sample below.

So I picked a use case I thought would show off its abilities beautifully -

✨Control a "living" creature✨

Setup

Setting: A fully deterministic underwater world, a hungry predator (Orin), a bunch of prey.

Goal: Can Jev control Orin and avoid starvation?

The idea is to have Jev "embody" a fictional alien creature. It will perceive the world through Orin's senses and control Orin's body (swimming, whipping, eating, etc) to achieve some goal, in this case, to feed.

We do this by feeding Jev data containing what the goal is (in natural language), bodily stats and conditions, sensory percepts from Orin's perspective, and a bunch of other context.

Then we pause the simulation, send off a network request, and receive a decision on what Jev thinks the body should do.

This decision is in the form of commands that is fed back into the simulation until the next decision is needed. Rinse and repeat.

Observations

It worked!

Jev successfully hunted, caught food, and rested while feeding. In one run, it caught 4 meals and achieved 100% energy.

In another (I tweaked the structure of Orin's sensory percepts), Jev avoided walls deftly but starved 😅

Sample output of Jev's decisions

The decision logs themselves are quite a thing to watch update in near realtime. If I had access to this locally, I'm almost confident I could get maybe 24fps of uninterrupted simulation playback.

I will be pushing this much further. If you're interested in observing the experiment (and perhaps exploring the Simceladus ecosytem 😉), you can do that here.

Final Thoughts (for now)

I think Jev is a paradigm shift. And for multiple reasons.

The obvious one that hits you square in the face when you get the first response from Jev is the SPEED. This thing flies 🦅.

Even with my bloated experiment context sent over the wire in regular JSON, each decision run took only ~0.7s on average. What??

The cost, too, is something that I think enables the crazy things we should expect in the near future from people hacking with it.

In ~2 hours, I burnt ~4m tokens and it only cost $0.16 🤯. There's more👇

The way I see it, anything requiring structured output over unstructured data is ripe for disruption. Examples,

• Autonomous embodied agents: should get much smarter in the coming years. Jev is uniquely well-suited to the kind of messy data they have to process in terms of architecture and latency.
• Querying your codebases: now makes sense in a way that didn't with LLMs. Seriously. Think about this. Given a bunch of code, what are the probabilities it behaves the way I think it should? You can reason about text in novel ways that LLM approximation just can't beat.
• • Extrapolate this further and ponder just how far you can take this when combined with LLMs, interesting data, and agency.

My opinions might change as we get to see more of this amazing technology but for now: Massive respect to @CompleteSkeptic and team 🚀