A single function Jev-like wrapper for LLMs, including vision models
Allan Riordan Boll’s blog
I was intrigued by Jev and the self-hostable projects appearing around it, such as OpenJev and SemIf. Reading about them introduced me to a neat trick: reading an LLM’s token probabilities.
Apparently this is an old trick for some people. See e.g. OpenAI’s logprobs cookbook. But it was new to me.
I believe the basic idea is to write a prompt like this:
Then add a few JSON request parameters to a compatible Chat Completions request:
The LLM API will return the letter plus the model’s log probabilities for alternative tokens.
Repeat for each question. Forcing it to generating only one token avoids a lengthy answer and is super quick, though processing the input still costs time. Though for each of the questions a shared state prefix can be KV-cached if the backend supports it.
The fun part: this works with vision models too. Jev’s documented request format currently describes only text/JSON state. I added an attachments field for images for my local experiments.
My example captures webcam frames, sends base64 JPEGs, and prints a table: is a person visible, are we indoors or outdoors, and how bright is the scene? With Gemma 4 12B on my RTX 3090, I get around 1 frames per second, with three questions per frame. I also ran it against OpenAI gpt-6-luna and got around 0.2 FPS. Presumably because I didn’t make any effort to avoid the cost of a separate connection through their system per question per frame.
Specialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text.
Here’s the standalone Python example (OpenCV is just used for convenient access to the webcam, not for any actual computer vision):
The script handles the API differences: llama.cpp uses Chat Completions and OpenAI uses Responses to get it to show alternatives.
I ran Gemma 4 12B QAT through llama.cpp. On Linux with NVIDIA drivers, curl, zstd, and uv installed:
Save the Python example as webcam.py. In another terminal, from that directory:
Current transformer models only attend to past tokens when processing input. So, you probably want the options before your input so that every “word” on the state will be processed taking into account the options. If you do that then you can disable reasoning saving some time (and money) and maybe improve radically the results.