How many penguins are in this wildlife video? Can you track the orange ball in the cat video? Which teams are playing, and who scored? Give me step-by-step instructions from this cooking video?

Those are examples of queries that can be fielded by Molmo 2, a new family of open-source AI vision models from the Allen Institute for AI (Ai2) that can watch, track, analyze and answer questions about videos: describing what’s happening, and pinpointing exactly where and when.

Ai2 cites benchmark tests showing Molmo 2 beating open-source models on short video analysis and tracking, and surpassing closed systems like Google’s Gemini 3 on video tracking, while approaching their performance on other image and video tasks.

In a series of demos for reporters recently at the Ai2 offices in Seattle, researchers showed how Molmo 2 could analyze a variety of short video clips in different ways. 

• In a soccer clip, researchers asked what defensive mistake led to a goal. The model analyzed the sequence and pointed to a failure to clear the ball effectively.
• In a baseball clip, the AI identified the teams (Angels and Mariners), the player who scored (#55), and explained how it knew the home team by reading uniforms and stadium branding.
• Given a cooking video, the model returned a structured recipe with ingredients and step-by-step instructions, including timing pulled from on-screen text.
• Asked to count how many flips a dancer performed, the model didn’t just say “five” — it returned timestamps and pixel coordinates for each one.
• In a tracking demo, the model followed four penguins as they moved around the frame, maintaining a consistent ID for each bird even when they overlapped.
• When asked to “track the car that passes the #13 car in the end,” the model watched an entire racing clip first, understood the query, then went back and identified the correct vehicle. It tracked cars that went in and out of frame.

Big year for Ai2

Molmo 2, announced Tuesday morning, caps a year of major milestones for the Seattle-based nonprofit, which has developed a loyal following in business and scientific circles by building fully open AI systems. Its approach contrasts sharply with the closed or partially open approaches of industry giants like OpenAI, Google, Microsoft, and Meta.

Founded in 2014 by the late Microsoft co-founder Paul Allen, Ai2 this year landed $152 million from the NSF and Nvidia, partnered on an AI cancer research initiative led by Seattle’s Fred Hutch, and released Olmo 3, a text model rivaling Meta, DeepSeek and others.

Ai2 has seen more than 21 million downloads of its models this year and nearly 3 billion queries across its systems, said Ali Farhadi, the Ai2 CEO, during the media briefing last week at the institute’s new headquarters on the northern shore of Seattle’s Lake Union. 

As a nonprofit, Ai2 isn’t trying to compete commercially with the tech giants — it’s aiming to advance the state of the art and make those advances freely available.

The institute has released open models for text (OLMo), images (the original Molmo), and now video — building toward what he described as a unified model that reasons across all modalities.

“We’re basically building models that are competitive with the best things out there,” Farhadi said — but in a completely open manner, for a succession of different media and situations.

In addition to Molmo 2, Ai2 on Monday released Bolmo, an experimental text model that processes language at the character level rather than in word fragments — a technical shift that improves handling of spelling, rare words, and multilingual text.

Expanding into video analysis

With the newly released Molmo 2, the focus is video. To be clear: the model analyzes video, it doesn’t generate video — think understanding footage rather than creating it.

The original Molmo, released last September, could analyze static images with precision rivaling closed-source competitors. It introduced a “pointing” capability that let it identify specific objects within a frame. Molmo 2 brings that same approach to video and multi-image understanding.

The concept isn’t new. Google’s Gemini, OpenAI’s GPT-4o, and Meta’s Perception LM can all process video. But in line with Ai2’s broader mission as a nonprofit institute, Molmo 2 is fully open, with its model weights, training code, and training data all publicly released.

That’s different from “open weight” models that release the final product but not the original recipe, and a stark contrast to closed systems from Google, OpenAI and others.

The distinction is not just an academic principle. Ai2’s approach means developers can trace a model’s behavior back to its training data, customize it for specific uses, and avoid being locked into a vendor’s ecosystem.

Ai2 also emphasizes efficiency. For example, Meta’s Perception LM was trained on 72.5 million videos. Molmo 2 used about 9 million, relying on high-quality human annotations.

The result, Ai2 claims, is a smaller, more efficient model that outperforms their own much larger model from last year, and comes close to matching commercial systems from Google and OpenAI, while being simple enough to run on a single machine.

When the original Molmo introduced its pointing capability last year — allowing the model to identify specific objects in an image — competing models quickly adopted the feature.

“We know they adopted our data because they perform exactly as well as we do,” said Ranjay Krishna, who leads Ai2’s computer vision team. Krishna is also a University of Washington assistant professor, and several of his graduate students also work on the project.

Farhadi frames the competitive dynamic differently than most in the industry.

“If you do real open source, I would actually change the word competition to collaboration,” he said. “Because there is no need to compete. Everything is out there. You don’t need to reverse engineer. You don’t need to rebuild it. Just grab it, build on top of it, do the next thing. And we love it when people do that.”

A work in progress

At the same time, Molmo 2 has some clear constraints. The tracking capability — following objects across frames — currently tops out at about 10 items. Ask it to track a crowd or a busy highway, and the model can’t keep up.

“This is a very, very new capability, and it’s one that’s so experimental that we’re starting out very small,” Krishna said. “There’s no technological limit to this, it just requires more data, more examples of really crowded scenes.”

Long-form video also remains a challenge. The model performs well on short clips, but analyzing longer footage requires compute that Ai2 isn’t yet willing to spend. In the playground launching alongside Molmo 2, uploaded videos are limited to 15 seconds.

And unlike some commercial systems, Molmo 2 doesn’t process live video streams. It analyzes recordings after the fact. Krishna said the team is exploring streaming capabilities for applications like robotics, where a model would need to respond to observations in real time, but that work is still early.

“There are methods that people have come up with in terms of processing videos over time, streaming videos,” Krishna said. “Those are directions we’re looking into next.”

Molmo 2 is available starting today on Hugging Face and Ai2’s playground.

Nvidia acquired SchedMD, the lead developer of Slurm, and launched the Nemotron 3 family of open source AI models.

The Allen Institute for AI (Ai2) released a new generation of its flagship large language models, designed to compete more squarely with industry and academic heavyweights.

The Seattle-based nonprofit unveiled Olmo 3, a collection of open language models that it says outperforms fully open models such as Stanford’s Marin and commercial open-weight models like Meta’s Llama 3.1.

Earlier versions of Olmo were framed mainly as scientific tools for understanding how AI models are built. With Olmo 3, Ai2 is expanding its focus, positioning the models as powerful, efficient, and transparent systems suitable for real-world use, including commercial applications.

“Olmo 3 proves that openness and performance can advance together,” said Ali Farhadi, the Ai2 CEO, in a press release Thursday morning announcing the new models.

It’s part of a broader evolution in the AI world. Over the past year, increasingly powerful open models from companies and universities — including Meta, DeepSeek, Qwen, and Stanford — have started to rival the performance of proprietary systems from big tech companies.

Many of the latest open models are designed to show their reasoning step-by-step — commonly called “thinking” models — which has become a key benchmark in the field.

Ai2 is releasing Olmo 3 in multiple versions: Olmo 3 Base (the core foundation model); Olmo 3 Instruct (tuned to follow user directions); Olmo 3 Think (designed to show more explicit reasoning); and Olmo 3 RL Zero (an experimental model trained with reinforcement learning).

Open models have been gaining traction with startups and businesses that want more control over costs and data, along with clearer visibility into how the technology works. 

Ai2 is going further by releasing the full “model flow” behind Olmo 3 — a set of snapshots showing how the model progressed through each stage of training. In addition, an updated OlmoTrace tool will let researchers link a model’s reasoning steps back to the specific data and training decisions that influenced them.

In terms of energy and cost efficiency, Ai2 says the new Olmo base model is 2.5 times more efficient to train than Meta’s Llama 3.1 (based on GPU-hours per token, comparing Olmo 3 Base to Meta’s 8B post-trained model). Much of this gain comes from training Olmo 3 on far fewer tokens than comparable systems, in some cases six times fewer than rival models.

Among other improvements, Ai2 says Olmo 3 can read or analyze much longer documents at once, with support for inputs up to 65,000 tokens, about the length of a short book chapter.

Founded in 2014 by the late Microsoft co-founder Paul Allen, Ai2 has long operated as a research-focused nonprofit, developing open-source tools and models while bigger commercial labs dominated the spotlight. The institute has made a series of moves this year to elevate its profile while preserving its mission of developing AI to solve the world’s biggest problems.

In August, Ai2 was selected by the National Science Foundation and Nvidia for a landmark $152 million initiative to build fully open multimodal AI models for scientific research, positioning the institute to serve as a key contributor to the nation’s AI backbone. 

It also serves as the key technical partner for the Cancer AI Alliance, helping Fred Hutch and other top U.S. cancer centers train AI models on clinical data without exposing patient records.

Olmo 3 is available now on Hugging Face and Ai2’s model playground.