Rules

The key rules.

We announce every change to these rules, with its date, on Discord and by email.

What you submit

  • One Docker image for amd64, in a public or private registry. For a private image, you give us a read-only credential.
  • Inside it, a server on port 8000 that speaks the Positronic policy API. It starts with no arguments.
  • Every weight and package is in the image. It gets no network when it runs.
  • You submit it with positronic eval run, pinned by its digest. We run that exact image in simulation and in the finale.

What your policy gets

  • On each call: the wrist camera, the two shoulder cameras, the robot state (end-effector pose, joint angles and joint speeds), the gripper, and the task in words. The order of the two shoulder cameras is drawn at random for each run, and nothing tells your policy which is left or right. The policy API doc lists every field.
  • It answers with arm commands (a target pose or target joint angles) and a gripper target, one command or a chunk.
  • One 24 GB slice of an NVIDIA RTX PRO 6000, 14 CPU cores and 150 GiB of memory. Up to 12 episodes run at once against your one container. From stage 2, they come from both simulators, in any mix.
  • Nothing else: no network, and no direct access to the simulator or the scoring.

What we provide

  • We run and score every submission on Nebius GPUs, at no cost to you.
  • The positronic command line and an HTTP API: submit, follow the status, and read the results.
  • The guide, example images that fit the 24 GB of GPU memory: π0.5 DROID (positro/pi05-droid), GR00T N1.7 DROID (positro/gr00t-n17-droid) and MolmoAct2 DROID (positro/molmoact2-droid), a test for your image, a practice eval on 20 public episodes, and the list of the board's episodes.
  • For each run: your score, the result file, the last 4 MiB of your log and a replay of the episodes. The board shows a public replay of each team's best run.
  • The top teams run their image on a real Franka robot arm in the finale, on film.

How we score it

  • We run every submission in simulation and post its score on the leaderboard.
  • The board shows each team's best score.
  • Thinking time counts: the simulation clock keeps running while your policy computes.
  • One submission per day. The count starts again at 00:00 UTC.
  • How a run works: each step from your submission to your score.

The finale

  • After final entries close, up to 10 teams run on a real Franka robot arm: the best teams that beat the best baseline, and at least 3 teams.
  • Your last scored stage-2 submission sent before the close is your final entry. After the close, we run it again on tasks that we do not publish, and that run ranks the teams for the finale. The same submission runs on the robot.
  • The robot results decide 1st, 2nd and 3rd.
  • Every run is filmed, and the scores and videos are published.

Fair play

  • One person, one account, one team.
  • Do not share code or models privately with other teams. Publishing them openly is fine.
  • Do not try to read or change the simulator, the scoring or other teams' data.
  • Baselines are reference models on the board. They take no rank and win no prize.
  • Train on any data that you have the rights to use. Do not train on data from our evaluation runs, other than the results of your own team.
  • We look for teams that cheat or break the rules. We keep the right to disqualify such a team and to remove its scores from the board.

Full rules

One policy. Simulation, then the finale.

You submit a policy as a Docker image. We score it in simulation, and the leaderboard ranks the teams. Stage 1 runs in MolmoSpaces. Stage 2 adds RoboLab. After the close, up to 10 teams run the same image on a real robot, and the real-robot results decide the winners.

Simulation

Two stages, one Total.

MolmoSpaces

From stage 1. Allen AI's simulated homes. A simulated Franka arm picks up objects and places them.

  • Score: the success rate, the share of episodes the robot completes.

RoboLab

From stage 2. Each submission also runs in RoboLab, with a simulated Franka arm.

  • Score: a graded score, the share of each task the robot completes.

The stages

  • Stage 1 starts when submissions open and ends at 23:59 UTC on the mid-season date. Each submission runs in MolmoSpaces. Its Total is the MolmoSpaces score.
  • Stage 2 opens after stage 1 ends. Each submission runs in both simulators. Its Total is the mean of the two scores.
  • When stage 2 opens, the board keeps every team. A stage-1 submission has no RoboLab score, so RoboLab counts 0 and its Total is half its MolmoSpaces score. A new submission runs in both simulators and gets a full Total.
  • The stage-2 board stays open for four days after the close, until the finale ends. A submission in those four days gets a score and a rank, but it cannot be a final entry. We publish the final board with the results.

The leaderboard

  • Submissions run in a queue, in the order they arrive. We score each one as soon as we can.
  • The board shows each team's best Total. Your team sees the scores of all its submissions.
  • One submission per day. The count starts again at 00:00 UTC each day. The guide says what counts. The limit may change during the season, and we announce each change on Discord and by email.
  • A retry with the same --transaction-key returns the same submission. A new key spends a submission.

Failed runs

In simulation:

  • If your policy crashes or does not answer a call in time, the submission stops. It gets no score, and it counts toward your submission limit.
  • If we lose the simulator machine during a submission, we run the whole submission again.
  • If another fault on our side stops a submission, we give the submission back.
  • A submission that fails to start gets no score. It still counts toward your submission limit. If the fault is ours, we give the submission back.

In the finale:

  • If your policy crashes or does not answer a call in time, the episode counts as failed.
  • If the robot or our setup causes a fault, the episode runs again.

Tasks and training data

  • We give you a way to practice and hints about the tasks.
  • Train on anything that you have the rights to use: public models and datasets, your own robot data, and data that you collect in the public simulators. A scripted policy is also permitted.
  • Do not train on data from our evaluation runs, other than the results of your own team.

How a run works

These steps describe a stage-1 run.

  • Your submission waits in the queue, in the order the submissions arrive.
  • We start a machine with one GPU slice, pull your image and start your server. Your server has 30 minutes to answer its first readiness check.
  • The simulator runs the board's episodes, up to 12 at the same time. Each episode opens its own session on your server.
  • In each episode, the simulator sends your policy the camera images, the robot state and the task. Your policy answers with actions, one or a chunk, within 15 seconds. The simulated arm carries them out, and the simulator asks again. The simulation clock keeps running while your policy computes.
  • An episode ends as soon as MolmoSpaces judges the task complete, which is a success, or when the episode's time runs out, which is a failure.
  • After the last episode, your score is the share of episodes that succeed, and it goes on the board.
  • In stage 2, each submission also runs in RoboLab, and the board shows the Total: the mean of the two scores.

Which episodes run?

  • In stage 1, each submission runs a fixed set of MolmoSpaces episodes, with fixed start states. All teams run the same set. We may change the number of episodes during the season. We announce each change, with its date, on Discord and by email.
  • The list is public: positronic eval catalog prints every episode that the board's eval runs.

Many episodes at once

  • We run up to 12 episodes at the same time against your one container. Each episode opens its own session. From stage 2, the episodes come from MolmoSpaces and RoboLab at the same time, in any mix, and share the one GPU slice.
  • Your server must answer all of them on its one GPU, within the limits.

When is an episode a success?

  • MolmoSpaces decides success or failure at the end of each episode. Your score is the share of episodes that succeed.
  • A published score always covers every episode of the set.
  • When two teams have the same best score, the earlier submission ranks higher.

What you see after a run

  • Your score, the result file of the run, the last 4 MiB of your container log, and a link to a replay of the episodes.
  • The board shows a public replay of each team's best submission.

How close is your local test?

  • The practice eval in the guide, molmo.franka_pick_mini, uses the same MolmoSpaces scoring on 20 public episodes. These are not the board's episodes.
  • The practice eval waits up to 180 seconds for a call and has no limit for the whole run. Time your calls yourself against the limit of one inference call.

Does the simulator predict the real robot?

  • We tested it. Six policies ran on our real DROID robot, on phail.ai, and on the stage-1 episodes. The robot results separate 14 pairs of these policies. The simulator put 13 of the 14 pairs in the same order.
  • That test did not charge thinking time. The board does.

Your submission

A Docker image that speaks our public API.

The image

  • A Docker image, public or private. For a private image, you give us a read-only credential for your registry.
  • We run the exact digest you submit.
  • No network at run time. Bake in every weight and package.
  • It runs within the limits below.
  • Test it before you submit: see Get started.

The policy API

  • Your server implements the Positronic policy API, defined in our public repository. The simulators and the real robot call it the same way. What your policy receives and returns: camera images, the robot state and the task in words; it answers with actions.
  • Your server declares its local stack: the processing that runs next to the robot before and after each call, such as action chunking or image resizing. We run that stack in our harness, from the public code. The policy design document explains how the parts fit together.
  • The time of the local stack counts as thinking time, like the call itself.
  • The same image runs in simulation and in the finale.

Time and resources

Thinking time counts.

The simulator clock keeps running while your policy computes, as it does for a real robot. We measure the full time of each inference call and add it to the episode. A slow policy acts late and has less time to finish the task. Every episode has a fixed time limit.

ResourceLimit

Thinking time bounds each episode. Start-up is outside it, so start-up has its own limit: the warmup. The limits in this table are published before submissions open. We may change them during the season, and we announce each change on Discord and by email with its date. A container that goes over a memory, image or time limit fails, and the submission counts.

To stay within the compute budget, we may change the number of episodes per submission, the submission limit and the number of teams in the final re-run. We announce each change on Discord and by email with its date.

Fair play

Build a better policy. Do not work around the test.

  • One person, one account, one team. If a person submits from more than one account, their whole team is disqualified.
  • A team has at most 3 members. The members share the team's account and its API key, and the account holder receives the team's prizes.
  • Teams may merge until 7 days before the close, if the merged team has at most 3 members. The merged team keeps one account, and we close the other accounts. Each account holder that merges away tells us by email before that date.
  • Do not share code or models privately with other teams. Publishing them openly for everyone is fine.
  • Do not try to read or change the simulator, the scoring or other teams' data from your image.
  • Staff of Positronic, Nebius and their affiliates may enter. Their submissions show on the board, but their teams take no place in the finale and win no prize.
  • Baselines are reference models. They take no rank and win no prize.

Finale

Up to 10 teams run on a real robot.

Who goes

  • The close is at 23:59 UTC on the close date. A submission sent before then can be a final entry, even if its score arrives later.
  • Your final entry is the last stage-2 submission that your team sends before the close and that gets a score, even if an earlier submission got a higher Total. You choose it when you send it: a later stage-2 submission sent before the close replaces it if it gets a score. A failed submission gets no score and does not replace your final entry. A submission that is still in the queue at the close counts when its score arrives.
  • After the close, we run each team's final entry again, with more episodes, on a set of tasks like the board tasks, in both simulators. We do not publish these tasks. The robot's start pose, the scenes and the camera images in these tasks may differ from the board tasks. This run decides the finalists. The best baseline is run again in the same way.
  • The 10 teams with the best Total in this run that beat the best baseline go to the finale. If fewer than 10 teams beat it, all of them go. On a tie, the earlier submission ranks higher.
  • The finale has at least 3 teams. If fewer than 3 teams beat the best baseline, the next teams by Total in this run take the free places, up to 3.
  • If a finalist is disqualified or withdraws before the finale, the next team takes its place.
  • The same final entry runs in the final simulation run and in the finale. There is no separate policy for the real robot. We run its image by the digest it had when you sent it, so you cannot change it after the close. We pull it from your registry, so keep the image there until the winners are announced.

How it runs

  • Our DROID setup in Positronic's lab: a Franka Research 3 arm with a Robotiq gripper, and three ZED cameras, two outside the arm and one on the wrist. If we change the setup, we announce it with its date.
  • Every finalist gets the same tasks, the same number of runs and the same setup. The tasks are a new set of table-top manipulation tasks, like the simulation tasks. They are not published before the finale.
  • The image gets the same limits as in simulation.
  • The finale score is the capped success rate: a run is a success only if the robot finishes the task within 2 minutes. On a tie, the better Total in the final simulation run ranks higher.
  • Every run is filmed. The scores and the videos are published with the results. Logs are shown only to your team.

Prizes

Who wins what.

PrizeCreditsTotal
  • 1st, 2nd and 3rd: the finale ranking.
  • Mid-season prize: at 23:59 UTC on the mid-season date, when stage 1 ends, we take the 5 teams with the best Total on the stage-1 board. We run each team's best submission again, with more episodes, on a set of tasks like the stage-1 tasks. We do not publish these tasks. The robot's start pose, the scenes and the camera images in these tasks may differ from the board tasks. The best Total in that run wins.
  • Open-source write-up prize: open to every team with a scored submission. Publish your code under an OSI-approved licence and a write-up of your method within 7 days after the close. Positronic and Nebius judge the entries together and choose the best one by these criteria:
    • Reproducible: the published code builds the image your team submitted, and a run of that image gets a score.
    • Clear: another team can follow your method from the write-up alone.
    • Useful: it teaches other teams something they can use again, such as a method, an ablation or a negative result.
    The leaderboard score does not decide. On a tie, the higher Total wins.
  • Every prize is a grant of credits: Nebius GPU credits or Positronic evaluation credits. No prize is paid in cash.

Our top priority is a fair and fun competition. We change the rules or the dates only for a reason, for example to close a loophole. We announce each change, with its date, on Discord and by email.