117,000 examples versus 6.2 million: alphaXiv tests the limits of small decision-making models

With OR-decision-4B, alphaXiv trains an open decision-making model from 117,186 examples. Its results approach those of Jev on several tasks, but sarcasm detection reveals a persistent weakness.

Can a model trained on 117,000 examples compete with rivals trained on millions? That's the experiment alphaXiv conducted with OR-decision-4B, a four-billion-parameter model specialized in decision-making. Built on Qwen3.5-4B using the OpenResearch environment, it achieves results close to Jev on several classification tasks, despite using substantially less training data than Baseten's decision models. One weakness stands out, however: sarcasm, which the model consistently struggles to interpret.

The experiment comes as models specialized in making decisions are becoming increasingly common. Unlike conversational assistants, these systems aren't designed to generate lengthy responses. Their job is to examine context, select an option, assign a score, or answer yes or no. Just days after Baseten announced joey-1, alphaXiv is releasing another open model, alongside its adapter weights, experimental results, and a research environment that allows others to pursue similar experiments. Artificial Intelligence Doesn't Always Need to Write an Answer

A large share of professional AI applications doesn't require elaborate text generation. Determining whether a request belongs with sales or technical support, assessing the urgency of a message, identifying intent, or measuring the severity of harmful content can all come down to a structured decision. In these situations, asking a conversational model to construct a complete response introduces additional processing that may not be necessary.

OR-decision-4B takes a different approach. Given a context, an instruction, and a list of possible answers, it assigns a probability to each option. The application can then select the most likely answer, apply a confidence threshold, or calculate an expected value when dealing with a numerical rating scale.

The method relies on directly reading the probabilities associated with choice identifiers. Each option is represented by a specific token, and the model evaluates its probability at the end of a predefined prompt format. Those probabilities are then normalized to produce a distribution across the available answers.

This avoids treating decision-making as an ordinary conversation. The model isn't evaluated on its ability to explain its reasoning in several sentences, but on its ability to correctly select an answer from the available options. Four Billion Parameters, Without Retraining All of Qwen

To develop OR-decision-4B, alphaXiv started with Qwen3.5-4B, an open-weight model with four billion parameters. Rather than fully retraining it, the team used a rank-16 LoRA adapter, a technique that specializes a model by updating a relatively small number of additional parameters.

Training used 117,186 examples, selected after splitting an initial dataset of 123,339 items. The corpus combines different types of decisions, including classification, reasoning, response adequacy judgments, and synthetic examples. Some examples also include probability distributions generated by a teacher model.

The objective wasn't simply to teach the system which option was correct. Training also encouraged its probability distributions to resemble those of the teacher model, while limiting certain deviations from the original Qwen model.

The process ran for two training epochs. The released checkpoint was selected based on the lowest negative log-likelihood on a fixed validation set of 1,024 examples, rather than the highest observed classification accuracy.

That distinction is technical but important: optimizing the quality of predicted probabilities isn't necessarily the same as maximizing the number of correct decisions on a particular set of questions. A Measurable Improvement on JevBench

The technical details published on Hugging Face provide several results that help establish the model's performance. On JevBench, a public benchmark consisting of 231 decision-making tasks, OR-decision-4B achieves 83.55% accuracy under the primary evaluation configuration, using a 4,096-token context window.

The original Qwen3.5-4B model reaches 76.19% on the same benchmark. Specialization therefore delivers a 7.36-percentage-point improvement, suggesting that a relatively limited amount of additional training can meaningfully improve performance on structured decision tasks.

A secondary configuration using a 3,072-token context window produces a result of 83.98%. This small difference should be interpreted cautiously: on its own, it doesn't demonstrate that the shorter context configuration is superior.

On the 1,024-example validation set, the model achieves 73.63% accuracy. However, because this dataset was also used to select the released training checkpoint, it cannot be considered a fully independent evaluation.

The authors also note that the published results were obtained using the native vocabulary-readout system employed during their experiments. The exported model running through Transformers and PEFT has not yet undergone a separate complete evaluation. Differences in numerical precision, batching, or runtime configuration could therefore produce slightly different results when other teams attempt to reproduce the findings. Jev and Baseten Enter the Same Comparison

alphaXiv also compared its model with Jev and Baseten's three decision models: joey-1, elephant-1, and mammoth-1. The published comparison covers several task families, including multiple-choice questions, ordinal ratings, and binary decisions.

On multiple-choice tasks, OR-decision-4B reaches 79.7%, almost identical to Jev's 79.6%. The alphaXiv model also scores 67% on ordinal ratings, compared with Jev's 65.4%, and 88.3% on yes/no questions, compared with 87.4%.

These results appear particularly interesting given the reported training-data difference. alphaXiv says it used approximately 53 times fewer examples than the 6.21 million cited for Baseten's decision models. That doesn't mean its training was 53 times more efficient, however. The datasets, underlying models, optimization