An antenna to design, fifteen AIs on the test bench

The HW-Bench test pits fifteen models against six antenna design tasks. The initial measurements distinguish response quality and reliability.

Fifteen AI models were given the same files and the same tools to work on the design of a 2.45 GHz antenna. The exercise launches HW-Bench, an evaluation created by Arena Physica to measure what models can accomplish in a design office. The first data cover six tasks, each repeated five times per model. They establish a ranking, but above all show how a good average score can mask a critical unmet requirement.

The results table puts GPT-6 Astra in first place, with 67% of runs satisfying all critical criteria, ahead of GPT-6.1 Sol’s 57%. These figures concern this first antenna design case. They do not constitute a general ranking of mechanical, electronic, or industrial engineering capabilities. A good average can hide a disqualifying criterion

Two measures coexist in HW-Bench. The accuracy score represents the weighted share of criteria met. An important criterion counts more than a secondary detail. The success rate, on the other hand, requires all critical criteria to be met in a single run. A result that is overall convincing can therefore fail if it misses an essential requirement.

The first model thus obtains an average score of 0.81 out of 1. This value does not mean that 81% of the work is usable: its complete success rate is 67%. The distinction also appears for Claude Opus 5, which reaches an average accuracy of 0.71, but satisfies all critical criteria in only 33% of runs. A score does not replace examining the causes of failure.

Repeating the trials adds another perspective. The pass⁵ indicator measures the share of tasks completed successfully across their five runs. For GPT-6 Astra, the 67% correspond to four of the six tasks; for GPT-6.1 Sol, the 33% correspond to two tasks. Eleven of the fifteen models remain at zero on this indicator: none of their six exercises were completed successfully five times out of five. Specialized software drags down the results

The task goes beyond writing a response. The six steps range from defining the antenna characteristics to choosing a reference design, then adjusting it in simulation, translating it into a printed circuit board, running a new simulation, and writing a test plan. The software involved includes Ansys HFSS for electromagnetic simulation and KiCad for electronic circuits.

The detailed results in the benchmark presentation show a clear difference between these categories of tasks. Across the three tasks using simulation and electronic design tools, 30 runs out of 225 satisfy all critical criteria, or about 13%. Across the text and code tasks, the total reaches 92 out of 225, or about 41%.

The conversion of the design into a printed circuit board concentrates the difficulties: only three attempts out of 75 pass all critical criteria. The antenna case study explains that the model must produce an openable project, respect the manufacturer’s rules, and deliver files consistent with the final board. Stating in a report that the circuit is ready is not enough to meet these requirements. Six steps evaluated separately

The initial scope imposes a reading limit: each task starts again from the reference answer prepared by the engineers for the previous step. The model therefore does not have to deal with errors it would have introduced earlier itself. The results do not yet show that a single AI can carry the entire project through, revisit its choices, and correct its own deviations all the way to the final deliverable.

The fifteen models use Atlas, the agent environment designed by Arena Physica. The published methodology specifies shared files, tools, instructions, and constraints, without any special adjustment for each model. The reasoning level is set to “high,” with a cap of 600 calls per run and a time limit of three hours. The results therefore describe the models in this specific environment.

Cost and time are also tracked. The former corresponds to API spending; the latter includes tool calls and simulations. For the figures aggregated by project, the site adds together the averages of the six tasks evaluated separately. These values do not represent the observed duration of an autonomous project run end to end, nor a direct comparison with an engineer’s working time. Verifying the files, then calibrating the judgments

The measurable criteria are checked on the produced files. For an antenna, this may concern simulation parameters or compliance with a specification. The model is judged on its deliverables rather than on its statements alone. The organizers indicate that the reference answers and grading rubrics are prepared and reviewed by domain specialists.

Some criteria, however, require judgment, such as the relevance of a test plan. They are evaluated by three judge models: Claude Opus 5.5, GPT-6 Sol, and Grok 4.7. Several engineers independently rate a sample of outputs, then the thresholds are adjusted until at least 90% agreement is obtained between the judges and the engineers on that sample. This rate describes their agreement during calibration.

Most of the data remains private to limit reuse in the training of future models and protect industrial information. Evaluations using partner data are carried out in their own environments, according to the announced method. Publishing a subset of instructions, files, and traces is intended to provide partial visibility into the tests.

The program also plans to examine unsupported claims in the work traces: a simulation announced but never launched, a fabricated value, or a file cited without having been opened. This automatic trace checking is among the forthcoming developments, separate from the scores already displayed.