Artificial Analysis updates evaluation index using unreleased test questions

Artificial Analysis announced an update to its AI evaluation index, moving the majority of its tests to private, unreleased question sets. By keeping 60 percent of the examination data hidden from the public, the benchmark firm aims to prevent tech companies from training their systems on the exact test answers. In the new rankings reported by the testing group, Anthropic's latest Claude model secured the top position, closely followed by OpenAI's newest offerings.

This shift addresses a growing problem in technology where software models score artificially high on public tests simply because they have memorized the material. For everyday users and businesses choosing between digital assistants, hidden benchmarks offer a more trustworthy measure of actual problem-solving ability on unfamiliar tasks. The updated results also show that proprietary services from major tech firms are widening their lead over freely available open source models.

What remains uncertain is how long these secret test sets can stay confidential before their contents are scraped or leaked into future training data. As benchmark creators rely more on hidden questions, public auditing of the tests themselves becomes far more difficult. This raises an intriguing dilemma: keeping test questions secret solves data contamination, but it might unintentionally reduce transparency in how AI progress is measured.