r/OpenAI • • 16h ago

Discussion Coding Benchmarks

How are you all feeling about the latest and greatest in representative benchmarks nowadays? It feels like TB4 is now benchmaxxed. AA also doesn't quite feel like it's hitting the mark as far as my experience.

My sense is the closest to reality is somewhere between Frontier 2.0 and TB Science 0.1 oddly enough. What are your alls' go tos?

1 Upvotes

3 comments sorted by

2

u/coblade14 14h ago edited 14h ago

I go around X, YouTube, and Bilibili (Chinese YouTube) to find privately made benchmarks and testing scenarios for 2 reasons.

  1. Because they are relatively unknown and niche, it's almost impossible that any company benchmaxxed them.
  2. Usually they are fun to watch so doubles as entertainment as well. For example this bridge building contest is peak entertainment

1

u/mboazak 13h ago

Yah, I've been a fan of Bijan Bowen's youtube channel just for the fun element on my end. I saw that Arena now has a channel that I was watching the other day. Some helpful comparisons. I was surprised with Theo's report on the 6.2 series, seems fairly strong. In general, I think he leans more into the agent should feel fun/easy to work with which the claude series definitely does, but I do think there is something to be said about a model that can do the same job for 10x less $.

3

u/pas_possible 15h ago

Tried Sol 6.2 today, seems to work fine for me