r/quant • u/The_God_was_Here • 5h ago
Models How do you tell a well-calibrated forecaster from a lucky one after only ~50 calls?
been comparing a few public btc range forecasts, the kind that say "80% chance it closes between X and Y in 24h". after about 50 days most of them look fine, hit rates between 72% and 86%.
problem is 50 calls is nothing. a forecaster thats actually calibrated and one thats just lucky look the same at this sample size, and the band around 80% on 50 tries is huge.
things i've been considering:
- pool across a few confidence levels (50/80/90) to get more signal per day
- score the width too, so a lazy wide range doesnt win
- interval score instead of plain hit rate
what do you actually use when the sample is this small? or do you just wait 6 months