I threw out a warning after three tries. It was right.
I built a small tool that guesses whether the price of Bitcoin will be higher or lower five minutes from now. There's a website called Polymarket where you can bet on exactly that. My tool makes its guess, writes it down, and checks afterwards whether it was right.
The warning light
The tool had a warning light. It came on when a guess looked shaky, as if to say "careful, this one might be wrong."
The first three times it came on, the guess turned out right twice anyway. So I decided the light was useless and took it out.
I left it running quietly in the background, though, just to see. After 24 tries, the picture was different. When the light was on, the guess was right only 38% of the time. When it was off, the guess was right every single time.
The light worked. I had judged it on three tries. That's like flipping a coin three times, getting two heads, and deciding the coin is rigged.
I made the same mistake the other way round
Three times I added something new to the tool because of one round that stuck in my memory. All three made it worse, and all three came back out.
Then there was a banner that told me to sit out unless the market was moving strongly enough. When I finally went back through the records, the market had never once moved "strongly enough". If I had obeyed my own banner, I would never have played a single round.
And the rounds it told me to skip? The tool guessed right in 12 of 13.
What I do now
Any new idea gets shown on the screen, but it isn't allowed to change anything until it has been tested many times. Watching an idea costs nothing. Acting on a hunch does.
This is a note about jumping to conclusions, not advice about betting.