Mine Betting Data Without Fooling Yourself: Test Out of Sample
More data does not make a rule true. Test it on bets it has never seen, and count how many rules you tried.
Data mining for a bettor means sifting past prices and results for a rule worth betting, and then checking it on bets it never saw. A Pinnacle article describes a betting model as a system that can identify unbiased reference points to determine the probability of all outcomes, then says it has to be tested and maintained. The trap is volume: test enough rules and the best of them looks like an edge. Suppose 250 rules with no edge are each tested on their own 200 independent bets at odds of 2.00, flat stakes; the best one averages +19.87% a bet (hypothetical). This page covers the data, the variables, that trap, and what a bet log lets you test on a betting exchange such as Orbit Exchange.
Where the data comes from
A Pinnacle article on building a betting model gives two ways to collect data: "by yourself, or by using other published data online". The scope is a judgement. The same article asks "should you consider all matches or just their league games?" and answers that the call is "determined by what your aim is". So the aim comes first, and the spreadsheet second.
Brokers' own screens, such as statements, analytics and price history charts, differ from one broker to the next. The broker tools page covers what each one puts on screen, so it is not repeated here. Orbit's interface has an "Account Statement" screen and a "Betting Profit & Loss" screen.
One condition belongs next to any talk of data. The terms of MadMarket and of BetInAsia each bar a user from using "any robot, spider or other automated mechanism" to retrieve or index information from the service.
Which variables, and why
Fewer, explained, and checked for drift.
The Pinnacle article lays out a cycle. Read it as a cycle and not a strict recipe. The steps:
- Set an aim, since "Without an aim you could be overwhelmed with numbers".
- Collect the data, by yourself or from other published data online.
- Choose a form for the model. The article's advice: "Our recommendation is not to overcomplicate."
- List its assumptions and their limits; for example, "averages and standard deviations assume events are normally distributed", which has to be tested.
- Test how sensitive it is to the results, and keep it up as time goes on, because "it needs to be maintained as time progresses".
Which variables go in is your call. AsianConnect's article on analysis with AI lists the data such a tool can process: past results, player statistics, team trends, home and away records, injuries, possession, shot conversion and market movements. Another AsianConnect article says "Possession measures only one thing", so a number on the list can measure less than its name suggests.
MadMarket offers a Poisson calculator as one example of a model fed by hand. You enter expected values, such as goals in football, and it returns the ten most likely scores ranked from highest to lowest probability. It shows how little a simple model needs and how much rests on the numbers you type in.
Data also goes stale. AsianConnect writes of esports that a game update leaves previous results "less predictive", and that historical data should be combined with recent performances under the current version. Another AsianConnect article says a few early matches in a season are a small sample. I would take two variables I can explain over twelve I cannot, because the twelve are twelve more places for the history to flatter the rule, and a model whose output you cannot trace back to its inputs is one you cannot repair when it stops working.
The best-of-many trap
Why the winner of a rule contest is usually luck.
Try enough rules and one of them will look brilliant. Which one is the question the history cannot answer.
Some plain definitions, in the site's words. Testing "on the history" means running a rule over bets already played. Out of sample means running it on bets it has never seen. Overfitting is picking a rule because it did well on the history. The risk is the number of rules you tried, so count them, and keep the losers in the record.
Hypothetical. Each rule is tested on its own 200 independent bets at odds of 2.00 (a 50% chance of winning), flat stakes, with no edge. The bar is the exact expected value of the best result among N rules.
Show the data table
| Rules tested | Best average return a bet |
|---|---|
| 1 rule | 0.00% |
| 10 rules | +10.87% |
| 50 rules | +15.87% |
| 250 rules | +19.87% |
Ten rules, none with an edge
Suppose you test ten rules, each on its own 200 independent bets at odds of 2.00, and none has an edge. The best of them still averages +10.87% a bet in this calculation. Pick it and bet it on the next 200, and it has no reason to repeat.
Pinnacle's resources describe the same luck from other angles. A contributor to Pinnacle's resources writes that a betting history with a positive expected value still ended in a loss in a share of simulations, and that the range of possible returns narrows as the history grows. One article says that even over a whole season "randomness does still influence a side's final points total", and another describes "periods of over and under performance lasting thousands of bets". A guest on a Pinnacle podcast says that, when a model is judged on error metrics, you want "a large number out of sample trials" to make sure that what you see is "statistically significant". How many bets that means for your rule is a separate question, and the break-even roadmap starts it without giving a threshold.
A fit to the past is not a plan
A rule that fits the past is not a plan for the next bet, and most bettors end behind. This explains how to test a rule. It does not tell you which rule to bet.
What your own bet log lets you test
Your own record is a test set, and you build it without touching a broker's data. One Pinnacle article suggests that we "Document our bets and look for any strange patterns on the spreadsheet". Orbit Exchange has an "Account Statement" screen. The log of what you intended is a separate record, and it is yours to keep. Comparing the price you took with the close is the job of the closing price page, which this page leaves alone. Which columns the log needs, and how often to read it, sit on the working routine page, and the page on estimating a fair price covers where the probability behind a rule comes from.
A Pinnacle article measures a bettor by how closely the predicted expected value tracks the actual return "over a sufficient sample size". The log is where the two sit side by side, so write a rule down before the bets it covers, and when you change a rule, start a new count, not a continuation of the old one.
Short answers
What is backtesting in sports betting?
Backtesting means replaying a rule you wrote down in advance on past prices and past results, to see what it would have done. The weak point is the price. The test needs the odds that were on offer when you would have bet, not the odds at the close. A Pinnacle article puts the problem for closing prices this way: "it's a retrospective exercise; we can't know the closing odds for a match until it kicks off". Write the rule first, test it once on bets it was not built from, and keep the record of every rule you tried.
What is overfitting in a betting model?
Overfitting is choosing a rule because it did well on the history you had in front of you. The rule then carries the luck of that history as well as whatever is real in it. The chart in the section on the trap shows the cleanest case: rules with no edge at all, and the best of many still averages a positive return on the history it was picked from. Out of sample, it has no reason to repeat.
How many bets do you need to test a strategy?
A Pinnacle article asks whether one season is a big enough sample, and another describes stretches of over and under performance lasting thousands of bets. A Pinnacle podcast guest, speaking of models judged on error metrics, says you want "a large number out of sample trials". The closing price page carries AsianConnect's suggested minimum for tracking closing line value, and the roadmap shows how long even a real edge takes to show. This page sets no threshold.
Where to go next
For the maths that sits under all this, the sample and the variance, go back to the roadmap. The calculators page lists what the site and the brokers offer, and the odds converter turns any price into a probability. A rule needs a place to be bet, and market depth says what a price is worth at size. To open the door itself, the way into Orbit Exchange explains which brokers do it, and the broker comparison sets them side by side.