Predicting MMA fight outcomes with XGBoost

[Ramon Antonio]

TLDR: I trained an XGBoost model on MMA data I've scraped from various resources, and then tested how well the model would do on test data that it wasn't trained on. The model scored an accuracy of 64% on the test set. This is considerably higher than 50% you'd expect at random. The model hasn't been used for actual betting yet, but the next step is to do just that.

A more detailed description follows here:

1. I've scraped various MMA websites for UFC, Pride, Bellator, WEC and Strikeforce results. I also scraped the profiles of all the fighters in these events, as well as their opponent's records, and their opponent's records. This allowed me to have an idea about the quality of competition each fighter has faced. For example: Jones vs. Smith, I have not only the records of Jones and Smith, but also for each fighter the combined record of their opponents indicating how many fights they've had, and what their win% is.

2. I took quite some time cleaning the data and created derrivative / aggregated variables. Every row does not contain future information, but only information that was available before the fight. I created lots of variables containing:

• event continent / home advantage
• reach, height, age
• experience (total amount of fights, wins, losses, draws)
• number of finishes (subs, ko's)
• current winning/losing streak, max winning/losing streak
• # of wins in last 1, 3 and 5 fights
• average fighting time
• fighter birth continent
• opponent strength (win% of opponent)
• opponent experience (number of fights of opponent)
• difference between both fighters in the above variables, e.g. reach difference, height difference etc.

3. Modeling process
I split up the data in a training and test set. The training set contained the 80% oldest data (since MMA is temporal, i.e. a fighters current record contains information about their past performance), and the test data contained the 20% most recent fights.
I used XGBoost as a model, since this has a great record over other modeling techniques, is quite fast to train, and has won several Kaggle competitions.
The model spits out a probability of fighter A winning. These are then converted into classes (i.e. a win% over .50 means a win, below .50 a loss).

4. Model performance
I reached 65% accuracy on the training set, and 64% on the test set containing data the model was NOT trained on. This indicated that the model was not overtrained, since the train and test set both performed very similarly.

5. Betting:
The next step is to use the model for betting (for fun). The plan is to bet on MMA fights following the model 100%, no emotions, no gut feelings.
Since I'm only willing to risk 100 euros and like to keep things simple, I'm planning to do fixed bet sizes at 1.5-2% of the bankroll.
Any fight that provides a positive EV will be bet on. So let's say Cain has a 67% win probability over Ngannou in the model, and the odds are 1.64 for Cain, it follows that 1/1.64 = 60.97%. This would be positive EV, and thus a good bet according to the model.

6. Future steps:

• Add opening odds to the model. I feel like this could greatly improve the model.
• Add recency (days since last fight). This could help in determining whether ring rust is an actual thing, and if so, it could add predictive power.

Any feedback on how to improve on the model / betting strategy is appreciated. Also looking for a good reliable betting site that accepts Dutch customers and has good odds.


Disclaimer:
I know the model isn't amazing, and the betting strategy is questionable. However, this project was mainly done as a side project to update my webscraping / scripting skills. Now for entertainment purposes I'm excited what will follow first: doubling my bankroll money, or losing it all.

 

[onC]

Why dont you do some backtesting vs the opening/closing odds (on Pinnacle for example) to get an idea if the model has any chance of working?

Fixed bet size of ~1% is okay, especially if the model is spitting out outrageous numbers. If they're at all in line with the odds offered, you could just use a conservative Kelly/.

 

[Ramon Antonio]

Why dont you do some backtesting vs the opening/closing odds (on Pinnacle for example) to get an idea if the model has any chance of working?

Fixed bet size of ~1% is okay, especially if the model is spitting out outrageous numbers. If they're at all in line with the odds offered, you could just use a conservative Kelly/.

The backtesting of the test data vs. historical odds data is a good idea. Might try both the 1% rule and the conservative Kelly for this and see which one would have a higher ROI.

In any case, MMA is so volatile. Not sure how many bets I'd need to backtest to get a good feel for longer term performance. I guess I'll start with 100 fights, since it's quite some manual work to judge every fight. Haven't found a good easy to scrape odds website or even better, API yet.

 

[PolarBearPaulVarelans]

It's a fun little project to improve your scripting skills, as you noted, but as you also admitted, it's not a good model. And if you're only predicting the winner of an event at a 65% clip, it's very unlikely that your assigned percentages are any better.

My one suggestion? Like in physics, try to reduce the number of variables you consider to the most important ones. Instead of considering height and reach, only consider reach. Average fighting time and birth continent should be scrapped entirely, since they're both so unreliable. There are others.

 

[Ramon Antonio]

It's a fun little project to improve your scripting skills, as you noted, but as you also admitted, it's not a good model. And if you're only predicting the winner of an event at a 65% clip, it's very unlikely that your assigned percentages are any better.

My one suggestion? Like in physics, try to reduce the number of variables you consider to the most important ones. Instead of considering height and reach, only consider reach. Average fighting time and birth continent should be scrapped entirely, since they're both so unreliable. There are others.

True, I'm still wondering how bad it will actually perform, which I can more accurately judge after the backtesting onC suggested. The getting rid of unnecessary variables is also good. However, the XGBoost model is a bit of a black box. You can get some info on variable importance, but not as easily as with standard logistic regression models or decision trees.

Do you happen to have an idea on what type of accuracy MMA models can achieve from past experiences / others? It remains a highly volatile sport, so I wonder what is realistically possible.

 

[Sharkey]

How far along into the development process are you, Ramon, if I may ask?

If you haven't reached the point of testing each of your factors individually and perhaps even in correlation with one or two other factors, that's something I would do if I was you as soon as you can.

As far as some of your individual factors go, there's a few there that seem to jump out as being the "master of the obvious" such as the fighters' records and the like. Some of your factors are also puzzling as why you're factoring them at all. But that's a good thing. A very good thing. You want those those factors that deviate from what (or how much) the public values provided there is value to be found in them for you. Some iconoclast thinking is required to have long term success in any investment game. "Following the crowd" and using the same handicapping factors that they do is a quick way to lose your bankroll, so I applaud you for thinking outside the box a little with some of those regardless if there is value in them or not.

Best of luck to you. And congrats on the development and the creativity of your model as well.

 

[Ramon Antonio]

How far along into the development process are you, Ramon, if I may ask?

If you haven't reached the point of testing each of your factors individually and perhaps even in correlation with one or two other factors, that's something I would do if I was you as soon as you can.

As far as some of your individual factors go, there's a few there that seem to jump out as being the "master of the obvious" such as the fighters' records and the like. Some of your factors are also puzzling as why you're factoring them at all. But that's a good thing. A very good thing. You want those those factors that deviate from what (or how much) the public values provided there is value to be found in them for you. Some iconoclast thinking is required to have long term success in any investment game. "Following the crowd" and using the same handicapping factors that they do is a quick way to lose your bankroll, so I applaud you for thinking outside the box a little with some of those regardless if there is value in them or not.

Best of luck to you. And congrats on the development and the creativity of your model as well.

Thanks Sharkey for your kind words and tips!

I spent some time cleaning up the update scripts, so it's now very easy to scrape and predict scores for new events. However, before betting I want to go back to my model and see if I can improve by adding / removing variables as was suggested in this thread and seeing how well the model would've done on the last 100 fights had I bet (still have to do this).

One of the major problems I ran into is finding relevant stats in the first place. For example fight metric has interesting fight stats (although you could argue about their reliability), but only reports them for UFC fights, leading to a very small sample size. Even simple information such as whether a fighter has a wrestling or striking background for example is very hard to find online in a structured manner.

Just out of curiosity, which variables did you find especially puzzling?

Thanks again!

 

[Sharkey]

Thanks Sharkey for your kind words and tips!

I spent some time cleaning up the update scripts, so it's now very easy to scrape and predict scores for new events. However, before betting I want to go back to my model and see if I can improve by adding / removing variables as was suggested in this thread and seeing how well the model would've done on the last 100 fights had I bet (still have to do this).

One of the major problems I ran into is finding relevant stats in the first place. For example fight metric has interesting fight stats (although you could argue about their reliability), but only reports them for UFC fights, leading to a very small sample size. Even simple information such as whether a fighter has a wrestling or striking background for example is very hard to find online in a structured manner.

Just out of curiosity, which variables did you find especially puzzling?

Thanks again!

Ah, it really doesn't matter which ones I have trouble making sense of, Ramon. Like I alluded to already, you want components such as those included in your system. Whether it's any of these that you're including already, or ones you may add in the future, you want those unique variables included provided your testing and research backs up those inclusions. Those are what will give you an advantage in the betting market over your competitors. Finding value where nobody else is looking.

I'd actually be more worried about the variables you're using that are more obvious in nature (or "too" obvious) which even a novice MMA fan/bettor could see without even looking, and are thus perhaps hurting your overall expected value. Things such as win/loss record, winning streaks, and of that nature.

The back testing you're planning on doing will answer those questions for you, though. I'd also test your model up and down if it is now easy for you to do so. Take note of the common variables your model has pointed to when it comes to results that have positive expected value. And do so with those with negative expected value as well. Test your variables in isolation if you can. And perhaps even test compounded variables together to see if there is value there. Be as creative as you want to be when it comes to testing and you might be surprised what surfaces and what combinations show positive expected value and which ones are negative. If using it as a betting model you obviously want to focus on those variables that are positive and base your model around them. Those that are negative can be used in a supporting role at best if they are one of your more predictive variables when it comes to finding winners outright regardless of odds. Or they can be disregarded completely if they are of the less predictive variety since they're only hurting your betting model. But your testing will show you all that.

 

[bytesize]

winning rate of FAVORITES in mma/ufc betting is at 68%. I think that should try to beat that benchmark if you want to try and make money betting live. There's this one guy who claimed that his model (MLP) achieved an accuracy of 70% on his test set. See his github page jasonchanhku. Goodluck

 

[Ramon Antonio]

winning rate of FAVORITES in mma/ufc betting is at 68%. I think that should try to beat that benchmark if you want to try and make money betting live. There's this one guy who claimed that his model (MLP) achieved an accuracy of 70% on his test set. See his github page jasonchanhku. Goodluck

Thanks for the info. I've decided to use the betting odds as a base and include those numbers in my model as predictors after reading the book 'fortune's formula'. Then I can see if my variables add anything or simply overlap with the odds.