I’ve been catching up with data analysis school work, where I’m learning about inferential statistics, and particularly, measuring the differences between means (two-sample t-test). Last night, I thought of a fun experiment to test my skills and whether I’m getting any of these lessons right.
The Toronto Raptors have not played well in the last ~10 games. Much of the discussion has been around the poor defense the team has played recently. And many are pointing to Pascal Siakam as a factor in the Raptors’ woes and the decline of their defensive play. The eye test can confirm some of these cases: you can regularly see opponents just drive by Siakam, or that he has his attention somewhere else, letting an opponent slip by for a corner 3 or a back-cut.
Let’s take a look at the statistics. To measure defensive efficiency, we can use the advanced stat “Defensive Rating” (DRtg). The lower the DRtg is, the less opponents score, and the better the efficiency. We can then compare DRtg when Siakam is on and off the court. There’s a few problems with this, but I’ll get to it later. (I’ll also be measuring Net rating, which matches offensive and defensive impact.) We should also note that Siakam returned to play November 7th and almost played every game until the writing of this article (last game November 28th), except for November 11th, which was the second night of a back-to-back.
If we look quickly at the NBA’s top level numbers, the Raptors’ DRtg with Siakam on the court is 115 and off the court is 107.5. Before Siakam’s return on November 7th, the Raptors’ DRtg is 103.4. That number increases to 117 after November 7th.
Yikes. From just this quick look, it’s clear that Siakam has affected the Raptors’ defence. But from a statistical standpoint, is it significant? I’m wondering if Siakam is the prime factor for the decline in defence, or are we missing other factors? I think the story is a little more complicated than the top-level numbers suggest.
To use the method I’m learning in class, I’ll state my null and research hypotheses:
Null: Siakam has not affected the Raptors’ defensive play.
Research: Siakam has negatively impacted the Raptors’ defensive play.
I’ll be conducting two-sample t-test(s) with an alpha level of 0.05.
First, I’ll test the on-off splits for the 10 games that Siakam has played.
Here’s a bunch of numbers that don’t mean much, unless you’re into statistics!
What this means is that we need a test score of greater than 2.179 for the difference to actually matter (statistically). When we calculate the scores of the differences between Siakam’s DRtgs and NetRtgs on and off the court, we get DRtg = 0.69 and Net = 1.2. Neither comes close to the critical score we need to assert definitively that Siakam is the issue. But one thing to note, the t-score for Siakam’s NetRtg is “fairly” high, which could suggest that (while it isn’t statistically significant), he has had a large impact on offence to compensate for poor defence.
Now here’s the numbers comparing Siakam’s 10 games vs all 21 of the Raptors’ games this season. Once again, we see here that the observed t-scores aren’t close to the critical scores we need.
In all of these tests, the observed t-scores are not greater than the critical scores. The top-line differences in the numbers that “show” Siakam has negatively impacted the Raptors are………. not statistically significant enough. Thus, we must retain the null hypothesis and reject the research hypothesis.
What does any of this mean? What’s going on here? If the eye-test and the top-line numbers suggest Siakam has been the issue with the defence, but these statistical tests suggest otherwise, there must be other factors.
The first should be that I might’ve made a mistake somewhere in my calculations. Which, I guess, sucks.
But assuming I didn’t make an error, it could be any sort of sampling errors. And there’s a lot of issues with these numbers that could lead to this result. For one, the sample size is not a lot. It’s hard to get a good understanding of Siakam’s impact with just 10 games as the sample size. The way DRtg’s are calculated can also affect the statistical results. The DRtgs per game I listed above (taken from NBA’s Advanced Statistics database) are hard to calculate in just one game – or in Siakam’s case, only in the minutes that he played in one game.
It’s also been a tough time for the Raptors, having been through a two-week six-game road trip mostly on the west coast. I’m sure these players are drained. And it looks like defensive troubles are plaguing the WHOLE team, not just Siakam. You can just see that the Raptors are not executing Coach Nurse’s defensive schemes all that well. That could be to lack of synergy – these players haven’t played with each other very long and the Raptors’ defensive style requires a lot of coordination. And players keep getting injured, which leads to weird lineups every game, further complicating the execution of the Raptors’ defensive schemes.
I’ll leave it at this for now. This was a fun experiment just to test out my statistics skills. And I don’t know if any of this made sense. But it was worth a try!