So, imagine you’re in the last moments of the championship football game. There’s a tension in the air so thick you could cut it with a knife. The away team is down by four, it’s 4th and goal, five yards to glory, one second on the clock. It’s not just a game; it’s a Schrödinger’s cat experiment with cleats and touchdowns. The outcome could collapse into either team’s favor, determined by this one last play.

But what if, like, the home team’s coach had a magical crystal ball—a predictive algorithm—that told him, “Hey, 90% chance they’re gonna toss it to #87 on the right”? Imagine how that would change the game!

Now, this isn’t some distant sci-fi concept. Nope. Not today. Welcome to the fascinating world of D.J. Lee—a professor at Brigham Young University for the last 22 years, and a man whose passion for computer vision is only rivaled by his love for football. D.J. Lee is basically the Dumbledore of AI-based video information processing. He’s even ventured into the very cool area of facial motion authentication—so, instead of just recognizing your face, your computer would know you by your secret wink or a spoken “open sesame.”

A Light Bulb Clicks On

A few years back, Lee was watching BYU play and he had this thought. It was like a proverbial light bulb, or perhaps a stadium light, clicked on over his head. “What if AI could help predict football plays?” So, he started this epic journey, with students as his fellowship, to create an algorithm that could do just that.

This AI technology being developed by engineers at Brigham Young University could significantly cut down on the time and cost that goes into film study for all NFL and college football teams, while also enhancing game strategy by harnessing the power of big data.

Using deep learning and computer vision, the researchers have created an algorithm that can consistently locate and label players from game film and determine the formation of the offensive team — a process that currently requires a slew of video assistants.

Game Stills Used to Train the Algorithm

While still early in the research, the team has already obtained better than 90% accuracy on player detection and labeling with their algorithm, along with 85% accuracy on determining formations. They believe the technology could eventually eliminate the need for the inefficient and tedious practice of manual annotation and analysis of recorded video used by NFL and college teams.

Lee and Newman first looked at real game footage provided by BYU’s football team. As they started to analyze it, they realized they needed some additional angles to properly train their algorithm. So they bought a copy of Madden 2020, which shows the field from above and behind the offense, and manually labeled 1,000 images and videos from the game.

So, how does this Hogwartsian spell of an algorithm work? First, it spots players on the field, ignoring all the background noise like the mascot doing the worm or whatever. It’s got like, a 95-96% accuracy rate at this, which is impressive. Then it identifies what roles they’re playing—quarterback, receiver, and so on. Finally, it figures out the formation they’re in, which is kind of like the syntax of the game—the rules that tell you how the words (or in this case, the plays) are gonna work together.

The BYU algorithm is detailed in a journal article “Automated Pre-Play Analysis of American Football Formations Using Deep Learning,” recently published in a special issue of Advances of Artificial Intelligence and Vision Applications in Electronics.

“Once you have this data there will be a lot more you can do with it; you can take it to the next level,” Lee said. “Big data can help us know the strategies of this team, or the tendencies of that coach. It could help you know if they are likely to go for it on 4th Down and 2 or if they will punt. The idea of using AI for sports is really cool, and if we can give them even 1% of an advantage, it will be worth it.”

Now, every awesome thing has its limitations, right? And this project is no exception. Lee and his team have been using Madden 2020 to collect data because getting real football data is like trying to find a horcrux—it’s tricky and not everyone is willing to help you. The camera angles are often problematic and not everyone has access to drone technology or magic flying broomsticks.

The Future’s so Bright

But the future is big and full of possibilities. Lee is hoping to make the algorithm smarter, evolving it to recognize even player motion, which would be like giving it the ability to read the football field like a book, predicting plays like plot twists in a suspense novel.

So, could this be the future of football? Will AI take the spontaneity out of the game? As Lee himself puts it, “Nah, it’s not gonna replace human coaches. The real joy comes in outsmarting your opponent. This is just another tool in the toolkit, another spell in the spell book if you will.”

So, for anyone who’s wondered if technology and sports can coexist in a meaningful way, D.J. Lee and his algorithm are proof that, yeah, they totally can. And that is some awesomeness we can enjoy until someone messes it up.