Minutes before a big event, you go online to place a bet. But wait, the platform is down! #$%@!!&? You see red. Frustration mounts. What to do ….
The old adage “it’s not rocket science” doesn’t necessarily apply here. It turns out that operating a sports betting platform, may, in fact, be even trickier than launching a rocket into space.
“These are teams of extraordinarily talented people, these are some of the best technology people in the world,” an industry source told Sports Handle of the people who build and operate sports betting platforms. “It’s really, really hard. It’s an extremely complex, difficult problem. The experiences that people are having today are the result of millions of hours of hard work, people working their asses off. There are tons of really smart, hardworking people trying to make your sports betting experience really, really good.”
But no matter how hard they work, we live in a world where outages — whether on your banking site, your food-delivery app, or Google — are part of the landscape. And when it comes to sports betting, avoiding them is even more difficult than it is on other cloud- or web-based applications.
Why? Volume.
“One of the things we’ve seen is an interest in online gambling, so there are more bets by more people in more regions,” Nate Stewart, chief product officer for technology provider Cockroach Labs, told Sports Handle. “When people are going through and placing bets, every transaction ends up in a database, so the more load, the more work … and when the database is at capacity, then you’re going to see outages.”
This week, more operators than ever in more U.S. states than ever are headed for a showdown with the single biggest multi-day wagering event of the year: March Madness.
One thing leads to another
Think of it this way — you’re waiting in line for a fancy coffee drink at your favorite coffeehouse, and they’ve got the coffee, the milk, the syrup, the sauce, the espresso maker, the steamer, but, oops, there are no cups. So, sorry, no coffee. The software and hardware that make up a sports betting website are kind of similar. If one small but critical thing isn’t available or malfunctions, the entire process is derailed.
One industry insider pointed to an outage in New Jersey earlier this year that was caused by a “geolocation hiccup.” All the other ingredients were there to take a wager, but until the “hiccup” was fixed, wagers could not be placed.
Sports betting operators and tech providers work hard to avoid outages, but as in all nascent businesses, they happen. Around the 2021 Super Bowl, Barstool Sportsbook, BetMGM, DraftKings, FanDuel, and BetRivers all had outages of varying lengths on or before Super Bowl Sunday, including some during the game.
Caesars Sportsbook down on day one of launch 🤦🏼♂️ terrible 🤦🏼♂️ #betting #NewYork #CaesarsSportsbook pic.twitter.com/cYGWYjo0zK
— Ragi Baba (@RagiRants) January 9, 2022
When Caesars launched and offered users a $3,000 deposit-match bonus in New York in January, its site went down twice during opening weekend. The company was slow to address the issue with customers, and the outages lasted up to several hours. Just last month, FOX Bet and PokerStars went out Feb. 4 for parts of four days before coming back online late on Feb. 7.
Washington, D.C.’s GambetDC was also down during the Super Bowl, but that outage wasn’t a tech problem of the kind we’re talking about — rather, platform provider Intralot failed to do an iOS update, meaning Apple users couldn’t access the platform.
The more bets, the more potential for problems
When a site goes down, users aren’t the only ones who panic. On the operator side, there is an immediate call to action to try to solve the problem, which can often be like looking for (yep, here’s another cliche) a needle in a haystack.
When they are built out, sports betting platforms are load tested (essentially simulations of how a product will perform) to see how the different pieces of technology will work together in different situations. While everything might go smoothly with 100 users, issues could crop up when there are 1,000, 100,000, or 500,000 users all at the same time.
Sports wagering presents a unique situation in that thousands or hundreds of thousands of people can try to access a platform at the same moment, temporarily overloading a system. Operators try to prepare for the crush, and IT specialists and engineers are constantly monitoring systems looking for error messages, increases in traffic, and other potential red flags.
“Generally, what will happen is that something that the company is monitoring will become concerning and some warning will go off, and you’ll start to look into what’s happening,” an industry source said. “They’ll look to see what the dashboard is saying, or the error logs. … Say you get five customer service messages in 15 minutes, and then suddenly, you get thousands. The best evidence of what the problem is is what the customers are saying.”
This same source said that when writing code, “you try to give yourself an idea of what could go on,” and coders write alarms into the code that will alert the IT staff to issues in the moment. Many digital wagering companies use third-party providers to build out platforms, which can further complicate the issue because the people monitoring or fixing the issue aren’t in the same building or connected by the same messaging systems as the actual operator.
Cockroaches are survivors!
At Cockroach Labs — so named because “cockroaches can survive anything” — Stewart says outages are uncommon. In fact, the company’s goal is to never have planned or unplanned outages caused by the technology. (If an employee mistakenly hits the “off” button on a server, well, then that’s on them.) The company provides technology services to Hard Rock Digital in the gaming space, as well as companies that specialize in finance, health care, and logistics. But in gaming, there are can be sudden tremendous surges in users around big events, making it a unique sector.
“The gaming space is more complex than most for a few reasons,” Stewart said. “The first is the massive spike in demand. Think about the last Super Bowl, 20 million people placed bets. That creates a massive amount of data, so that creates scale problems. You have to make sure all the data ties out. For example, [in banking] if someone transfers $100 from one account to another, and something breaks, you have to make sure the bookkeeping ties out and there is not $200 in the system.”
Cockroach Labs, founded by former Google employees, has found that “scaling out” rather than “scaling up” and creating a “native-cloud database” are the building blocks to eliminating outages. Stewart explained that while most companies look for bigger and bigger servers to handle increased load, Cockroach Labs instead adds servers, and there is, in theory, no end to how many could be added (so, going back to the coffee analogy, think 10 small bags instead of one giant one — if one bag is stale, ideally the other nine are not).
In addition, the company recommends that its clients have multiple servers in multiple regional locations, so even if the servers or data center in, say, Boston, were to go down, the servers or data center in San Francisco would seamlessly pick up the load.
What do you if there is an outage?
But with technology, outages are always a possibility. And with an outage comes frustration.
“People are very upset if they cannot place a bet,” Stewart said. Long diatribes on Twitter following outages back Stewart up. There are a handful of things that customers can do when faced with an outage, including use another platform.
Industry insiders say that most people have multiple betting apps loaded onto their devices. So, while the odds at XYZ sportsbook may be more favorable than at ABC sportsbook, if XYZ is down, the easiest fix is to move to ABC. But some bettors are brand loyal, or, in the case of Caesars in New York, the company was the only one offering up a $3,000 deposit match to new customers. The company could, of course, credit users with the deposit match after Super Bowl weekend, but if you were trying to access the site when it was down, the match couldn’t be used and a bet couldn’t be placed.
DraftKings throwing shade on the FanDuel outage when they will soon be running on SBTech’s notorious tech stack is the very definition of whistling past the graveyard. 🙄 https://t.co/UCezUcTGDN
— Captain Jack Andrews (@capjack2000) August 17, 2020
While there’s really no way for a consumer to know how long an outage will last, those in the industry have a few recommendations for things to try — if you’re not in a rush to place a bet. First and foremost, they say, bet early, so you don’t get caught in the crush. It’s also important to have multiple apps available. In the early going of an outage, customers can go the tried-and-true routes for all things technology: reload the app, sign out and sign in, or shut down and restart your device.
In the case of a short outage or a simple error affecting only you, these options may well work. But for a bigger outage, all you can do is wait. Trying again every few minutes can sometimes work if an outage is sporadic.
“These are the general tech things that you do when your toaster doesn’t work,” an industry source said. “That could work.”
Or not. In which case, bettors can check message boards or social media for updates or contact the operator, though if you do so, industry insiders say, be specific about what the problem is. That will help the tech staff to sort out the issue.
What can operators do to prevent outages?
Operators certainly don’t want outages. It’s bad PR, and with the millions being spent on customer acquisition, it can turn dollars spent on that into customers lost. Some of the biggest names in the game, including DraftKings and FanDuel, have long pointed to owning their own technology as a way to limit outages.
After the 2021 Super Bowl, DraftKings, which at the the time was powered by Kambi, released a statement saying the outages were a key reason the company was looking to bring all its tech in-house. Kambi also powered Penn National’s Barstool Sportsbook at that time.
@FDSportsbook I better see some sight outage bonus. #FanDuel pic.twitter.com/pjtEF26tM7
— coldturkey69 (@coldturkey69) February 26, 2022
“This outage was caused by a surge in traffic that caused problems for our backend provider,” DraftKings wrote in a statement, according to CNBC. “Our DFS and pools products supported by in-house technology are functioning without issue. This incident is why we believe owning our own technology is important.”
The company ended its deal with Kambi in September 2021 and brought its technology in-house.
Whether technology is in-house or contracted, outages are tough on operators, and they’ll do what they can to retain a customer after an outage. As examples, Caesars was able to credit customers with the deposit match after its site came back up in February, and in 2021, DraftKings offered up $20 in free bets to affected customers.
But when it comes right down to it, “#$%@!!&!” Sites of all kinds go down, and your best defense is to bet early, have a plan, and as hard as it may be, stay patient.
