In Rainbow Six Siege, attackers and defenders have to outwit each other in tense, competitive matches. What players might not know is that a different kind of rivalry was forming behind the scenes at Ubisoft Montreal between designers and data scientists.
The result of that clash was the subject of a presentation at this week’s Game Developers Conference in San Francisco. Analyst Geoffroy Mouret and game director Leroy Athanassoff talked about how they learned to trust each other’s respective teams. Similar to other competitive games, Ubisoft has to make sure that Siege’s characters (who have their own weapons and skills) aren’t too weak or too powerful.
But identifying balance issues can be tricky, and often requires a sophisticated understanding of both player feedback and player data.
Athanassoff explained that it’s difficult for a designer or even a design team to keep track of everything because “you only see the context that you’re in.” An individual’s preferences — from the console or platform they’re playing on to the skill level bracket they’re in — can greatly affect how one perceives the game’s faults.
“That’s why you need data, so you have a 360-degree vision of your game,” said Athanassoff.
Analysts can help developers interpret that data when trying to address a problem. But it’s easier said than done. The reason that designers and data scientists butt heads is because designers are used to having total control and making the last call on every decision. If they can find a way to work together, however, the game will only benefit.
Athanassoff and Mouret realized this in 2016 when they joined forces to fix Blackbeard, an operator who had extremely high win ratios when he first came out.
Before they could change him, they first had to figure out what was causing the problem. The team looked at feedback from focus groups and social media. They had internal play sessions where they tried to recreate situations players were describing online. It helped them come up with a hypothesis: Blackbeard’s rifle-mounted shield was too strong. It protected his head, making him stay alive much longer than other operators. That was a huge problem for a game designed around high mortality rates (where it only takes a few bullets to kill someone).
The data that the analysts gathered supported the shield hypothesis. So the design team lowered the shield’s health points from 800 to 150, and gave Blackbeard a second shield (also with 150HP). At the time, it was the biggest balancing update the studio ever made to the game. But nothing changed; Blackbeard was still wreaking havoc. Frustrated, the designers returned to the analysts and asked for help once more.
“At the time, we were super happy. Having designers coming to you [for help] — we were like, ‘Maybe they love us or something!’” Mouret joked.
Among other data points, the analysts knew how much damage the new shields received and how many times they’d break (only one out of every 20 rounds). The shields were still overpowered. The designers wanted the shields to be destroyed 30 percent of the time, so Mouret and his team did some calculations and came up with a new solution: reduce their health to 60HP each.
Athanassoff thought they were insane.
“We were like, ‘But guys, we do math!’ … Luckily enough, these guys [weren’t math experts], so they trusted us on this,” said Mouret.
Ubisoft updated Siege with the new changes, and the community reaction was harsh. They started calling the operator “Nerfbeard” and thought the patch made him useless. But the data proved otherwise: Blackbeard’s usage and win ratios fell in line with the other operators, and he was still one of the top attackers. Mouret said “it took months” for the Siege community to figure this out.
The experience taught Athanassoff that rather than going back-and-forth with the analysts (who worked in another location), it’d be more useful to have them embedded with the rest of the team. Blackbeard was a huge turning point for their relationship, and now both groups work closely with each other.
“Keep in mind that features should be owned by both game designers and analysts when you’re working on a live project. We really believe this is part of [our] success. … We share everything with them,” said Athanassoff.