The Future of Concerts: Stagecraft, AR, A New Kind of Live Show? - Rolling Stone
Home RS Pro Music Biz Features

All Aboard the Magical Mystery Tours

Concert tech zoomed ahead during the Great Pause — and it’s going to keep growing by leaps and bounds. Inside the future of stagecraft

Muse on their Simulation Theory Tour, using Notch FX 3D softwareMuse on their Simulation Theory Tour, using Notch FX 3D software

Muse on their Simulation Theory Tour, using Notch FX 3D software.

Hans-Peter van Velthoven/TAIT

Shaun Clair has seen the future of rock & roll, and it’s a 3D pig. Starting in the Sixties, Clair Global, the company co-founded by Clair’s grandfather, has provided sound systems for major rock tours, from Elton John’s and the Stones’ up through more recent ones like Roger Waters’ The Wall extravaganza. But with advances in AI, lighting tricks, robots, lasers, and other new technologies in the offing, what could the Wall of the future look like?

For one, Clair says, the music would be piped into your ears by way of AirPods or similar in-ear tech — so when that plane smashes into the brick wall at the show’s end, “it will sound authentic,” no matter where you’re sitting. Then, picture everyone sporting Google Glass or similar headgear as the pig arrives. “But instead of that pig being physical and remote-controlled, it’s reproduced through somebody’s lens,” Clair continues. “The pig could explode into a thousand different pigs and bounce around. Or it says over your shoulder, ‘Hey, welcome to the show. Get ready for what’s going to happen next.’”

In many ways, nothing quite compares to an old-fangled, high-tech-free show, where you’re jammed into a club and experiencing music on a primal, visceral level. But in the grand scheme, experts foresee dazzling changes to the concert experience. The gearheads who put on concerts are working to make the live-music experience bigger, bolder, and more immersive than ever. Design companies are scrambling to develop new ways to take visuals that were once reserved for screens and place them all around the room — on various parts of the stage and in the air above attendees’ heads.

And though the concert industry shut down for more than a year, that downtime has allowed technicians to ponder what’s next, like Augmented Reality (AR) and Extended Reality (XR). “The pandemic will lead to an explosion of new technologies and the rapid development of things that were on the backburner,” says expert concert designer Cory Fitzgerald, who’s long worked for production company Silent House — and has been putting together virtual worlds for livestreams and telecasts through XR Studios since the pandemic started.

Creative developments Fitzgerald expected to go mainstream in two or three years instead zoomed ahead in months. He explains that a research project turned into the XR-driven VMAs, which would have occurred in front of a sweeping audience if it wasn’t for 2020. Covid concerns forced MTV to adjust its filming style, resulting in what looked like a compilation of highly stylized, larger-than-life live music videos. Demand led to the ultimate display of this tech’s potential.

Here are a few of the changes that we should see in the decade ahead.

Concert visuals in real-time

Advances in software are allowing AI to react to spontaneous activity in an automatic way, transforming a space in which prerecorded visuals had become the norm. When Stevie Nicks sang “Gold Dust Woman” on the most recent Fleetwood Mac tour, she would extend her arms out. Right then, a mini tornado of gold dust floated across the screen behind her. These “real-time effects” are the result of “faster processing and clever software,” says lighting veteran Paul Normandale, who worked on that tour.

Last October, XR Studios collaborated with multimedia studio Moment Factory to put on Billie Eilish’s lauded online show, which was “100 percent live,” says expert Fitzgerald, who’s long worked for production company Silent House. Fitzgerald adds that his team was one of the first to pull off such a multicamera event with no prerecorded components. Based on where the star was standing, a gargantuan, animated spider appeared to pounce and have its pillar-size legs land perfectly around Eilish like a cage. At another point, the show tapped into fans’ computer cameras, and their faces suddenly popped up to float around Eilish as she sang.

Fitzgerald sees that experience as a lesson and harbinger for concerts. While live music professionals have been playing around with thermal sensors, tracking beacons, and Kinect cameras for years, they’ve “accelerated exponentially in the last year,” Fitzgerald says. “The cost and ease of use has dropped way down, and it’s more affordable and accessible.”

While live-music professionals have been playing around with thermal sensors, tracking beacons, and Kinect cameras for a while, they’ve “accelerated exponentially in the last year,” Fitzgerald says. “The cost and ease of use has dropped way down.” If a performer moves in a certain direction, elicits a certain reaction from the crowd, or sings a different note, they could change the color or the intensity of an effect, he says. Theoretically, the loudness of audience members’ screams could control a choose-your-own-adventure scenario: Maybe a virtual piñata bursts, flinging rainbow stuffing amok — or it falls, and trots away.

Tricks like these might start off in a fancy and expensive category, but — as Fitzgerald points out — “the advent of the moving light was a big change” in the Eighties and Nineties.Plans like these could entice fans to see multiple shows on a tour, since each night’s visuals could be different.

Don’t blink

As for 3D tricks, much can be achieved with eyewear, which, Fitzgerald says, is “without a doubt on the horizon.” Imagine wearing glasses while Lady Gaga performs “Rain on Me” and rain appears to fall all around you.  “Augmented reality in that way is going to be our future,” Silent House CEO Baz Halpin says. “It’s like an iPhone instantly there… As much as it will be a part of the live concert experience, it will be me walking down the street and going, ‘Hey, where’s the nearest Starbucks, my map will appear in front of my eye.’”

“That technology was getting there at the end of 2019,” Fitzgerald explains, adding that “there have been a lot of conversations around 5G technologies that are able to work at much-faster wireless speeds.” He envisions attendees linking branded eyeglasses to watches or phones to experience the same AR seen by fans at home as the lenses replicate what’s processed in the camera. This kind of tech is versatile, too. “You could, conceivably, perform at the Staples Center and have that motion captured and streamed live in an avatar world,” says Halpin.

Music (right) to your ears

Companies like Clair are working to develop technology that would allow concertgoers to have the music onstage piped right into their headphones for sharper, less-muddy sound. “As mixed reality creeps into the live performance, this will be incredibly valuable to the fan experience,” says Clair. “They’ll have the impact of the PA and the subtlety and articulation from the earbuds.” The technology, Clair admits, is still years away.

Meanwhile, pioneers like Meyer Sound are in the early stages of developing concert sound systems that will make the music come directly from the musicians onstage, rather than from speakers hanging over them. “If you’re down in front at a traditional rock show, the voices are 80 feet above you,” says Meyer’s Tim Boot, who worked on, among other big shows, Ed Sheeran’s 2017-19 stadium tour. “I don’t want to hear Ed above me. I want to hear the sound right from Ed, and that kind of localization of sound will make it possible. It’s really hard, but we’re excited.”

It’s a controversial topic, though, as is that of VR headsets, because “having this in your ear and having this in your face isolates you,” says Moment Factory’s Daniel Jean, who emphasizes that the company’s mission is to enhance an artist’s vision while not taking fans out of the moment. “We love the collective, emotional impact,” adds Jean’s professional counterpart Tarik Mikou, who argues that this kind of offering must be optional. “We believe that a real experience is shared. It’s OK to have a layer, maybe with glasses, but there has to be a balance between real, physical elements and the technology itself.”

The arena to the bedroom

Just as music creation has gotten more DIY with the evolution of production softwares like Logic, Ableton, and Pro Tools — even GarageBand — graphic design and 3D-animation software have become more accessible and laptop-ready. If Billie Eilish can make the album of the year in a bedroom, so can a graphic designer create a concert experience.

Halpin says he’s received “more emails from 16-year-olds in the last year” than the entirety of his career: Designers no longer need a college degree or a job at a huge production house, like they did 20 to 30 years ago. With the playing field more open, visuals commonly associated with arena-level acts should start to trickle down to the theater and club level.

Marshmello’s and Travis Scott’s headline-grabbing virtual concerts in Fortnite were put together in Unreal Engine 4, a popular platform for designing full-on video games, but in-person shows can be designed there, too. Software is catching up to visual artists’ imaginations, becoming more all-encompassing and streamlined. Gone are the days of designing 3D graphics on one platform and then having to animate and play out their actions on another.  (Re-rendering was once a terribly time-consuming process.)

Whereas Unreal was created for gamers by gamers, Notch FX is the only option created with concerts in mind, according to founder Matt Swoboda — who worked at Sony, doing R&D for PlayStation before starting Notch in 2014. “If you look at the top-grossing tours [in a normal year], you’ll find that Notch is on at least two-thirds of those, maybe more,” he says. Notably, Muse’s otherworldly Simulation Theory Tour came to life with Notch

App & roll

On U2’s 2018 Experience + Innocence Tour, fans were able to download an app that, when aimed at the stage, turned an MRI of Bono’s brain into a melting iceberg, among other effects. More of this is likely on the horizon. As James “Winky” Fairorth, CEO of the leading stage-building company Tait, half jokes, “That goes to another place, where people need to be entertained while they’re being entertained.”

Concert apps may also evolve to allow more interactivity. When the Jonas Brothers did their pandemic-era online show for the tech company Lenovo, fans could request songs by online  vote, and also submit visual-effect designs. In some ways, the winners helped direct the show. They’re not the only ones intrigued by the possibilities this opens up. For example, could fans end up voting on a show’s final effect in a choose-your-own-adventure moment? “We’re constantly trying to incorporate interactivity and engagement so people become a part of the show,” says Mikou. “When people first started wearing light bracelets in the crowds, that was huge, but that was the first step.”

Recycled stages and headliners at all corners

For decades, stages and hardware from major tours would often be bought by acts, then stored in warehouses after the shows wrapped. But given current environmental concerns, using many of the elements from one stage for another act’s tour is looking more viable.

After one tour, Tait’s Fairorth says, “I probably saw 300 tractor-trailer loads of shit we had built — steel, decks, wheels, and stuff that was not going to sit well in a landfill. At the end of the day, it’s a better business model for the artist [to rent]. I’m recycling all the usable pieces and materials. Artists are going to be worried about their carbon footprint … and they’re going to be kinder to the Earth.”

Advancements in LED stage lighting also play a role here, since LEDs don’t get hot and use a fraction of the power required by moving spotlights known as discharge lamps.  (Those housed gases “under incredibly high pressure” and were created by an “arc with massive voltage,” explains Halpin.)

Meanwhile, an uptick in AR visuals also means building fewer physical props. “Sustainability has become a huge concern,” says Fitzgerald. “Does this [tech] affect those types of concerns and issues? Is it better to make virtual scenery versus building a bunch of scenery out of wood and non-reusable materials? There are a lot of questions that are going to come up that I don’t think have really been asked ever before.”

Hydraulic lifts and cranes that transport artists around stadiums and arenas are getting lighter, safer, and more comfortable. Artists will be more “willing to explore and have fun,” says Fairorth. Meanwhile, projectors have gotten brighter, LED screens sharper — so even if the artist doesn’t fly by your seat, you’ll still feel like they did.

This feature appears in Rolling Stone‘s 2021 Future of Music issue, a special project delving deep into the new era of the multibillion-dollar global hitmaking business. Read more of the stories in the print issue, which is on newsstands now, or online next week on June 15th.


Powered by
Arrow Created with Sketch. Calendar Created with Sketch. Path Created with Sketch. Shape Created with Sketch. Plus Created with Sketch. minus Created with Sketch.