Sony's Richard Marks Talks What Works, What's Next for PSVR

Sony's Richard Marks Talks What Works, What's Next for PSVR


As head of Sony's own Magic Lab, Marks is responsible for some of the most exciting (and sometimes baffling) PlayStation technology

As head of Sony's own Magic Lab, Marks is responsible for some of the most exciting (and sometimes baffling) PlayStation technology

For 17 years Richard Marks has been tinkering, dreaming and experimenting for Sony. As head of Magic Lab, the skunkworks and research department of the PlayStation conglomerate, he's seen his group's imaginings turned into a series of hi-tech, mostly low-cost peripherals and games that have changed the way we play. There was the 2003 release of the EyeToy camera which brought AR to our TV screens by bringing toys to life in our hands – that later became the PS4 camera. There's the PlayStation Move controller from 2010, which started out as a Wiimote-style wand to compete with Nintendo's 100-million-selling competitor to the PS3, and now finds itself a centerpiece of Sony's attempt to mainstream VR. And most recently the biggest skunkworks project of them all – the PSVR system itself, a years-long global project, codenamed Morpheus, that occupied not just Magic Lab but multiple Q-Branch-like R&D departments across the company.

Today, some seven months on from the PSVR launch on a bright Spring day at Sony's San Mateo campus in California, he's showing off the latest gadget to emerge from his team's relentless pursuit of cool tech – the Aim Controller. Essentially a lightgun for the VR age, the Aim is at first blush yet another example of a well-designed peripheral for the 55 million-selling PS4. It's a gray-white loop of futuristic plastic that feels utterly natural in my hands as I make my way through the alien-infested rock formations of Farpoint, a VR sci-fi shooter game that feels every bit like an homage to Paul Verhoeven's Starship Troopers.

In person, the 48 year-old Marks is an affable blend of geek and substitute gym teacher – a cleancut, trim ambassador for Sony's technical ambition, with the enviable energy of a man who's in love with his job. Even in the confines of a windowless demo room, he gets visibly giddy when our attention turns from Aim-powered alien slaughter to a cute tech demo that Marks and his team created to inspire VR game developers to build more engaging experiences. Wary of giving it a brand name ("It's just a tech demo – it's not a 'thing' as such so we just call it 'Believable Characters'"), he dons the headset and uses the Move controllers to interact with several childlike robots in a playroom. When he points at a toy, they all turn to look in unison. When he nods his head, one of them waddles over and picks it up. When he looks at them, they turn to face him. Marks believes this kind of simple AI is really the next building block towards a more convincing VR experience – to making VR games places where you don't just move through movie sets like a floating camera, but encounter a world where everything you see can also see you. Add in other human players – so called "social VR" – and we all take another step towards the Star Trek Holodeck becoming a reality.

With the Aim Controller and playroom demos out of the way, Marks takes time to answer some more general questions about his Magic Lab, PSVR and what's next for virtual reality.

What were you working on at Magic Lab when you first heard that Sony wanted to do VR?
When I first joined here we talked about having sensors on your head and doing stuff like VR – so that's 17 years ago we talked about doing VR. So it's kind of been a backstory forever. And the screen, and having a good screen, is something that seemed not viable at the time, and the optics and stuff was hard to do and expensive, so we worked more on the interaction part of it. We worked a lot on new ways to interact with a virtual world but not the display side of it at all. What we were doing in my group right before the VR stuff kind of happened is we were seeing what you could do with a television set if you could track the person's head relative to the TV, so that what's on the screen would change relative to where your head is. So you'd see kind of a 3D effect on the TV, and then you could have Move controllers and reach out and interact with it – but it wasn't VR. It's kind of what people call Desktop VR or Fishtank VR. And we were talking about how viable that would be. And the problem we felt with it is it's a little bit weird to have this big TV but only one person is having that experience and everyone else doesn't understand what they're seeing, because when you render it from the point of view of that one person it looks weird for everyone else, and so it always looks wrong to everyone else except for you – like chalk paintings on the sidewalk where from one spot it looks really good but from everywhere else it's very weird and distorted.

People are scratching their head.
Yeah. So it's just like that, and we're like, "This doesn't feel like the right use of the screen," and at the same time a guy in R&D, and actually a few different parts of the company –

When was this, roughly, when you were doing the TVs?
So this is 2009, 2010. We had the PlayStation Move controllers, and a lot of people were just doing different stuff with them. We put them on our head and did this with the TV. Another guy in R&D was putting it on his head and having a phone screen in front of his eyes, and that's, you know, more like VR. And the same thing happened down in Santa Monica Studios. They made, like, a God of War VR demo. They took the code and just did the same thing with a screen in front of their face, a phone screen. And then some guys in the UK were doing the same, but they were working more on better optics, so they actually made it so it looked better. And then some people in Japan, too. So it was this grassroots thing at Sony – a lot of engineers liked VR and wanted to do something with VR. So they all had a poor man's VR system because they had this cheap tracking system thanks to the Move controller, that they could use, and it just worked with the PlayStation hardware already, so they could just plug their game straight into it and do stuff. And so everyone came to these internal events and showed each other what we were doing, and kind of got some bottom-up momentum, and then from the top down the head of hardware and the head of content development and the head of the game studios, they decided, "Yeah, let's do this as an official full-blown thing."

When was that?
That was 2011.

It's kind of amazing how all these hardware manufacturers – and some guy in his garage that we all know about – all started working on this modern iteration of VR at roughly the same time, after an abortive start with Virtuality and all that stuff in the Nineties.
It was kind of a perfect storm of different tech. There's the tracking tech, which had been expensive and complicated and now was becoming kind of commodity for a lot of people, and cameras for doing the tracking were becoming low cost, so yeah, gyros, accelerometers, cameras all became low cost. And then you had screens from cellphones that had gotten high res enough to produce a good quality image. And then you had the graphics horsepower to render all of that stuff, so like the PlayStation 4 graphics seemed like they could cross the bar where you could create a believable environment. Because you do have to render for each eye, you know, and you have to change the rendering as you move, it puts a lot of stress onto the graphics side of things. So I think those big three things – the tracking, the display, and the graphics – were why it's like, "Now is the right time." And maybe also it takes some amount of time from the previous attempts before you decide, "Okay, yeah, it's viable again."

What was the first hurdle for PSVR?
Well, I think one of the things that internally we focused on all along was, 'This has to be comfortable', and you have to be not thinking, "This is horrible to put on my head." So we really wanted it to be easy to put on your head and be comfortable. The tech is what we usually focus on at Magic Lab, but we realized to make it a product you have to solve those design issues, especially. Sony is a consumer electronics company, and just having a good tech demo is not the same as having a good product, and from the very beginning the mechanical engineers were figuring out how to make it fit people's heads because, for example, one of the often overlooked aspects is that the optics we have in the PSVR headset have a really wide ‘eyebox' it's called, so you don't have to have it perfect and you still get a clear image, and it fits a large variety of eye sizes because the spacing between people's eyes varies a lot.

So our eyebox covers a big range so that you don't have to have any adjustments. It just works for a huge portion of the population. And so that one little thing matters a lot, because it just means you put it on and it's kind of already right for everybody. Then having it be easy to wipe off and that kind of stuff also, I think, is important. So I think the challenge we felt originally – we were really happy with the basic tech pretty early on – but we thought, "Okay, now we need to make it comfortable."

So it was the industrial design part that was actually the first big challenge?
Yeah, and then another thing is we already had this whole system of the PlayStation and where the HDMI comes out of the PlayStation and goes to the TV, so now we needed to add this device in, and we needed to intercept the HDMI and split it, and then send that into the headset and then still send something up to the TV. And pretty early on we decided that it's important to have something on the television so that other people can share the experience with you, or potentially even see something different. Some of the games have a social screen that's not what you're seeing but they can collaborate or compete against you, so you can have this asymmetric VR experience, which is good for social interactions and means it's not just a single-player experience all the time. And so, making a box that would do all that and get the wiring figured out I think is another challenge. And we're still continuing to work on all of these things. It's like the tech isn't ever really done, the ergonomics and the simplicity is never done either. So that's still undergoing constant evaluation.

We were having a debate in our office the other day about whether it felt less weird to play VR alone or in a room with other people. We were sort of evenly split on that one. Is that the kind of stuff you have to think about, too? How people relate to the idea of being in two places at once?
Yeah, some product decisions kind of were related to that. Like, if you have the, kind of, ski-mask style headset, not only does it press on your face – which a lot of people don't like, and it makes it hard to support glasses as well – but it also cuts off the outside world completely, which you think, "Oh, that's good, because I can completely be immersed in the virtual world," but then some people feel uncomfortable not even being able to look down and see their feet and not having any connection to the real world. And so, we decided to design the headset so there would be always some light coming in and some space, and so it actually doesn't press on your face at all, and if you want to be able to see nothing you can shut the lights off and play, right? So you have some control over that. But if you want to have the lights on and be able to always see your feet, you can play that way too.

It's not an all or nothing kind of experience.
Right, so I think we felt that was a good choice to make it accessible to more people.

PSVR has been out in the wild since October of last year. What have you learned since? Presumably you're getting data sent back when people are using this stuff?
Well, we know that they're being played, which is good. We know that a lot of people are using them. We know that there's some games that they play for pretty long periods of time and there's some games that are played for shorter periods of time. And I think we didn't know how long we expected people to be in the headsets. As developers, we're in there for hours on end, so we know that it's possible to spend hours in there, but we didn't know if people would play that way. And some people do stay in there for a significant amount of time. I don't think we have any announcement of a number, but it is a fairly long amount of time. We do encourage the developers to make stopping points possible so that people don't have to feel like they're locked in, but people do play past those – they keep playing.

Are some people playing for an hour or more?
Yeah, they are.

Are some styles of games better at keeping people playing than others?
I think it's just like most games that are story-based you play for longer periods of time because you want to get further in the story. And then definitely games that are multiplayer people seem to be in there longer, because you're interacting with other people, so I guess just that causes you to stay there longer. Same thing we see with regular games, same kind of data. It's very similar.

So is part of Magic Lab's job now to work with developers to encourage them to use best practices when they're working with PSVR? Do you meet with developers and look at what they're doing, and give them advice and help?
So, we have a developer support group that knows about the main things, and they would do that, kind of, first level. If they do have something a little bit off the normal path and they want to ask about that, then we might help them evaluate it and we might even make a prototype at Magic Lab to help answer that question if we don't know the answer. So we're kind of like a second tier of developer support sometimes. Or we're sharing things that we've tried. We go out and just, "We tried a bunch of stuff, and here's the stuff we tried, so that you don't have to try all that yourself," and try to share what we've learned. We often take a piece of technology and make an experience, and we have our own internal developer conference and we'll show tech demos, like, at a science fair and let them try it. We have a lot of different little demos that they can try out, just to say, "Oh yeah, maybe I'll make a game that has some of that element in it." But not so much, like, per-game kind of answering the first level of questions that they have, because there's just too many developers now. There's several hundred. We're only a small team, so that would be a full-time job.

So you're more working on the technology that they could deploy later, perhaps?
We work on the technology, or sometimes we take third-party technologies and try to see what kind of experiences it would be good for. So if somebody had, say, some eye-tracking hardware we'd ask "Oh, what would you use that for?" Well, not a lot of people have done that, but a few years ago we were working on that to see how this could matter to our company or to other developers. And similarly with some of the locomotion hardware; how easy would this be to interface this kind of idea to some games, and what kind of games would it be good for?

You look at most of the games that are the most popular in the world right now. There doesn't seem to be a VR version that makes sense – the likes of anything from League of Legends to Overwatch, and even open world games like The Witcher 3 or Horizon Zero Dawn, these very deep play-for-five-hours experiences. Is there any reason why VR can't play in those worlds?

I don't think there's any reason they can't be. I think there are reasons it's not right now, and number of seats is one, right? So if you have a million people versus you have 60 million people, the kind of game that will work is different. The number of people that are playing Overwatch right now is a huge number, and no matter what level you are you'll find a right level for you when you play. And also, for the same reason – the number of seats – it's just the right amount of money for a developer to spend developing a game because that number has to match the number of possible sales. So the return on investment still isn't going to work out. That's not the right approach for them right now. I think one of the really interesting use cases is something like Resident Evil, because their game works really well in VR and also outside of VR. So they can still sell to a huge number of people, but they can make the VR version a great VR experience for the people who do own VR. I don't think all games can do that, unfortunately. It just doesn't work out. Some games, the gameplay doesn't feel right in VR at all. So when the game can do it I think it's a great thing for them to do, because they can take advantage of the huge installed base of non-VR players too. But I think once the installed base of VR gets big enough then obviously we won't have that issue. You can just make an amazingly deep long game that's super high production value and it'll be the same as these kinds of games you're talking about. It just won't be exactly the same game.

But you feel this is coming soon?
Yeah, I believe that's coming.

It seems like people get lost in worlds, now, for extended periods of time, and you get these ideas of fandoms springing up around franchises. That hasn't happened yet in VR because you haven't had a game that's been big enough, deep enough, or evocative enough yet or that's really been able to capitalize on those franchises that gamers get really attached to.
Well, there's a game coming out very soon called Star Trek: Bridge Crew, which I think will be, you know, a lot like what you're talking about. So you can have this feeling of connecting to this huge well-known IP, and you can do stuff that would only make sense to do in VR, that game, sharing the bridge and feeling like you're there with these other people who help fly the ship, that just wouldn't be the same if it wasn't in VR. So I think it's a good use case of a well-known IP mixed together with a really good VR experience. I think that's one of the ones that's coming soon. I think there will be a lot more of those in the future. I'm glad they're doing one.

David Votypka at Red Storm, who are making Star Trek: Bridge Crew believes that social is what VR can deliver uniquely, because you can turn your head and look right into somebody's eyes and have them react – as you were doing earlier with those robots. Is this concept of 'co-presence' VR's trump card?

For sure. And I think one of the challenges is, just like with normal games, is if you have a multiplayer-only game, and you go on and there's nobody there, then you don't want to play that game. And so, until you have enough players with VR you can't not really think about making a multiplayer-only game or a very heavily multiplayer-focused game. I mean, Bridge Crew is more like that. It definitely is better with multiple people. So they are doing it. And I think we'll have enough people by then, when it comes out. But I think that's why you haven't seen as much of it in the past – it just doesn't make sense to make a game where you have to have a whole bunch of people there and a whole bunch of people aren't there yet.

I actually think this network effect is gonna be huge for VR – this co-presence effect. It just hasn't really kicked in yet because they're mostly single-player experiences being made, for obvious reasons.

I like to compare it to video conferencing. I think you feel more present with somebody in VR than you do with a video conference. With video conference – and that's what we were trying to highlight with that robots demo – is the spatial cues are all wrong with video conference. The people don't look you in the eyes. They look over at something, and you don't even know what it is they're looking at because you're not sharing that same environment with them. And so you never quite feel like you're there with them. You know, it always feels like they're just kind of stuck on a TV screen. Whereas you go into VR and suddenly they feel like they're right next to you. And that's a very compelling feeling.

So, in your sort of James Bond Q lab that you have, is this what you guys are working on? Is that foremost in your mind right now, this idea of social VR and co-presence?
Yeah, we've been trying to distil what matters the most for VR and what do you really want to make sure you get right, and what things are going to really be big as far as content.

So one of the first things we did was this co-presence thing with NASA where you're actually puppeting a robot on the space station and then there's another robot that someone else is puppeting, and you can reach over and hand stuff back and forth. And this was a couple of years ago that we did that, and we showed it at SIGGRAPH, so it was a very academic-focused thing. But that feeling of pointing and doing this kind of stuff was really compelling. What we realised is it's really neat to just feel like you're there with somebody right away, but if you're more of a gamer, that euphoria of just seeing somebody wears off, and then you want to do more. The gamer in you wants to have a task to do, and I think that's why, for us, we started saying, "Oh," like, in the NASA project there's actually a task where you have to do this stuff and cooperate to get it accomplished, and that mattered to us. And a lot of gamers you put in, they think it's really cool at first but then they want to do the thing that they need to do, or they just start throwing stuff. If you could just sit there and talk, then it's just a chatroom, and I think that will appeal to some people but I think especially gamers want to actually do stuff together. And so we started looking into what kind of tasks would be good to do together, and then also how could I do stuff with non-player characters so that it feels as good as it does in VR with real people.

Is there ever a tension between what you guys as a group are looking at or wanting to pursue and the quite real needs of, 'We need to sell more PlayStation VRs so we need stuff that's relatable to a male gaming audience,' for example, or something like that?
No, we don't have to solve that problem with our group. I think that the game studios sometimes get, you know, "Okay, here's the catalog, and we need to make sure we have everyone well-represented in this space" but that's not really our job to do. We do look at, if there's a technical hurdle or something, "Can we remove this hurdle?" So we've looked into locomotion, different kinds of locomotion, because that's something that a lot of developers run up against. It's like, "How do I move through the game." I think all of the VR community is trying to solve locomotion in different ways. You can just walk forward, but some people don't like to do that. Teleporting works for a lot of experiences. There's some things people have done with throwing themselves through the world, or jumping. You know, there's different experiences – different things work for different experiences – and trying to make a set of those is something that our group has done. But more on the technical side and less on the content side.

Yeah, locomotion is a tricky one. Do you feel like we've kind of exhausted the options there?
I don't think so. I definitely think there's more to come, and I think that it will be more experience-focused. People will find the thing that is really good for this or that genre of game, and then that'll stick for that genre, and then for a different genre it might be a pretty different thing.

From the data you guys are getting from PSVR being out there in the world, have there been any surprises for you? Have you seen data that's made you go, "Wow, we had no idea people would figure this out, or use it this way,"?
I'd say that the most data we got was when we started just doing live demos and watching people use it directly. We gave 500,000 demos last year in stores, and my group did a lot of demos at trade shows and to developers early on, and what we noticed – the thing I noticed the most – is that when anything ever comes within arm's reach, people reach to touch it. They want to interact with it. And that's one of the strongest things, I think, in VR, is that you can actually bring things within arm's reach. When you're looking at a television set, nothing is ever within arm's reach, right? And so it never feels like something is so close that you just want to reach out and grab it. And in VR, every time something got close, people put their hand out and a lot of the demos we were showing didn't even have any controllers at all. It was just their own, empty hand they were extending. And they somehow expected that their real hand would do something in the game, and of course it doesn't. And then they'd get a little hit of disappointment. And so I think with the hand controllers you have a lot more ability to do that where you can reach out and touch something. In the robot demo I just showed you, you can't pet these characters but everyone wants to pet them when they get close enough. They want to pet their heads and stuff. We just chose not to do that, because that's not what we're focused on, but I think a game developer definitely should put that in there, because that's what everyone wants to do.

I feel like Kinectimals took a little step in that direction on Xbox 360 – but without the VR, obviously.
Yes, but here it's so much more. We had a demo where there's a bunch of little guys all around you, and a lot of people tried to touch them, but then there was another demo with a DualShock controller, and it opens up and a guy pops up out of it, and, you know, you feel like you have control over him, and he's standing there and you can make him jump off of it and stuff, and that's a really powerful feeling too. So I think VR – a lot of people focus on the scale, and, I mean, seeing a dinosaur is amazing – but actually when stuff is within arm's reach I think is maybe its most unique selling point. There's nothing else that can put things next to you like that.

Are you seeing different types of experiences now from when PSVR was in development? Are you seeing an evolution happening?
Yeah, definitely. I think the initial goal for developers was always, "Oh, we already make this kind of game. Let's make that in VR now," and I think a lot of developers are starting to think, "Oh, well, what is VR good at?", and then focusing more that way. So you see a lot of people focus on puzzle games that involve some kind of spatial aspect or you mentioned, like, League of Legends, which feels like watching a sort of chess game, and I think having that all play out around you in VR could be really compelling. We haven't quite seen that game yet, but I actually expect there will be a really compelling god game where you're God manipulating these things. I mean, can you imagine The Sims in VR? That seems like it would be a really natural fit. You could bend down and peek into their little house.

My whole world can just be remanufactured by the system at will, by a simple voice command

It does seem as if even the most skeptical game companies out there are seeing that there's something there, but no one really knows quite what yet.

Well, it's a very easy sell with all of the developers and the technical people. They all love it, and they all want to work on it all the time. So I think that's usually a good indicator that people like it. I want to make it clear, too that my group – Magic Lab – was working on something but VR was also being worked on by all these other groups at Sony, and then there was a huge engineering effort to make the PSVR product. My team isn't the one who made the headset, right? We often get this credit as being the inventors of VR – PSVR even but we helped do a lot of things. It's a pretty big effort in Sony and there's a lot of different departments, and even the game studios were very involved early on in it, because they were the ones who would love to be able to make the content for it. I mean, if we just made the hardware and nobody cared, that would be the worst possible situation, so that's why we don't do that very often. We try not to do that at all, and make sure the game developers want it to happen, and so they were involved.

When you come in to work right now, is there anything that's top of mind in terms of the future of VR?

Well, I guess the thing that I'm into...this Christmas break I did three things. I got a self-vacuuming robot and it has a lidar scanner so at the end of its vacuuming it sends you a map of what it actually did and what parts of the room it covered. So, you know, there's a lot of tech there for this thing to move around your real house – so that's one thing. The other thing I did is I got all the voice stuff, all the voice-input devices that are out there, and tried all of them. And then the other thing is I watched all the Black Mirror episodes. So I don't know if you've watched Black Mirror, but yeah, so I did all that. 

And then, the thing I'm most excited about is actually the voice stuff at home. I can tell it to play music and it'll play something. You can say, "Oh, just play some artist." That's pretty neat to just at-will be able to say that. But it didn't change that much from what I could already do. And I could ask it questions about something, the same way I could quickly type it onto my phone and ask the internet something. But you can also tell it to turn on the lights, or turn off the lights, if you hook all that up. But that's kind of it. Then it's kind of done. But if you took that same voice interface and connected it to VR, it could change anything in your world that you want it to. There's no limit to the things that we can do with software, right? So my whole world can just be remanufactured by the system at will, by a simple voice command, like, "Let's go to a castle." Boom! I can be in a castle. And I can be like, "Oh, no, a castle that has a drawbridge," and it'll just happen. Like, having your whole world controllable by something that's kind of smart like that, that can do what you're asking it to do, I think that's a pretty cool thing, and that doesn't exist yet but it doesn't seem very far away.

There should be a lab somewhere that could work on that...
Well, that's the kind of thing that will involve probably multiple groups and multiple companies even to get all the content that you would want to have happen, but that's what I think the vision of VR is in the future. That's why I see it as the holodeck. I just put it on and I can make my world anything I want right now. I can say, "Oh yeah, let's go to a place where..." and if the characters are kind of smart and kind of believable then, yeah, it can all be generated in a pretty rich way. And you'll have stuff that's really well authored by game developers, and that'll be the places you'll go the most. But you could change almost anything – and everything could potentially be procedurally generated. So yeah, I guess that's this kind of science-fiction answer that's not too science-fictions.