Expert interview: how should humanoid robots see the world?

0:08 Ihsane Debbache: The mistake most companies do is premature optimization on the hardware. For humanoids, it's cheaper and more lightweight to put more cameras than to put more neck muscles. And that data is your biggest asset. The humanoids see tremendous shock and vibe, actually more than we see in off-road vehicles. Don't try to innovate everywhere. Don't reinvent the wheel that's currently being invented, or almost invented.

0:37 Interviewer: LiDAR or camera?

0:37 Ihsane Debbache: I don't know, let's see… Camera.

0:41 Interviewer: Humanoids in homes by 2028: myth or reality?

0:43 Ihsane Debbache: Reality.

0:44 Interviewer: C-3PO or R2-D2?

0:46 Ihsane Debbache: R2-D2.

0:47 Interviewer: Robot future: utopian or dystopian?

0:48 Ihsane Debbache: Utopian.

0:54 Interviewer: Why do humanoids need more eyes than humans?

0:59 Ihsane Debbache: Well, not all humanoids do, first, right? I mean, we see some of our customers, they are using only two eyes, but you're right, the majority use many more. And I think the main reason is really: like humans, we have a neck, and we have also muscles in our eyes that allow us to turn around. But that doesn't mean that's a better design. I think for humans, that was done to compensate: our vision is only clear in the center and blurry on the eyes. So we evolved the system to point our eyes where we need to look.

1:42 Ihsane Debbache: But in the industry, we spent a lot of time improving cameras. So now cameras are very good. They are clear on the whole image, not just the center, and they're fairly cheap to manufacture. So that's why for humanoids, it's cheaper and more lightweight to put more cameras than to put more neck muscles and turn around. Then you get motion blur, and also you're not seeing everything at the same time. So I think that's one area where actually humanoid robots, it's not that they need more cameras, but they are able to see everything all around, continuously, because they have access to high-performance and lower-cost cameras.

2:29 Interviewer: How would you explain a stereo camera to a 10-year-old?

2:34 Ihsane Debbache: So it's really just two cameras, two normal cameras next to each other, and it works a bit like our eyes. So for example, in our eyes, if you put your thumb here, and then as you move it forward, each eye will see it in a different position. Then the way stereo cameras work is they compare where the object is in the first eye and where it is in the second eye. And then from that difference in angle, which we call the disparity, they can tell how far it is. So that's how it's really taking two cameras, knowing how far apart they are, and using that information, you can have the depth. You can know where everything is, and it's how humans see. And most animals as well.

3:24 Interviewer: What's hardest for a stereo camera to see?

3:29 Ihsane Debbache: Yeah, actually, the most difficult things usually for stereo cameras are objects that have no texture, because then it's harder to match, to do the stereo matching. And in that aspect, what we do at Stereolabs is the same thing that humans do. We fill in those gaps with a neural network.

3:56 Ihsane Debbache: So stereo vision has been the specialty for Stereolabs for 15 years. And in the early days, it was done with computer vision. So, like I said, we see where this thumb is positioned in each eye, and then we calculate the depth. But if you notice, for humans, even when we close one eye, we still can perceive depth. And that's from memory. It's from knowing how big objects are usually, and then knowing how far they are. So the human brain uses both together. Now, if you have only one eye, some objects, it's a lot harder to perceive their depth. And it's also a lot easier to trick your brain. That's why we have two eyes. So they're complementary.

4:43 Ihsane Debbache: And today at Stereolabs, we do the same thing. So our neural depth uses priors to compensate for areas where stereo doesn't work so well. And that is, yeah, a plain white wall, for example. It's fairly hard to know exactly: is it concave? Is it convex? Is it flat? Right? Because you don't have any point to compare between the two. And also very dark areas, but that's usually under our control, because you can always have the robot have some lights or some other objects. But yeah, those are the usual suspects: when there's zero texture or insufficient lighting.

5:33 Interviewer: What role do cameras play in training robots?

5:38 Ihsane Debbache: I mean, the camera is the main sensor today used in robot learning, right? That's by far the most important data input, and it's evolved the same way as humans. That's why I call it humanoids. Even us humans, we are very predominantly vision-based, and we rely on our vision more than any other sense, and we see robots following the same thing.

6:03 Ihsane Debbache: That's why right now a lot of our demand is for customers doing data collection. And for that, the optimal way is to always record as close to the sensor as possible. Record the data with the least preprocessing, and also, ideally, recording stereo, calibrated and synchronized data, so that later you can always compute it. So when you're recording data, you want to record the maximum, close to the sensor. Because if you record stereo, and then the state of the art, let's say for VLMs, works with mono, then you can still have the mono. But if you record in mono and then it turns out you need stereo, you can't go the other way, right? So you should always record a superset when you're in the early days.

6:59 Ihsane Debbache: And similarly, if you record the pre-computed depth, then later, when there are new models that work better, you can't recompute it. But if you record close to the sensor, just the raw stereo pairs, you can today feed them to the depth model, then when next year there are much better models, you can redo that and simulate how your robot would have behaved if it had access to this new model. You can know ahead of time the performance, the failure cases, the failure rates.

7:34 Ihsane Debbache: So yeah, those are my main two inputs here. And we see a lot of the industry doing that: it's recording in stereo, recording a lot of perspectives and raw data in as high quality as possible.

7:48 Interviewer: Where is Stereolabs seeing the most demand?

7:53 Ihsane Debbache: I think, yeah, definitely for humanoids. And actually, the more generalization and the more intelligence a robot needs, the better fit our cameras are. And that's because, so in the stereo camera landscape, there are all the other competitors, most of them: the way it works is the depth is computed on the camera. So you have the two stereo pairs and then an ASIC computing depth. And then what comes out of the camera is one image and depth. The way Stereolabs does it is the two cameras go to the host computer, to the brain.

8:38 Ihsane Debbache: So I think both have valid use cases, but the first one, right, it's more useful in robots that don't have a lot of compute, right? Let's say a lawnmower, or a Roomba, or a floor cleaner that needs to be autonomous. Then, yeah, absolutely: you want ASICs to compute depth as much as possible and reduce it.

9:07 Ihsane Debbache: And funnily enough, we see that same trend in evolution. Some insects, they have a small brain behind their eyes that's doing the disparity, that computes depth and then sends it to the main brain. But for all the mammals or all larger animals, right, it's the other way around: it's the raw data going to the big brain. And this shows that evolution explored both paths. But whenever you have more compute available and you need more intelligence, because even a dog is more intelligent than an insect, then it makes more sense to co-locate all the sensor data in one brain. And that opens access to higher intelligence.

9:57 Ihsane Debbache: And that's what we see today in most humanoids that need to be a bit more generalistic. Then they realize, okay, we need to send the raw stereo pairs. And initially, they use our depth model in the main brain, and often it's a Jetson Thor, for example. And then gradually their VLAs or VLM, it learns implicitly how to calculate depth. And then the other depth model we calculate is redundancy, for example.

10:34 Ihsane Debbache: But yeah, so that's the trend where we see our cameras more popular: it's data collection for generalization, and then robots with more compute who need more intelligence, I would say.

10:51 Interviewer: LiDAR or cameras: when does a robot need which?

10:56 Ihsane Debbache: I think the jury is still out on this one, right? I mean, for autonomous cars, we saw that that was the main debate, and up until today, you have both options there. And for humanoids, if we fine-tune that trade-off for humanoids, I think, first, some of the nice-to-have advantages of LiDAR is range, which is less relevant on humanoids, right? And also, the more generalization you need, the more data you need to train intelligent models. And the reason why, for example, Tesla used only cameras is because if you reduce the cost of your vehicle, you can make more of it, and then you can collect more data.

11:51 Ihsane Debbache: So I think we will see both on humanoids. The main decision point is: if the failure is very expensive, then definitely you should keep both cameras and LiDARs, to minimize the failure rates, right? If downtime is more expensive over time, then you should have a LiDAR. Otherwise, if cost is more important per unit, then you can use cameras.

12:23 Ihsane Debbache: But usually, I think the mistake most companies do is premature optimization on the hardware. And what we recommend is: on your first few hundred units, put both, collect the data, and that will be tremendously useful later. Because we have software that can calibrate the camera to LiDAR, and then use LiDAR to tell exactly when the depth of the camera was a bit less accurate. So you know exactly how often, and when, and which scenarios you cannot rely on cameras. Then, if you decide to do the cost-down measure to remove LiDAR, you do make a data-driven decision. And you can have policies for the situations where cameras doesn't work: maybe you put some lighting, or you put some things. But when you start off, it's, I think, a mistake to optimize prematurely on the cost of the hardware. You just want the maximum data, and then with volume, all the costs will come down anyway. So the important thing is to keep the eye on the ball, and at high volume, the costs will be reasonable. And I think a lot of customers are going with that approach.

13:48 Ihsane Debbache: Exactly. Yeah, it can be a teacher. And actually, Tesla bought a lot of LiDARs initially to train the depth on their camera, so to double-check.

14:03 Interviewer: 2030: how will robots see the world?

14:08 Ihsane Debbache: I think, yeah, it's hard to predict. Definitely, I think the specs will move forward tremendously, right? I think we'll see cameras that match the human eyes or that beat human eyes in every spec, right? But I think stereo is still there to stay, because we are confident, because that's what 500 million years of evolution taught us. And it explored LiDAR-similar approaches, branches. It explored ultrasounds and things like that in bats, and still stereo vision dominated.

14:50 Ihsane Debbache: So I think since humanoids often operate in similar environments as humans, we're very confident that the same methodology of perception will stay the same. Now, the spec and the quality of the image sensors and the models behind and everything will evolve tremendously. But I think it will centralize around stereo. And to be more accurate: stereo cameras sending raw sensor data to a main brain, which is going to compute.

15:28 Ihsane Debbache: And today we see a mix of that and stereo with onboard compute. I think that's a bit more for convenience and ease of use. But over time, as the central brain evolves, when the next thing after Thor comes out in 2030, right, it's going to be negligible to compute depth on the host. So you want the lowest cost possible and lightest, low power in all the sensors, and then centralize in one brain. And then, yeah, it will make a lot more sense to have just raw stereo cameras in a lot of places in the robot.

16:12 Interviewer: LiDAR or camera?

16:12 Ihsane Debbache: I don't know, let's see… Camera.

16:16 Interviewer: Humanoids in homes by 2028: myth or reality?

16:18 Ihsane Debbache: Reality.

16:19 Interviewer: The robot eye of the future will be…

16:21 Ihsane Debbache: Stereolabs.

16:22 Interviewer: C-3PO or R2-D2?

16:23 Ihsane Debbache: R2-D2.

16:25 Interviewer: Terminator or WALL-E?

16:26 Ihsane Debbache: WALL-E, definitely.

16:28 Interviewer: Robot future: utopian or dystopian?

16:30 Ihsane Debbache: Utopian.

16:31 Interviewer: Physical AI in 2026, in one word?

16:32 Ihsane Debbache: Scaling.

16:34 Interviewer: Your Munich talk: what's the one takeaway?

16:38 Ihsane Debbache: I would say the most important thing to get right in a first prototype, in a robot, is the sensor. Everything else can evolve later, but it's critical to get the sensor right, because that's what collects your data, and that data is your biggest asset, right? We see it in a lot of robotics companies today: whether they scale and succeed or whether they get acquired, even if they get acquired, their most valuable asset is the data collection. So you need to record data in high quality, synchronized, raw, and with, ideally, the same sensor you plan to use in production. So I will say that's the one thing that's worth investing some effort in.

17:29 Ihsane Debbache: And then, yeah, I'm very excited. We will share some new products that are specially designed for humanoids. So that's something to look forward to. And then, yeah, I don't want to ruin any more of that surprise.

17:46 Interviewer: Robot data: too much, or not enough good data?

17:51 Ihsane Debbache: Yes, there is a lot of low-quality data and not enough high-quality data. I think that is the actual, so I would say: raw image sensor data, without any lossy compression, in a synchronized format, and very importantly, synchronized with the robot joints, so you can later replay: as you move, this is what happened. So images, IMU, robot joints in a synchronized raw sensor feed.

18:24 Ihsane Debbache: Now, that's been a challenge for cameras specifically, because raw image streams are very high bandwidth, right? So we see, for example, that's one reason Stereolabs invested a lot in making GMSL-based cameras. Because for the last few years, gigabit, for example, is usually capped at one gigabit per second. Now you have multi-gig, 2.5, 5, 10 gig, even more, but for the last few years, it's been usually one gig per second, which is very insufficient to get the raw stereo pairs, the raw images, into the host.

19:10 Ihsane Debbache: So for any Ethernet camera, what happens is you compress it, and then it goes to the host, and the host decompresses it to give it to the neural network. But it's lossy compression, it adds latency, and most of those compression algorithms, they are optimized for human viewing. So they compress the image in such a way that humans don't notice the difference, but neural networks lose a lot of the details they rely on to make inference, right?

19:43 Ihsane Debbache: So for us, we use GMSL because it's six gigabits per second bandwidth. It allows us to send the raw streams to the host, and then you can record the high-fidelity data. Now, the challenge is, it's, I'll say, a lot more annoying to work with, with the drivers and things like that. It's less plug-and-play than RJ45, but we do our best to make that easier to work with, as easy as possible.

20:14 Ihsane Debbache: But I think if I was to give one advice, that's one effort worth investing, because you really don't want to be recording lossy compression data and then, years later, regretting, saying: oh, I wish I had higher quality.

20:30 Interviewer: Who do you most want to meet in Munich?

20:34 Ihsane Debbache: In all those conversations, usually, we love to hear from our customers what are their pain points. It always is the most important input to our product roadmap, right? And right now, I think, I'm speaking for humanoids, since it's a humanoid conference, and it is, I think, right now one of our biggest segments, and by far our fastest growing segment is the humanoid market. And we are already a supplier of, I would say, probably the majority of the leading humanoid makers. So we are looking forward to talking with them face to face and exchanging what are their pain points.

21:22 Ihsane Debbache: And especially on the integration and systems engineering side: I think right now, the depth and our software is very popular and very highly regarded. What we need to see with humanoids is: now, as a lot of customers are moving from prototype to production, they need tighter integration. So it's about better form factor of the camera, smaller cables, more flexible, better connectors, and thinking of the whole system as well: how well does it fit in, in a super compact and lightweight manner, in a humanoid.

22:09 Ihsane Debbache: So those are the conversations I think we need to have. And also discussing the failure cases: for example, the humanoids see tremendous shock and vibe, actually more than we see in off-road vehicles or others. So that's a fun new challenge for us, and how we reinforce our calibration to be tolerant to that. And a lot of other conversations, but really listening to customers, understanding their pain points, and then getting back home and executing on that.

22:41 Interviewer: One piece of advice for anyone building robots today?

22:46 Ihsane Debbache: I would say: don't try to innovate everywhere. That's a general advice for most companies, but don't try to do everything yourself. For humanoids, the ecosystem is growing, and you have so many companies working heavily on it that if you try to do, let's say, the mechanical design, and then all the PCBs yourself, and then all the software yourself, and some sensors yourself, and all those things, you'll dilute your attention and your funding across those, and you'll come up with decent solutions, I'm sure, over all of them. But in the meantime, you're gonna have companies focusing only on the actuator, or sensor, or carrier board for the brain, or something else, and then an integrator will have a better solution overall.

23:50 Ihsane Debbache: I think as engineers, we usually are overly, maybe, proud, or we seek the intellectual stimulation, and we think we can innovate everywhere. And also, we look at the market today and we say: oh, there's nothing better, we can't find a solution, so we'll do it ourselves. But by the time we do it, the solution will be there, because everybody's currently working on the solution for every specific problem in humanoids.

24:21 Ihsane Debbache: So I'd say, yeah, talk to people in the ecosystem, try to see what's coming, not what's there right now, but what's coming. And then if you see a gap that not only isn't on the market today, but it's not coming anytime soon, that's an interesting area to innovate on. And it will give you a moat and a technical differentiator. But don't try to do it all, because it'll just be too diluted, I'd say. That's my advice.