Meta Tech Podcast

72: Multimodal AI for Ray-Ban Meta glasses

Meta Tech Podcast

Show Notes

In this episode of the Meta Tech Podcast, host Pascal sits down with Shane, a research scientist at Meta, to explore the cutting-edge research behind Ray-Ban Meta glasses. Shane shares insights from his seven-year journey at Meta, where he focuses on computer vision and multimodal AI within the Wearables AI organization. Tune in to learn how Shane's team is pioneering foundational models for Ray-Ban Meta glasses, tackling unique challenges, and pushing the boundaries of AI-driven innovation. Discover how multimodal AI is transforming user experiences and get a glimpse into the future of wearable technology. Whether you're an engineer, a tech enthusiast, or simply curious about the latest advancements, there is something for everyone in this episode. Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/. Links AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model - https://arxiv.org/abs/2309.16058

Be My Eyes Programme: https://www.forbes.com/sites/stevenaquino/2024/10/11/inside-the-be-my-eyes-meta-collaboration-and-the-allure-to--impact-humanity/

Meta Open Source on Threads: https://www.threads.net/@metaopensource

CacheLib: https://cachelib.org/

Meta's AI-Powered Ray-Bans Are Life-Enhancing for the Blind - Wall Street Journal: https://www.wsj.com/tech/ai/metas-ai-powered-ray-bans-are-life-enhancing-for-the-blind-3ae38026

Timestamps Intro 0:06

OSS News 0:56

Introduction Shane 1:30

The role of research scientist over time 3:03

What's Multimodal AI? 5:45

Applying Multimodal AI in Meta's products 7:21

Acoustic modalities beyond speech 9:17

AnyMAL 12:23

Encoder zoos 13:53

0-shot performance 16:25

Iterating on models 17:28

LLM parameter size 19:29

How do we process a request from the glasses? 21:53

Processing moving images 23:44

Scaling to billions of users 26:01

Where lies the optimisation potential? 28:12

Incorporating feedback 29:08

Open-source influence 31:30

Be My Eyes Programme 33:57

Working with industry experts at Meta 36:18

Outro 38:55

More from Meta Tech Podcast

Oct 15, 2026

How do you add end-to-end encryption to a messenger that people have already been using for years, without taking away the message history they expect to be there? Emma Connor works on the team behind Labyrinth, Messenger's encrypted storage protocol. We get into where the encryption keys actually live, which turns out to be nowhere near Meta, and why a six-digit PIN is far harder to attack than six digits ought to be. Emma also explains the hardware security modules that Meta deliberately locked itself out of, what happens when you lose a device and want it cut out of your account, how key transparency lets you verify who you are talking to without having to trust us, and why Labyrinth 1.1 moved to post-quantum cryptography. All of this and more is discussed in this episode. Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/. Links Opaque Rust Crate - https://crates.io/crates/opaque-ke

Jul 29, 2026

Pascal is going on Recharge for August so there will be no regular interview episode. However, we want to hear your questions for our mailbag episode. Send them to us on Instagram (https://www.instagram.com/metatechpod) or Threads (https://www.threads.com/@metatechpod) or via good old email to [email protected].

Jul 13, 2026

What changes for you as a developer when the device moves from your pocket to your face? The battery shrinks by an order of magnitude, the thermal budget is whatever won't make your nose uncomfortable, and your mental model of "one process, one device" goes out the window. Rey and Dmitrii talk us through their custom power monitors, how they've extended Perfetto to give a coherent trace across a multi-device system, why robotic arms make surprisingly good QA engineers, and what it takes to catch the kind of regression that quietly costs you thirty minutes of call time. All of this and more is discussed in this episode. Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/. Links Migrating Data Ingestion Systems at Meta Scale: https://engineering.fb.com/2026/05/12/data-infrastructure/migrating-data-ingestion-systems-at-meta-scale/

Jun 9, 2026

Your smart glasses run all day on a battery narrower than your pinky finger and building it required reinventing how batteries are made. In this episode, Pascal talks to Karthik and Myuran, the engineers behind Meta's steel can battery technology, to explore why traditional pouch cells couldn't cut it for the ultra-slim temple arms of AI glasses like Meta Ray-Bans and the Oakley Vanguards. Tune in to learn how Meta designed, built, and scaled the batteries powering your glasses, wristbands, and cases from first prototype to mass production. Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/. Links SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems - https://engineering.fb.com/2026/05/26/ml-applications/silvertorch-index-as-model-new-retrieval-paradigm-recommendation-systems/ Timestamps Intro and News 0:06

May 8, 2026

You've probably spotted those little circles of your friends' faces popping up on Facebook Reels. They look simple enough, but building them was a proper engineering challenge. In this episode, Pascal chats to Joseph and Subasree about Friend Bubbles, a feature that surfaces which of your close friends have been watching and reacting to the same Reels as you. We get into the details of how prefetching keeps things snappy without wrecking scroll performance, why the team's ML model had to move from survey-based friend rankings to real-time interaction signals, and the surprising discovery that showing fewer bubbles actually made the whole feature click. If you've ever underestimated a "simple" feature, this one's for you. Got feedback? Send it to us on Threads (https://threads.net/@metatechpod), Instagram (https://instagram.com/metatechpod) and don't forget to follow our host Pascal (https://mastodon.social/@passy, https://threads.net/@passy_). Fancy working with us? Check out https://www.metacareers.com/. Links https://engineering.fb.com/2026/03/18/ml-applications/friend-bubbles-enhancing-social-discovery-on-facebook-reels/

See all episodes →