Microsoft Research has unveiled Project Zanzibar, a brand-new sensing platform that can locate and communicate with objects through a flexible mat that also senses a user’s touch and hover gestures. (Think drawing imaginary shapes in the air.)

Hailing from a team of US and UK Microsoft researchers, Project Zanzibar lets kids play with toys, cards and blocks as their actions come alive on-screen. The mat does not provide its own display, but instead connects to existing screens via USB or Bluetooth.

The mat combines non-contact capacitive sensing with Near Field Communication (NFC) to enable multi-touch input as well as physical object manipulation and control. Objects are tagged with NFC stickers, and the platform tracks their position, movement and orientation. Hardware designs also let sensing extend beyond the mat’s surface, which means the platform can detect the presence and order of stacked objects.

Each physical object connected to the platform has a globally unique ID, which enables an object to “learn” as it collects and stores data over time. Children can also introduce their own playthings into Project Zanzibar, manipulating toys and props on the mat to control the corresponding avatars on-screen.

The platform’s Toybox scenario uses sound effects and audio dialog to complement physical play. The combination and proximity of different characters (like pirates and naval officers) triggers lines of dialog, while the interaction between characters and toy pieces leads to sound effects (such as the firing of a cannon or climbing to the top of a tower).

Through the sensing mat’s Movie Maker scenario, additional props could change the setting of a virtual scene (a palm tree could create a tropical setting, for example). Camera and light source objects, meanwhile, affect the framing and lighting of a scene.

Project Zanzibar also features a number of learning scenarios, with a particular focus on language and coding. The full scope of the platform will be presented at CHI 2018 in Montreal later this month.

Project Zanzibar arrives as interactive gaming toys continue to struggle. Year-over-year sales of interactive gaming toys in the US fell by 66% in December 2017, according to market research firm The NPD Group. Slumping sales of toys-to-life properties are timed with recent departures of industry heavy-hitters like LEGO and Disney from the category. In October, Warner Bros. Interactive Entertainment, TT Games and The LEGO Group announced they were ceasing production of further expansion packs for LEGO Dimensions, and Disney announced in 2016 that it had shuttered its Infinity toys-to-life series.

The gap between physical and digital words is far from closing, though. Earlier this year, for example, Nintendo announced Labo, a new line of cardboard-based, interactive DIY experiences for kids that work with the company’s bestselling Switch console system and Joy-Con controllers.