Google “Project Glass” and the Sensics SmartGoggles

Without a doubt, the biggest news of the week of Google’s preview of their “project Glass“, an augmented reality monocular viewer that, in the words, of the Google bloggers, “helps you explore and share your world, putting you back in the moment“.

The focus in this particular incarnation appears to be more of a personal assistant and phone accessory. All the actions shown in the video teaser could be performed on a smart phone, except that the phone is in your pocket and performing these action would typically require taking the phone out. In that respect, the preview is true to the stated goal of getting technology out of the way, though there is a bit of contradiction in that to get technology out of the way you’d have to wear a piece of technology on your glasses.

The user interface shown is simple and non-intrusive and the type of information goes to answering the question ‘what kind of information would I like to have in any particular context?’. No use cases are shown with true image augmentation such as “how would a sofa look in this particular place in the room”. The goggle concept shown is monocular, so no 3D/stereo vision in this particular one, though perhaps one is not needed for the type of user interface actions performed, not to mention that having dual displays would likely double the costs.

User interface at the moment seems to be a head tracker and voice recognition. No launch date has been set yet, nor has a formal commitment to launch been publicly made.

This preview from a tier 1 player, especially one that now has acquired Motorola Mobility, is likely to accelerate the market. What will be the response from Microsoft, Apple and maybe Samsung? Is this a breakthrough concept like an iPad on your head, or is this a nice-to-have accessory like a Bluetooth headset.

It is clear to us at Sensics that Google – and companies like it – could benefit greatly from incorporating our SmartGoggles technology inside such designs and we are excited about the opportunity to make a contribution to the field.

 

Two use cases, three configurations for SmartGoggles

As we continue our march towards taking SmartGoggles to the masses, we speak with consumer electronics leaders about their thoughts, expectations and dreams.

We hear of two prevalent use cases for SmartGoggles:

  • A GamerGoggle which takes games on PCs, consoles and tablets to new levels. In this case, a fully-immersive 3D goggle is typically used indoor in a known environment such as a game room or basement. Some gamers seek increased immersion for the same reasons that caused them to buy multiple large-screen monitors. Some seek active gaming such as in the Kinect case. They are looking for extended use, yet don’t treat the goggles as a monitor replacement.
  • An augmented reality goggle that is used indoor and outdoor. This would typically be standalone or connected to a portable device such as phone or tablet. Augmented reality goggles would overlay information over what you normally see. They could be used as navigation aids, a platform for receiving context-sensitive information on the go and more. These would be for very extended use – much like sunglasses or Bluetooth headsets.

Many people want the best of both worlds – something that can be augmented but also optionally very immersive. “Make it fully immersive when you don’t want to see your wife, but make it semi-transparent so that you don’t trip over your kids”.

As we consider the engineering implications of this, three configurations come to mind: at one end of the spectrum, the classic goggle  essentially serves as a display device. This has to run next to a phone/computer/console and simply provides 3D viewing and simple sensing such as head motion. At the other end of the spectrum, you will find the SmartGoggle that allows you to run substantial applications locally. You’d have to have a powerful CPU and GPU, cloud connectivity and of course all the sensors you might need to make this immersive and interactive. A middle ground is – for lack of a better name – the semi-smart goggle. Something that has sufficient processing power to provide sensory information to a host platform, and perhaps also enough to run applications that are not terribly demanding. You won’t run “Mass Effect 3″ on the semi-smart goggle, but you might stream a 3D movie from Netflix.

Thoughts? Comments? Wishes? Talk back!

Natalia SmartGoggles to be a featured hardware platform at Dare to be Digital competition


Dare to be Digital
is a video games development competition for extremely talented students at Universities and Colleges of Art. Teams of 5 students, usually a mix of artists, programmers and audio, assemble at Abertay University, Scotland, for 9 weeks during June to August to develop a prototype video game, receiving mentoring from industry. The students also receive a weekly stipend, free accommodation at the University’s halls of residence and a modest team budget.

At the end of the competition, the prototypes are displayed at talent showcasing event Dare ProtoPlay. The general public and industry experts get to play and vote for the games. At the Dare awards ceremony, three prizes of £2500 will be awarded to the three highest scoring teams based on the criteria of innovation and creativity, market potential and use of technology. Seven months later the winning teams attend the BAFTA Video Games Awards to compete for the coveted “Ones to Watch Award”.

Sensics Natalia SmartGoggles will be a featured hardware platform at the event, meaning that students will have the opportunity to develop unique games on it.

From the competition Web site: “If you really “Dare to be Digital” then you must consider developing a game for the Sensics SmartGoggles.  These goggles combine high performance 3D virtual reality goggles with two innovative features:  an embedded processor running Android 4.0 and multiple cameras to track hand and movement.  Imagine an Android based game played on a stand-alone 3D head worn device where your head movement is captured so you are truly in the game.  Your hands and arm movement is also tracked so you can include actual movement and gestures in the game and you can even include a peripheral like a sword or gun.  The gamer will be fully mobile and will have an experience unlike any available from current phones, tablets, consoles or PCs.  We provide the hardware the SDK and the support, you create the experience.”

SmartGoggles with a Smart Phones

One demo that was often overlooked at CES was a demonstration of the Natalia SmartGoggles together with a smart phone. For the purposes of the demo, we used a Motorola Droid Razr that was paired with the SmartGoggles. A 3D gaming application was running on the Droid, and the SmartGoggles were used both as a display mechanism – to show the 3D video – as well as an input device which captured the user’s head movements and sent them to the phone. Thus, while SmartGoggles can certainly run applications on the goggle itself, they can be paired with phones, tablets or PCs to work in tandem. This is particularly useful if you already made a content or application investment in the non-SmartGoggles platform.

An interesting twist in the demo was the game navigation itself. Our content partners developed a very nice interface which used the touch screen of the phone to navigate in space.

We believe this is an early sign of some excellent phone-centered applications.


SmartGoggles FAQ

In recent weeks, we received numerous questions about SmartGoggles. Here is a brief summary.

What are SmartGoggles?

SmartGoggles are a new type of virtual reality goggles. SmartGoggles are to goggles what smart phones are to phones. By adding a CPU, an operating system and the ability to install applications, smart phones became much more than just phones. By adding a touch screen, smart phones made it easy to interact with these new features and applications. SmartGoggles follow the same route. By adding a powerful CPU, initially running Android, SmartGoggles can run numerous applications right on the goggles. By adding the unique ability to monitor hand movements, SmartGoggles add a powerful way to interface with the goggles. At the same time, SmartGoggles are still goggles: they have a 3D display, wide field of view, head tracker and more.

What version of Android are you running?

Natalia, the first SmartGoggles product, runs Android 4.0 “Ice Cream Sandwich” on a 1.2 GHz, dual-cord processor.

When can I buy one?

If you are a corporate or government customer, you can buy SmartGoggles now. If you are a gamer or consumer, you should know that Sensics – the SmartGoggles manufacturer – is working to partner with consumer electronics companies to bring a consumer version of the SmartGoggles to market. We hope to have those later this year.

I would like to develop SmartGoggles products. Where do I start?

If you are a manufacturer and are interested in developing SmartGoggles products or licensing some or all of the SmartGoggles technology, please contact Sensics.

Is Natalia battery operated?

Yes, Natalia can run off rechargeable batteries or off a standard 5V power supply. Depending on the application, the batteries typically hold 1.5 to 2 hours before recharge. There are two batteries inside Natalia and they are hot-swappable, meaning that you can replace one while the other one is used to power the device.

What is your screen resolution?

Natalia, the first SmartGoggle product, has dual 1280×1024 (SXGA) OLED displays. It can also run them in 1280×720 (720p) mode

Can the goggles display 3D?

Absolutely. By including two displays – one for each eye – you get the most natural 3D experience and avoid any crosstalk sometimes associated with 3D TVs.

Are SmartGoggles fully immersive or see-through?

SmartGoggles is a technology and an architecture. Various manufacturers can create different versions of SmartGoggles. The initial product, Natalia, is fully immersive. We expect see-through versions to come to market as well.

Can I use SmartGoggles with my phone or PC?

Yes. You can use SmartGoggles in three different modes:

  • Standalone – with the application running on the SmartGoggles.
  • Paired – application runs on a phone or tablet, and the SmartGoggles serve as both a display device as well as a user interface device.
  • Classic – SmartGoggles connect to a PC and function just like classic goggles.

Casual demos at CES 2012

At CES 2012, we had the opportunity to perform quite a few demonstrations of our products to customers, partners and members of the press. As you can see below, they seem quite popular. Here is an example of a Natalia gameplay sequence. Clearly, the player is very engaged and is in the game as opposed to just controlling a character from over the shoulder.

Demonstrations of both existing and new products elicited many smiles, some of which we were allowed to capture:

Multi-modal interfaces and the SmartGoggles

SmartGoggles are virtual reality goggles with a on-board application processor, hand and head position tracking, a built in-microphone and expansion options such as USB. With SmartGoggles, user interface becomes simpler.

For instance, imagine that you are playing a game and you the game asks you to make a simple yes/no decision. With SmartGoggles, you can answer in multiple ways:

  • Say “Yes”, and have the Android processor on board use voice recognition to understand it.
  • Nod your head. Since head movements are monitored, a gesture engine on board can understand the movement.
  • Reach out in space and tap a “yes” area with your hand. Since hand positions are understood by the SmartGoggles, it can tell what you mean.
  • Use an external device such as a wireless game controller to indicate this selection.

What is the best way to do this? It depends. If your hands are busy, you might want to speak or nod. If you like voice interfaces, just say “yes”

 

What smart phones teach us about SmartGoggles

Technically speaking, what made smart phones so much more useful than flip phones? Two main things:

  1. An operating system, such as iOS or Android, running on a sufficiently-strong processor, that allows adding applications to the phone, thereby endlessly expanding its based functionality. At the same time, basic phone functions are kept or enhanced
  2. Enhanced user interface – usually a touch screen – that allows the user to conveniently interact with the phone

Given this, it’s no surprise that we adopted the same approach to make SmartGoggles so much more useful than traditional goggles:

  1. An open operating system, Android 4.0 in the first product, that allows access to numerous applications as well as allows executing them locally – on the goggles. At the same time, maintain traditional goggle functions such as 3D viewing, head tracking and the ability to connect to external video sources.
  2. Enhanced user interface – powered by the ability to sense hand motion – thus allowing easier interaction with the goggles

That’s why we believe that some day soon, most goggles will be smart goggles.

New Unity-Powered Gaming Experience for SmartGoggles

I had the opportunity to try an early version the new Unity-powered game delivered by our partner SideKick for the SmartGoggles platform.

Wow!

In the game, you are a big robot (called “HateBot”) that has serious anger management issues and goes on a destruction tour of a city. The game begins with simple Hatebot Hates and Total Annihilation levels in which sheer destruction is the goal. As the game progresses, your goals will become more complex, your enemies will grow stronger and the time limit for each mission will become stricter. You travel over different portions of the city: The financial district, Chinatown and the elegant shopping district will all be pulverized. Shady bankers, traffic wardens, pushy taxi drivers, lawyers and more: Hatebot must destroy a number of them in each level and the clock is ticking. Those pesky choppers and tanks aren’t helping either.

As you’d expect from a SmartGoggles experience, there are no wires. I just put the goggles on the head and the game started – no need for a game console, or PC or video cables that limit mobility. What makes it particularly exciting is that in this game, I am not controlling Hatebot, I am Hatebot. As I move my head, the scene changes, in full 3D of course. As I physically jump around, Hatebot jumps over tall buildings as soon as I leave the ground.

The version I tested uses a game controller (wireless, of course) to control Hatebot’s hands, but the final version in a few weeks will use our built in-hand tracking so that you will really need to swing your hands for Hatebot to take down buildings and helicopters.

You can see some screen captures at our partners page. Below is a video to how this game looks when played in a Kinect, but to see it in 3D, 360 degree mode, you’ll need to try it on SmartGoggles

If this is sign of things to come, fasten your seat belts.