Always be Spatial Mapping

This week is Unite 2016 and I'm down in Los Angeles to soak up some sun, get the latest on Unity, and meet great people.

Travelling with a HoloLens, I can't help but capture spatial mapping data wherever I go.  As a developer, I find it's a useful tool for testing when back in the office-- having a library of environment meshes handy can help you test how your holographic application will behave in different situations. So, after checking-in at the hotel and unpacking, out comes the HoloLens. Here's a downloadable scan of my hotel room for you to add to your spatial mapping library. 


Casting holographic shadows in Unity

Can your holograms cast shadows on the real world?  Let's find out. 

In a normal app running on a PC, phone, or even in virtual reality, enabling shadows is as simple as checking a few boxes.  But in mixed reality, things get a bit more complicated because the shadow needs to fall on the real world as well as your holographic creations.  So, what happens if you "just turn on shadows" and deploy to HoloLens?  This.  I promise you this was captured from an actual device, not the emulator, and is representative of what you see. 

Ah, hrm.  That's not gone well. 

Ah, hrm.  That's not gone well. 

Obviously, that's not what we want.  We want shadows on the real world but nothing else.  So, we need a material that renders only the shadow cast upon it.  Unity doesn't provide a shader that'll do that for you by default, so we need to create our own.  Here's what that looks like in Unity. Shadows have been made red for emphasis.

Shadows only screenshot in Unity

Shadows only screenshot in Unity

Normally, casting a shadow on another rendered surface would render the material and textures, only darker, using the other lights in the scene and ambient lighting.  But the real world doesn't have a material in Unity.  We can only add light, not take it away.  Two problems that'd normally be taken care of for us to produce a properly lit (or darker) shadow.  In this example, we'll simply set the shadow color directly as a property on the Holographic Shadow shader.  What color should our holographic shadow be?  It can't be black, because that'd be transparent, but it should be close-- 20, 20, 20 (RGB) is a good place to start.  When deployed to HoloLens we now see shadows on the table, but nothing else. 

Holographic shadows side-by-side with real shadows from real blocks. 

Holographic shadows side-by-side with real shadows from real blocks. 

For consistency, the shader applied to our holograms also sets shadow color directly so that shadows falling on holograms and the real world are the same color.  This is significant for mixed reality as the normal influencers for areas in shadow, such as ambient light color, also impact the color of the holograms themselves.  With them separated, we can control the ambient light level of objects in the scene independent of shadow color and intensity. 

For more convincing shadows we could build a light scanning phase into the room scanning process.  Aligning the light source in-game with the dominant real world light source would cause shadows to fall in the same direction.  The above example is close, but not perfect.  If the shadows fell in the opposite direction the illusion would break immediately.  Sampling (somehow) the correct ambient light level in the room would also make shadows more convincing by matching color and softness. 

Cast shadows from your holograms onto the real world.

Clearly, holographic shadows are possible.  Whether holographic shadows will add value to your project is up to you. With the right level of fine-tuning shadows help connect your holograms to the real world. 

It's worth nothing there is a performance impact associated with shadows.  In the case of HoloBlocks, enabling shadows increased the draw could from 5 to over 30, although it was still able to maintain a steady 60 frames per second.  Your mileage may vary.

We're preparing a standalone package for the Unity Asset Store, follow @DeckTwelve on Twitter and be the first to know when it arrives!



Bug fix and minor feature update for HoloBlocks

HoloBlocks has been updated to resolve some bugs and annoyances, available in the store now.  Here's the change list. 

  1. Fixed a bug that caused the "finish" button to disappear during room scan if you walked behind it. 
  2. Scanning UI now follows you instead of remaining fixed in one place.
  3. Fixed a bug that caused a frame-rate hit when clicking on the spatial mapping mesh during room scanning
  4. Menu can now be moved and resized after placement using the move and resize tools
  5. Menu automatically orients to the user, so you don't have to always face the same direction as when it was initially placed
  6. Menu can no longer be collapsed
  7. Other minor bug fixes

HoloBlocks available now in the #hololens store

Download HoloBlocks now, free on your HoloLens. Build something awesome and share it with us by tweeting a picture to @DeckTwelve

Get the app from the store 

HoloBlocks is still in development and you will notice some rough edges but you'll also have a lot of fun!  Here is a list of known issues in this build to help keep you on the happy path. Bug fixes are in the queue, we'll let you know when they're live. 

  • Frame rate drop occurs if you air tap on spatial mapping mesh while room scan is in progress
  • Walking behind "finish" button will cause it to disappear. Restart app to work around problem.  
  • Menu cannot be repositioned after placement.   

HoloBlocks (formerly Project Blocks) update

Project Blocks is now HoloBlocks

Coming soon to the Windows Store for Microsoft HoloLens for free!  Along with the name change comes some new blocky art, a finalized list of features for the first release, and a healthy dose of lessons learned.  Without further ado, here's what you can expect when HoloBlocks lands on your HoloLens.

Basic building blocks

Challenge your creativity and ingenuity as you build with simple, classic building block shapes including cube, cylinder, semi-circle, triangle, and bridge.  Select a shape from the menu and air-tap to place at the cursor location.  Tap and hold to fine-tune its placement by moving your hand in any direction. 

Pro tip: try placing shapes from different perspectives and see how they behave. 

HoloBlocks menus-- create and modify

HoloBlocks menus-- create and modify

Basic manipulation tools

Move and scale placed blocks to add a bit of dimension and variety to your creations.  Tap and hold on a block to start manipulating, simply complete the tap gesture to finish. 

Fun with physics

Use the push tool to apply some force!  Push a block and watch it tumble, try pushing from different perspectives and see how things change.  Looking for a more drastic change?  Turn gravity off and experience how fun and challenging it'd be to build with blocks on the International Space Station!  Zero-gravity + push = tons of fun.

Zero-gravity physics running amok

Zero-gravity physics running amok


That's it for today, follow us on twitter for more information and lessons learned from development.