ARKit and Unity

One of the major updates for the new release for IOS-11 has been ARKit. As of the date IOS 11 is still not released for the public but there’s a beta version out for developers to play around with. If you browse the web there are already some interesting demos circulating, showcasing some of the features…  

AR has been around for some time, but in older versions of IOS you had to rely on computer vision based libraries. One common approach was marker based tracking.

Marker based AR works by recognising and tracking predefined square shaped images. It scales and aligns the 3D-world to the marker and draws the 3D objects in the foreground.

ARKit is only supported by 6S+ and newer models but if you want to build an app to support all models here are some libraries to check out for marker based AR;

ARKit

ARKits new technology is based on Visual Inertial Odometry, it works by finding and tracking points in the camera view while you are moving about. Together with the motion sensor data from the phone it can keep an approximation of your position in the room. From this data ARKit can extract horizontal planes, feature points and lightning information in the scene.

To identify feature points there has to be enough visual information in the view. The more contrast there is in an image the easier it is to find points. For instance a white wall will have no feature points, while a black and white mesh which will have multiple points.

ARKit doesn’t have any depth sensing technology. It can’t occlude objects which means that objects are always drawn on top of the video image. There are speculations that the new Iphone will have a depth sensing camera like the Tango phone and Hololens. Both of them creates a spatial mesh of the environment that is used for occlusion and collision detection for really immersive effects.

Things to keep in mind when designing the app…

  • The plane tracking is not very accurate at finding the edges of a surface and it’s not very fast at detecting planes. You have to move the phone around for some seconds to track the surface. It’s battery draining to track for feature points. Turn off the tracking when you don’t need it.
  • The tracking won’t work when there’s not enough light information and wont work in dark environments.
  • For rendering you have the choice of SceneKit, SpriteKit, Metal or your custom rendering implementation.
  • If you want to use Unity with ARKit you can download a plugin here with some easy to get you started example scenes; https://forum.unity3d.com/threads/arkit-support-for-ios-via-unity-arkit-plugin.474385/

Make a portal hole in the ground with Unity and ARKit

This tutorial is for making a peek hole in the ground with the Unity plugin and ARKit. The final result you can see here;

When you press on the floor a hole opens up and you can have a look inside. In this tutorial we will only be making the hole. To make the inner world is up to you.

For this tutorial you will need;

  • Unity 5.6.1p1 or later
  • IOS 11 or later
  • Iphone 6S+ or later
  • Unity ARKit plugin:

https://forum.unity3d.com/threads/arkit-support-for-ios-via-unity-arkit-plugin.474385/

  • hole.zip file with the hole animation and textures that you can download here: 

http://blog.jayway.com/wp-content/uploads/2017/08/hole_ARKit_Tutorial.zip

 

Preparing the scene

1. Create a new Unity project and load the Unity plugin by dragging it into the assets folder.

2. Open the UnityARKitScene in the assets folder. It already has everything setup for AR and is a good base to start off.

3. We won’t be needing some of the assets, so in the Hierarchy view delete the RandomCube and HitCube objects. You can disable the Point cloud particles and Generate planes, we won’t be needing those. In the inspector view uncheck them. The hierarchy should look like this;

4. Next we are going to prepare our asset. Create a new Prefabs folder in Assets and import the hole.fbx file by dragging it in the Prefabs folder.

5. Fbx files stores both object and animations. Drag the provided hole.fbx file into the scene hierarchy. Make sure you drag it under the root of the hierarchy. The scale is off, but that’s an easy fix. Just select the hole, under model in the hole Import Settings change the Scale Factor from 1 to 2.

Setting up the Animation

If you take a look at the prefab it consists of 4 meshes; innerbox, hole_edge, hole_inside and hole_top. If you select some of the objects in the hole you’ll notice in the Inspector that there’s a skinned mesh renderer applied. The hole has a blend shape animation attached.

If you look in the prefab there’s also an animation clip “Take 001”. It’s the animation for the blend shapes that animates the hole from open to close. Next we are going to connect the animation clip to the hole.

6. Create a new Scenes directory in the Assets folder. In there create a new animator controller; Assets > Create > Animator controller. Rename it to Hole animator controller.

7. Select the hole asset in the view, it has an Animator component attached already. Drag the “Hole animator controller” to the Controller field of the Animator to connect it.

8. If you double click Hole animator controller the Animator view will open up. Go to Prefabs and drag the “Take 001” into the view to connect it to the Animator. Now the animation clip is connected to the hole.

9. Since the hole animation animates from open to close we need to reverse it. Select the “Take 001” and in the inspector change the speed to -0.6. This way it’s going to play from close to open and make the animation a bit slower.

Materials

Next we are going to create the materials for the hole. The insidebox will get a black material just to simulate an emptiness when you peek in. For the hole_inside there’s a dark texture. The hole_top will get an occlusion material that will hide everything inside the box. And the hole_edge that is the outer glowing edge of the hole will get an additive shader that will blend with the environment.    

9. Drag the downloaded textures “edge.png” and “hole.png” to scenes to import them.

10. First we’ll create the black material for the inside of the box. Assets > Create > Material. Rename the material to Innerbox_mat. Select the material and in the Albedo property change the color to black and smoothness to 0. Turn on Emissions. This will give the shader a static black color. Now drag the material onto the “innerbox” to apply it.

11. Next create a new material and name it “Hole_inside_mat”. Select the material and in the Albedo property select the little circle that will let you pick a texture. Pick the “hole” texture. There’s an alpha inside the texture. To be able to render it you have to change the Rendering Mode from Opaque to Cutout. Drag the material onto the hole_inside object.  

Depth mask shader

12. Now we are going to create the Occlusion material for the outer box that will hide everything that renders inside of it. For this we have to create a custom mask shader that will occlude geometry behind. The custom shader will make the geometry appear as if it was invisible by rendering the geometry in different passes and then reversing the order of how the passes are drawn in the view.

Create a new material. Rename it to “hole_top_mat”. Then create > shader > standard surface shader. Rename it to Mask. Double click the shader to open it and in the editor replace everything with this code:

12.We are going to add the Mask shader to the hole_top_mat material. In the inspector view of the material change Shader from standard to Occlusion/Mask. Drag the material on top of the hole_top object and you should see in the viewport how the box seem to disappear.

13. Lastly we will create the glowing edge material. Create a new material. Rename it to “hole_edge_mat”. In the inspector change the shader from standard to Mobile > Particles > Additive. This material will blend with what’s rendered behind. The additive effect will appear as it’s glowing. In the inspector change the texture to hole_ege.png. Apply the new material to the hole_edge obj and your scene should look like this.

Hit test

When we have the animation and materials setup, we are going to setup so that we can place the box in the world. This will be done with help of a hit test script . The Unity ARKit plugin comes with a starter example script for this.

14. In the scene view set the hole as child to the HitCubeParent object. Also delete the HitCube object, we won’t need it.

15. We want to center the origin of the hole to the HitCube parent, so in the Inspector view set the x,y,z properties for the hole position to 0.

16. Add a new script Component to the hole by selecting the hole and in the Inspector view select Add Component; Scripts > UnityEngine.XR.ios >Unity AR Hit Test Example. The script component will have a Hit Transform property. Drag the HitCubeParent from the Scene to connect it to the HitCube. Consequently, whenever the screen is pressed and a collision is detected it will move the HitCubeParent to that position. 

17. Now the box moves the right position when a collision is detected, but we need to trigger the animation to start animate from close to open. To do this we have to open up the Hit test example script. Right click the cog icon next to the Unity AR Hit Test Example Script. And select edit script in the menu.

18. We need to add a reference to the animation in order to start it. Add an animation variable.

19. We’ll instantiate it in the Start function; Add the start function. Since we already have connected the animation clip to the hole object you can get it through getComponent;

20. Next we are going to play the animation whenever we get a touch and collision detection. In the Update function add the play animation call just before return. Set the animation name to the name of our clip “Take 001”, the layer to 0,  and the frame to 0f which will start the animation from frame 0.

Here is what the final UnityARHitTestExample.cs  looks like;

Making the inside of the box

The world inside the portal is empty for now. Making the inner world is not a part of this tutorial. But there are great free assets in the Assets store that you can download and put inside the hole. I used the Vast Outer Space 1.0 which have planets sky textures, particles and other great space assets. For now we are going to put some spheres inside as a temp.

21. Create an empty game object. GameObject > Create Empty and parent it under the hole object in the hierarchy. Rename it to “inner_world”. Inside we’ll create some spheres.Create 2 spheres, scale them and move them so that they are fitted inside the box. Make sure to parent them under the inner_world.

Export to XCode

Now that we have everything setup, let’s export our app and try it out.  

22. Go to file > build settings Select IOS as platform and press Switch platform.

23. Select Player Settings. Change type in your app name in the Bundle Identifier section. And write a message in the camera Usage Description.

24. Build the app. Unity will create an XCode project for you. Open the the project in xcode, select your signing certificate and run the app on a physical phone.

Done!

Press somewhere on the screen, you will see the hole opening up. It might take a while for the plane detection to work and nothing might happen right away. But be patient and move the camera about.

15 Comments

  1. Guille

    Thanks a lot for this tutorial.
    I love it! And now that we have the Unity Arkit Remote plugin things are faster to update and test.

    Keep uploading tutorials like this please.

    Best

  2. Guille

    Also I am using an iPhone 6 and it all works fine, just need to cheat a bit to get it working.

  3. Guille

    Right now Im having some errors with your project but in the past I managed to run apps with real time world tracking on my iPhone 6 through Unity/Vuforia.

    The trick was to Build for iOS selecting UnityARKitScene instead of ARKitRemote/UnityARKitRemote from the Build Settings screen.
    For other ARKit projects I just go to info.plist and from Required device capabilities remove Item 1 – arkit
    And for the Apple ARKit demo app what I did was comment the following code of ViewController.swift:
    if ARWorldTrackingSessionConfiguration.isSupported {
    /
    resetTracking()
    } else {
    /
    let sessionErrorMsg = “This app requires world tracking. World tracking is only available on iOS devices with A9 processor or newer. ” +
    “Please quit the application.”
    displayErrorMessage(title: “Unsupported platform”, message: sessionErrorMsg, allowRestart: false)
    }

    And then uncomment just the following line:
    resetTracking()

  4. Awesome tutorial. It is my lack of coding knowledge that has let me down :( Any chance I could have a scene file to look at the code? I am so close.

  5. Awesome tutorial. It is my lack of coding knowledge that has let me down :( Any chance I could have a scene file to look at? I am so close.

    • Gina Adamova

      Thanks :). Sure thing, here’s the Unity project to download: http://blog.jayway.com/wp-content/uploads/2017/08/Portal-Tutorial.zip

      • Thanks for the upload :) I have the ‘Missing Prefab’ issue. Did you use Blender by any chance? I just want to see the finished Hit test example script (I am code noob). Any chance you could screenshot the script for me?
        One final Q.. I failed a previous test using the Unity ARPlugin. I think it is because I am using the iPhone 6S, not the iPhone 6Splus?
        Many thanks for your help Gina, you are a wizard!

        • Gina Adamova

          Hi Ashley,
          I used Maya for the hole. weird you have the missing prefab issue. Can you see which one is missing?
          If you look in the blogpost part 20. I’ve updated that part to include the whole Hit test script.
          I haven’t tried with the 6S, but it should work with both 6S and 6S+.
          Hope this helps :).

          • Hey Gina,
            I got it working! epic.. thanks for the pointers. I was using the wrong version of unity to start with.
            animation = GetComponent ();
            fixed my error.
            Thank you for your hard work. All is working with iPhone 6s. If you would like me to send a screenshot of the missing prefabs issue please drop me an email.
            Ash

  6. animation = GetComponent (); I mean

  7. Jørgen

    Hi Gina,

    Really loved the tutorial. Built it three times and still could not get it right. I am a newbie to Unity :-)

    I am wondering if it is possible to ask you to do a project for me. It will be based on what you already have done in this tutorial.

    Looking forward to hearing from you.

Trackbacks for this post

  1. Vũ trụ đang ở ngay trong phòng bạn với ứng dụng AR ARKIT Portal Demo - thế giới Ar

Leave a Reply