ARKit and Unity

One of the major updates for the new release for IOS-11 has been ARKit. As of the date IOS 11 is still not released for the public but there’s a beta version out for developers to play around with. If you browse the web there are already some interesting demos circulating, showcasing some of the features…  

AR has been around for some time, but in older versions of IOS you had to rely on computer vision based libraries. One common approach was marker based tracking.

Marker based AR works by recognising and tracking predefined square shaped images. It scales and aligns the 3D-world to the marker and draws the 3D objects in the foreground.

ARKit is only supported by 6S+ and newer models but if you want to build an app to support all models here are some libraries to check out for marker based AR;

ARKit

ARKits new technology is based on Visual Inertial Odometry, it works by finding and tracking points in the camera view while you are moving about. Together with the motion sensor data from the phone it can keep an approximation of your position in the room. From this data ARKit can extract horizontal planes, feature points and lightning information in the scene.

To identify feature points there has to be enough visual information in the view. The more contrast there is in an image the easier it is to find points. For instance a white wall will have no feature points, while a black and white mesh which will have multiple points.

ARKit doesn’t have any depth sensing technology. It can’t occlude objects which means that objects are always drawn on top of the video image. There are speculations that the new Iphone will have a depth sensing camera like the Tango phone and Hololens. Both of them creates a spatial mesh of the environment that is used for occlusion and collision detection for really immersive effects.

Things to keep in mind when designing the app…

  • The plane tracking is not very accurate at finding the edges of a surface and it’s not very fast at detecting planes. You have to move the phone around for some seconds to track the surface. It’s battery draining to track for feature points. Turn off the tracking when you don’t need it.
  • The tracking won’t work when there’s not enough light information and wont work in dark environments.
  • For rendering you have the choice of SceneKit, SpriteKit, Metal or your custom rendering implementation.
  • If you want to use Unity with ARKit you can download a plugin here with some easy to get you started example scenes; https://forum.unity3d.com/threads/arkit-support-for-ios-via-unity-arkit-plugin.474385/

Make a portal hole in the ground with Unity and ARKit

This tutorial is for making a peek hole in the ground with the Unity plugin and ARKit. The final result you can see here;

When you press on the floor a hole opens up and you can have a look inside. In this tutorial we will only be making the hole. To make the inner world is up to you.

For this tutorial you will need;

  • Unity 5.6.1p1 or later
  • IOS 11 or later
  • Iphone 6S+ or later
  • Unity ARKit plugin:

https://forum.unity3d.com/threads/arkit-support-for-ios-via-unity-arkit-plugin.474385/

  • hole.zip file with the hole animation and textures that you can download here: 

http://blog.jayway.com/wp-content/uploads/2017/08/hole_ARKit_Tutorial.zip

 

Preparing the scene

1. Create a new Unity project and load the Unity plugin by dragging it into the assets folder.

2. Open the UnityARKitScene in the assets folder. It already has everything setup for AR and is a good base to start off.

3. We won’t be needing some of the assets, so in the Hierarchy view delete the RandomCube and HitCube objects. You can disable the Point cloud particles and Generate planes, we won’t be needing those. In the inspector view uncheck them. The hierarchy should look like this;

4. Next we are going to prepare our asset. Create a new Prefabs folder in Assets and import the hole.fbx file by dragging it in the Prefabs folder.

5. Fbx files stores both object and animations. Drag the provided hole.fbx file into the scene hierarchy. Make sure you drag it under the root of the hierarchy. The scale is off, but that’s an easy fix. Just select the hole, under model in the hole Import Settings change the Scale Factor from 1 to 2.

Setting up the Animation

If you take a look at the prefab it consists of 4 meshes; innerbox, hole_edge, hole_inside and hole_top. If you select some of the objects in the hole you’ll notice in the Inspector that there’s a skinned mesh renderer applied. The hole has a blend shape animation attached.

If you look in the prefab there’s also an animation clip “Take 001”. It’s the animation for the blend shapes that animates the hole from open to close. Next we are going to connect the animation clip to the hole.

6. Create a new Scenes directory in the Assets folder. In there create a new animator controller; Assets > Create > Animator controller. Rename it to Hole animator controller.

7. Select the hole asset in the view, it has an Animator component attached already. Drag the “Hole animator controller” to the Controller field of the Animator to connect it.

8. If you double click Hole animator controller the Animator view will open up. Go to Prefabs and drag the “Take 001” into the view to connect it to the Animator. Now the animation clip is connected to the hole.

9. Since the hole animation animates from open to close we need to reverse it. Select the “Take 001” and in the inspector change the speed to -0.6. This way it’s going to play from close to open and make the animation a bit slower.

Materials

Next we are going to create the materials for the hole. The insidebox will get a black material just to simulate an emptiness when you peek in. For the hole_inside there’s a dark texture. The hole_top will get an occlusion material that will hide everything inside the box. And the hole_edge that is the outer glowing edge of the hole will get an additive shader that will blend with the environment.    

9. Drag the downloaded textures “edge.png” and “hole.png” to scenes to import them.

10. First we’ll create the black material for the inside of the box. Assets > Create > Material. Rename the material to Innerbox_mat. Select the material and in the Albedo property change the color to black and smoothness to 0. Turn on Emissions. This will give the shader a static black color. Now drag the material onto the “innerbox” to apply it.

11. Next create a new material and name it “Hole_inside_mat”. Select the material and in the Albedo property select the little circle that will let you pick a texture. Pick the “hole” texture. There’s an alpha inside the texture. To be able to render it you have to change the Rendering Mode from Opaque to Cutout. Drag the material onto the hole_inside object.  

Depth mask shader

12. Now we are going to create the Occlusion material for the outer box that will hide everything that renders inside of it. For this we have to create a custom mask shader that will occlude geometry behind. The custom shader will make the geometry appear as if it was invisible by rendering the geometry in different passes and then reversing the order of how the passes are drawn in the view.

Create a new material. Rename it to “hole_top_mat”. Then create > shader > standard surface shader. Rename it to Mask. Double click the shader to open it and in the editor replace everything with this code:

12.We are going to add the Mask shader to the hole_top_mat material. In the inspector view of the material change Shader from standard to Occlusion/Mask. Drag the material on top of the hole_top object and you should see in the viewport how the box seem to disappear.

13. Lastly we will create the glowing edge material. Create a new material. Rename it to “hole_edge_mat”. In the inspector change the shader from standard to Mobile > Particles > Additive. This material will blend with what’s rendered behind. The additive effect will appear as it’s glowing. In the inspector change the texture to hole_ege.png. Apply the new material to the hole_edge obj and your scene should look like this.

Hit test

When we have the animation and materials setup, we are going to setup so that we can place the box in the world. This will be done with help of a hit test script . The Unity ARKit plugin comes with a starter example script for this.

14. In the scene view set the hole as child to the HitCubeParent object. Also delete the HitCube object, we won’t need it.

15. We want to center the origin of the hole to the HitCube parent, so in the Inspector view set the x,y,z properties for the hole position to 0.

16. Add a new script Component to the hole by selecting the hole and in the Inspector view select Add Component; Scripts > UnityEngine.XR.ios >Unity AR Hit Test Example. The script component will have a Hit Transform property. Drag the HitCubeParent from the Scene to connect it to the HitCube. Consequently, whenever the screen is pressed and a collision is detected it will move the HitCubeParent to that position. 

17. Now the box moves the right position when a collision is detected, but we need to trigger the animation to start animate from close to open. To do this we have to open up the Hit test example script. Right click the cog icon next to the Unity AR Hit Test Example Script. And select edit script in the menu.

18. We need to add a reference to the animation in order to start it. Add an animation variable.

19. We’ll instantiate it in the Start function; Add the start function. Since we already have connected the animation clip to the hole object you can get it through getComponent;

20. Next we are going to play the animation whenever we get a touch and collision detection. In the Update function add the play animation call just before return. Set the animation name to the name of our clip “Take 001”, the layer to 0,  and the frame to 0f which will start the animation from frame 0.

Here is what the final UnityARHitTestExample.cs  looks like;

Making the inside of the box

The world inside the portal is empty for now. Making the inner world is not a part of this tutorial. But there are great free assets in the Assets store that you can download and put inside the hole. I used the Vast Outer Space 1.0 which have planets sky textures, particles and other great space assets. For now we are going to put some spheres inside as a temp.

21. Create an empty game object. GameObject > Create Empty and parent it under the hole object in the hierarchy. Rename it to “inner_world”. Inside we’ll create some spheres.Create 2 spheres, scale them and move them so that they are fitted inside the box. Make sure to parent them under the inner_world.

Export to XCode

Now that we have everything setup, let’s export our app and try it out.  

22. Go to file > build settings Select IOS as platform and press Switch platform.

23. Select Player Settings. Change type in your app name in the Bundle Identifier section. And write a message in the camera Usage Description.

24. Build the app. Unity will create an XCode project for you. Open the the project in xcode, select your signing certificate and run the app on a physical phone.

Done!

Press somewhere on the screen, you will see the hole opening up. It might take a while for the plane detection to work and nothing might happen right away. But be patient and move the camera about.

62 Comments

  1. Guille

    Thanks a lot for this tutorial.
    I love it! And now that we have the Unity Arkit Remote plugin things are faster to update and test.

    Keep uploading tutorials like this please.

    Best

  2. Guille

    Also I am using an iPhone 6 and it all works fine, just need to cheat a bit to get it working.

  3. Guille

    Right now Im having some errors with your project but in the past I managed to run apps with real time world tracking on my iPhone 6 through Unity/Vuforia.

    The trick was to Build for iOS selecting UnityARKitScene instead of ARKitRemote/UnityARKitRemote from the Build Settings screen.
    For other ARKit projects I just go to info.plist and from Required device capabilities remove Item 1 – arkit
    And for the Apple ARKit demo app what I did was comment the following code of ViewController.swift:
    if ARWorldTrackingSessionConfiguration.isSupported {
    /
    resetTracking()
    } else {
    /
    let sessionErrorMsg = “This app requires world tracking. World tracking is only available on iOS devices with A9 processor or newer. ” +
    “Please quit the application.”
    displayErrorMessage(title: “Unsupported platform”, message: sessionErrorMsg, allowRestart: false)
    }

    And then uncomment just the following line:
    resetTracking()

  4. Awesome tutorial. It is my lack of coding knowledge that has let me down :( Any chance I could have a scene file to look at? I am so close.

    • Gina Adamova

      Thanks :). Sure thing, here’s the Unity project to download: http://blog.jayway.com/wp-content/uploads/2017/08/Portal-Tutorial.zip

      • Thanks for the upload :) I have the ‘Missing Prefab’ issue. Did you use Blender by any chance? I just want to see the finished Hit test example script (I am code noob). Any chance you could screenshot the script for me?
        One final Q.. I failed a previous test using the Unity ARPlugin. I think it is because I am using the iPhone 6S, not the iPhone 6Splus?
        Many thanks for your help Gina, you are a wizard!

        • Gina Adamova

          Hi Ashley,
          I used Maya for the hole. weird you have the missing prefab issue. Can you see which one is missing?
          If you look in the blogpost part 20. I’ve updated that part to include the whole Hit test script.
          I haven’t tried with the 6S, but it should work with both 6S and 6S+.
          Hope this helps :).

          • Hey Gina,
            I got it working! epic.. thanks for the pointers. I was using the wrong version of unity to start with.
            animation = GetComponent ();
            fixed my error.
            Thank you for your hard work. All is working with iPhone 6s. If you would like me to send a screenshot of the missing prefabs issue please drop me an email.
            Ash

  5. animation = GetComponent (); I mean

  6. Jørgen

    Hi Gina,

    Really loved the tutorial. Built it three times and still could not get it right. I am a newbie to Unity :-)

    I am wondering if it is possible to ask you to do a project for me. It will be based on what you already have done in this tutorial.

    Looking forward to hearing from you.

  7. Airvant

    Hi Gina, your tutorial is cool, especially shader, it’s amazing.
    I had made my portal prototype use your method, it works. But I saw youtube some ARKit portal video that has borderless virtual world, such as this video. https://www.youtube.com/watch?v=5_KPiG7Q5EQ&index=1&list=LLoJsf1OPeQW-hsCXKY3ElCQ
    Do u know how made that, I had tried my best, hope u give some advice.
    Thanks in advance!

    • Gina Adamova

      Thanks! It looks like a skybox map. Basically a 3D texture of the environment on the inner side of the 3d world.
      I haven’t tried it but you probably would also be able to achieve that by triggering the hide/unhide of different parts of the environment based on the camera position, when you enter the door you could display a larger part of the environment.

      • Airvant

        Thanks, Gina!
        I had use trigger hide/unhide other parts of environment which behind the default camera. It works, but not very well. I try to improve it now.
        by the way, your occlusion material is Awesome, it’s key to portal effect!

  8. Valerie

    Hi! I can’t seem to make the box become transparent even though I followed your instructions for the shader…

  9. Jenita Alice

    Hey there!! I have been struggling for the past week to make the HitTest and canvas with buttons to work alongside. Can you help me with them please.

  10. BarryWhite

    Hi Gina,

    getting the following error.

    Shader Custom/Mask uses undefined Queue: ‘Geometry -1’

    any help would be appreciated.

    • Gina Adamova

      Can you open the Mask.shader and exchange the content with this instead:

      Shader “Masked/Mask2” {

      SubShader {
      // Render the mask after regular geometry, but before masked geometry and
      // transparent things.

      Tags {“Queue” = “Background”}

      // Don’t draw in the RGBA channels; just the depth buffer

      ColorMask 0
      ZWrite On

      // Do nothing specific in the pass:

      Pass {}
      }
      }

  11. Preston

    This is great. I’m still trying to get it to work after about 5 tries now. I keep seeing a pink shader after putting the code up above in the comment. “Not supported on GPU”

  12. IMC89

    Hi Gina,
    I have the same error :
    Shader Custom/Mask uses undefined Queue: ‘Geometry -1’
    I take the download file on the post but the error persist.
    any help, thanks.

  13. Dieter

    when I export to Xcode there are 20 errors , It doesnt work u-u

    • Gina Adamova

      What are the errors?

      • Dieter

        Apple Mach-O Linker (ld) Error Group
        : “_GetTrackingQuality”, referenced from:
        : “_GetYUVTexCoordScale”, referenced from:
        : “_GetAmbientIntensity”, referenced from:
        : “_unity_CreateNativeARSession”, referenced from:
        : “_IsARKitSessionConfigurationSupported”, referenced from:
        : “_PauseSession”, referenced from:
        : “_StartSessionWithOptions”, referenced from:
        : “_HitTest”, referenced from:
        : “_StartWorldTrackingSessionWithOptions”, referenced from:
        : “_GetVideoTextureHandles”, referenced from:
        : “_GetLastHitTestResult”, referenced from:
        : “_SetCameraNearFar”, referenced from:
        : “_StartWorldTrackingSession”, referenced from:
        : “_GetARPointCloud”, referenced from:
        : “_IsARKitWorldTrackingSessionConfigurationSupported”, referenced from:
        : “_StartSession”, referenced from:
        : Linker command failed with exit code 1 (use -v to see invocation)

      • Dieter

        Si lo intento pasar al movil entonces aparecen unos 50 errores, no sé porque

  14. Dieter

    I have an iphone 6s and when I put the build project in Xcode there are a lot of errors.
    Can you upload the finish proyect for iphone 6s please ? thanks a lot

  15. Dieter

    I have this error when I add animation = GetComponent ();
    The error :
    using the generic method unityengine.component.getcomponent t ()’ requires 1′ type argument(s)

    please help I want to finish this tutorial u-u

  16. Daniel

    Hi Gina,

    Thanks so much for the tutorial. I am getting the following caution and error:

    private Animator animation; ‘UnityEngine.XR.iOS.UnityARHitTestExample.animation’ hides inherited member ‘UnityEngine.Component.animation’. Use the new keyword if hiding was intended

    and the error is:

    animation = GetComponent (); Using the generic method ‘UnityEngine.Component.GetComponent()’ requires ‘1’ type argument(s)

    I’m trying to teach this tutorial to some students next week and would love if you could help!

    Best,
    Daniel

    • Gina Adamova

      Hi Daniel,

      Check the comment just above yours from Dieter. It seemed like he had the same issue. See if you are missing the Start function. Just add this in the UnityARHitTestExample.

      void Start() {
      animation = GetComponent ();
      }

      If you still have trouble, let me know and I’ll se if I can help you.

      -Gina

  17. Gina:
    This is really amazing work. I am an Earth Science Teacher at a SW Virginia High School, and I would love to use this app in my classroom as a magical “worm hole” when I discuss cosmology in my class. Imagine the students filling in what is in the hole, maybe guided by Hubble images. I am running Unity 2017.1.1f, and cannot get it to work. I am running into the component error, and the scripting reference says the animation Get Component code has been removed for Unity 2017.1.1f. See: file:///Applications/Unity/Unity.app/Contents/Documentation/en/ScriptReference/Component-animation.html.
    I have tried changing the animator variable animation from private to public and setting the animator in the inspector for the hole but that is not working. I also cannot get the hole to line up with the floor or get the box to be transparent. For some reason, the occlusion/mask shader is not in 2017.1.1f. I am going to revert my Unity back to
    Unity 5.6.1p1, and try your project folder.
    Bill

    • Gina Adamova

      Hi Willam,
      What a wonderful idea, to use it in your science class!
      After some updates in Unity, I had to edit the tutorial slightly. If you take a look at the new UnityARHitTest example, you can see a working version. I also have uploaded a test project for you download with animation.
      http://blog.jayway.com/wp-content/uploads/2017/10/ARKit_tutorial_Unity_Update.zip

      Let me know if you run into any other issues, I’ll be glad to help you out!

      The version of Unity I’m using is 2017.1.0, latest XCode 9 to date.

  18. DnSweben

    Hi all,

    I really like this tutorial.
    I just have some small issue’s with locationing the ‘portal’, it’s always… ‘offset’ how can we do this more proper on the ‘floor’ ?
    p.s. the given fix for the error : animation = GetComponent (); Using the generic method ‘UnityEngine.Component.GetComponent()’ requires ‘1’ type argument(s)
    Is not working, but might be due to html tags being stripped out here
    Change animation = GetComponent (); to: animation = GetComponent<Animator>();
    (Change < to )**Just in case it’ll show up as the actual symbol…

    • Gina Adamova

      Glad to hear :)
      You can adjust the positioning by select the “hole” object in the hierarchy and move it around as you please. If you look in the Inspector panel > Transform > Position, The x, y, z should be 0 if you want the hole to appear in the center aligned with the floor.

  19. DnSweben

    Sorry, the symbols are shown :) Please ignore the (Change < to )** part.

  20. Hi, I have Mac with not the newest macOS Sierra (10.12.6). Could it be a reason, why I have dark blue screen, after running this app, with no response? I had updated my iOS on Ipad to 11.0.2 version, and it did not help.
    Does anyone have similar issue with dark blue screen aftern run this project on device?

  21. Did anyone resolve the Occlusion/Mask missing Shader? I have version 2017.2.0b11

  22. Thanks Gina! This is a great tutorial, my first stab at ARKit. I had to change the code in the Mask shader to “Queue” = “Background”, and changed “animation = GetComponent();” to “animation = GetCompoenent():”, then it all worked for me.

    Can you tell me how you learned unity and ARKit? I’m new to both (but have 3D and software background) and I’m looking for the best and fastest way to learn this stuff. Any suggestions on training?

    Thanks so much for this!

    Deb

    • Gina Adamova

      Thanks! I’ll update my tutorial.
      I’m an IOS-developer but I have worked with VFX for many years. For that reason it was easy to pickup for me. If you are comfortable with Unity, you can do a lot without having to know anything about IOS. I learned Unity by reading up online there’s plenty of tutorials as well. For me it’s easier to do handson projects of my own, and read up on issues that comes along the way. Handson experience is always the best. And for Unity and ARKit there’s tutorials online as well. I would recommend also getting to know the basics what is possible and what’s not and how it all works. Which is explained pretty good at the Apple introduction video: https://developer.apple.com/videos/play/wwdc2017/602/

  23. sorry, my last post was incorrect. I changed GetComponent(); to GetComponent\();.

  24. Sorry, it won’t let me enter the proper text here! it’s GetComponent, followed by the less than sign, Animator, greater than sign, ();

  25. Gina:
    Success:
    I finished my first ARKit project and put a spiral galaxy in the hole. I added a rotate script to the galaxy to slowly make it spin. I would like to put the app for free on the Apple App Store if that is ok with you. I have about a dozen students who could then download it and use it in class. I have gotten good feedback from other teachers, and would like to work with you on an AR project. Please email me if you are interested at: wschmachtenberg@gmail.com for more details.
    Bill

  26. Lars Matthäus

    When i test the app, the Portal isnt on the ground. Somehow i stand right into the Portal. Any ideas ?

    • Gina Adamova

      It might be the positioning of the hole object inside the hitcube parent that’s off. Check your transformation panel for the hole and make sure the position is set to 0 for x,y,z.

  27. Bella

    Hi Gina!

    Not sure if I missed something obvious – I think I’m fairly experienced with Unity so many I’m just not paying close enough attention. But I think I’m missing a folding or something. What exactly are you referring to in step one when you say ” load the Unity plugin by dragging it into the assets folder.” I went into the asset store and downloaded the ARKit plugin and went in and grabbed the example scene. Is that correct?
    And also, I don’t have all these files that you’re referring to. I downloaded your hole.zip and imported that, but there’s only the .fbx and two pngs in there.
    Please help!

    Thank you :)

    • Gina Adamova

      Hi Bella,

      Yes you are right, you can just download the plugin from the Asset store and open the example scene and work from there. That’s perfectly fine. One of the early releases you could have downloaded the plugin the way I described.
      The hole.fbx and the two pngs are what you need to finish the tutorial.

      -Gina

  28. Hey Gina, thanks for the super tutorial!

    I’ve been trying it for a while, I was getting similar errors to the other guys, but in the end I managed to publish it to my phone fine; but my issue was that the “Hole” and everything was already there and I was inside it; not sure where I went wrong, so I tried opening your scene from the link you’ve posted above. That works fine up to the final point, where I get this error when Xcode tries to Run it on my phone:

    clang: error: no such file or directory:
    ‘… … /ARkitHole/Libraries/UnityARKitPlugin/Plugins/iOS/UnityARKit/NativeInterface/ARSessionNative.mm’
    clang: error: no input files

    The files are there, so no clue what it’s talking about!

    Sorry, not a coder, so it might be something very simple that I’m missing.

    Thanks again!

    • Gina Adamova

      Sorry for the late reply,

      About the problem with that you appear in the hole, it sounds like the position of the hole is off… it should appear about where you press on the ground. If you select the hole object in the hierarchy if you look at the transformation, it should be 0. for x, y and z.

  29. Hey Gina,
    Was wondering if you know if its possible to combine markers with ARkit?
    Thanks!
    Vincent

    • Gina Adamova

      Hi Vincent,
      Not from within the ARKit framework. But ARKit is based on AW-Foundation video stream. You could use the underlying video to track for markers with a separate library. Like the Vision library.

Trackbacks for this post

  1. Vũ trụ đang ở ngay trong phòng bạn với ứng dụng AR ARKIT Portal Demo - thế giới Ar
  2. ARKit – list of learning resources | Tomáš Mahrík

Leave a Reply