AR transparent video playback in Unity

① Invisible shader Version 1
② Video with alpha channel (not works now)
③ Invisible shader Version 2

This page was last modified on 21 February 2020, the additional 3rd chapter is recommended to read.

There are great examples of mixing animation and graphics, by EYEJACK based on AR technology:

EYEJACK has also released an AR book includes these examples:

I feel it is not such perfect in this period, because of limits of devices and algorithm, but a pleasant way to use AR form in the future, with HMD as Google Glass and etc..

So the problem now is how to achieve this effect with present technology in an easy way that designers with basic coding background are able to do.

① Invisible shader Version 1

The first way works well, I learnt a lot from this tutorial, but the shortage of being tough to locate the video in the ideal place makes me think it might not be a good method. Here is the result:

1. Softwares

1. macOS 10.13.2;
2. Unity 2017.3;
3. Vuforia SDK included in Unity’s installation package;
4. After Effect CC2017.

2. Video

1. In AE, use effect/keying/color range to make the background invisible, and export two videos: one with RGB channel and another with Alpha channel.
2. Create a new composition and put two videos together as this, then render it as mp4

3. AR

1. If you don’t know how to set up Vuforia’s ImageTarget environment, please follow this Vuforia official tutorial, it is quite basic and I don’t want to copy it here.
2. Create a plane in Hierarchy, later it will be used as the platform of video playing, and drag it under the ImageTarget
3. Create a new Material in Project, and drag it on the plane, and also Add Component/Video Player on the plane, and import your video in Project and drag it into the Video Clip which is in the Video Player
4. Create a new Shader in Project, and replace all default codes with these, which I learnt from the mentioned tutorial:

Shader "Custom/Transparent" {
      Properties {
          _Color ("Color", Color) = (1,1,1,1)
          _MainTex ("Albedo (RGB)", 2D) = "white" {}
          _Glossiness ("Smoothness", Range(0,1)) = 0.5
          _Metallic ("Metallic", Range(0,1)) = 0.0
          _Num("Num",float) = 0.5
      }
     SubShader {
          Tags { "Queue"="Transparent"  "RenderType"="Transparent"}
          LOD 200
          CGPROGRAM
      #pragma surface surf NoLighting alpha:auto
        fixed4 LightingNoLighting(SurfaceOutput s, fixed3 lightDir, fixed atten)
          {
              fixed4 c;
              c.rgb = s.Albedo;
              c.a = s.Alpha;
              return c;
          }
      float _Num;
         sampler2D _MainTex;
         struct Input {
              float2 uv_MainTex;
          };
         half _Glossiness;
          half _Metallic;
          fixed4 _Color;
          UNITY_INSTANCING_BUFFER_START(Props)
          UNITY_INSTANCING_BUFFER_END(Props)
      void surf (Input IN, inout SurfaceOutput o)
           {
              o.Emission = tex2D(_MainTex, IN.uv_MainTex).rgb;
                                        if (IN.uv_MainTex.x >= 0.5)
              {
                  o.Alpha = 0;
              }
              else
              {
                  o.Alpha = tex2D(_MainTex, float2(IN.uv_MainTex.x + 0.5, IN.uv_MainTex.y)).rgb;
                                      }
         }
          ENDCG
      }
      FallBack "Diffuse"
      }

5. Select the plane, and change the shader in the material part

4.Test & Build

1. If you have finished that Vuforia’s official tutorial, you should know how to play it with your webcam;
2. If the webcam works well, so you can build it into your mobile device with File/Build and Run;
3. It will automatically run the Xcode software, and this process you can learn from this Unity official tutorial.
4. Done

② Video with alpha channel

The second way is much easier but it has a shortage that it can only work on apple’s devices because it use Apple ProRes 4444 which is only supported on OSX, check Unity’s official documentation about Video transparency support for more information.

1.Video

1. It is the same workflow as the first method, in the After Effect you will have the result that video has an transparent background;
2. Then you have to export the video as QuickTime format with RGB+Alpha channels
3. On the same window, there is a button named Format Options, click it and in the new window you have to choose Apple ProRes 4444 as the Video Codec
4. Render it and you will have an Unity-Ready transparent video

2. AR

1. Import it into Unity and in the Inspector click Keep Alpha and click Apply
2. Drag it into Video Clip of Video Player Component of Plane which has been mention in the first method, and create a new Material for the plane which shader should be selected as Transparent/VertexLit with Z
3. Done

③ Invisible shader Version 2

Make sure you are using a green screen or any single-color-background video.

1.ChromaKeyShader.shader

Shader "Custom/ChromaKeyShader" {
 		Properties {
                _MainTex ("Base (RGB)", 2D) = "white" {}
                _MaskCol ("Mask Color", Color)  = (1.0, 0.0, 0.0, 1.0)
                _Sensitivity ("Threshold Sensitivity", Range(0,1)) = 0.5
                _Smooth ("Smoothing", Range(0,1)) = 0.5
        }
        SubShader {
                Tags { "Queue"="Transparent" "IgnoreProjector"="True" "RenderType"="Transparent"}
                LOD 100
                ZTest Always Cull Back ZWrite On Lighting Off Fog { Mode off }
                CGPROGRAM
                #pragma surface surf Lambert alpha
 
 				struct Input {
                    float2 uv_MainTex;
                };
 
                sampler2D _MainTex;
                float4 _MaskCol;
                float _Sensitivity;
 				float _Smooth;
 
                void surf (Input IN, inout SurfaceOutput o) {
                        half4 c = tex2D (_MainTex, IN.uv_MainTex);
 
                        float maskY = 0.2989 * _MaskCol.r + 0.5866 * _MaskCol.g + 0.1145 * _MaskCol.b;
						float maskCr = 0.7132 * (_MaskCol.r - maskY);
 						float maskCb = 0.5647 * (_MaskCol.b - maskY);
 
						float Y = 0.2989 * c.r + 0.5866 * c.g + 0.1145 * c.b;
 						float Cr = 0.7132 * (c.r - Y);
 						float Cb = 0.5647 * (c.b - Y);
 
 						float blendValue = smoothstep(_Sensitivity, _Sensitivity + _Smooth, distance(float2(Cr, Cb), float2(maskCr, maskCb)));
						o.Alpha = 1.0 * blendValue;
						o.Emission = c.rgb * blendValue;               
                }
                ENDCG
        }
        FallBack "Diffuse"	
}

2.Settings

1. Create a gameobject to display the video;
2. Add Component>Video Player, and drag your video into Video Player>Video Clip;
3. Apply the material using ChromaKeyShader.shader on the gameobject;
4. In “Mask Color”, select the background color of the video which should be invisible;
5. Adjust “Threshold Sensitivity” and “Smoothing” to the best solution as you like.