Wednesday, January 15, 2014

Cel-shaded sprites in Unity3D 4.3 and higher

So a lot of people was requesting details or the actual implementation on how I got the cel-shading effect on the recently added Unity sprites as seen in this video:
Actually, it's not very different to any other cel-shaded materials in Unity. Given no such shader is provided by vanilla Unity, the very first thing to do is to create a new shader for this lighting model. I downloaded the sources for the built-in shaders and started working on my own shader from there.

The shader

Here's the final shader. I'll discuss the details of its implementation below.


Shader "Sprites/CelShadingSpriteShader"
  [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
  _NormalsTex ("Sprite Normals", 2D) = "bump" {}
  _CelRamp ("Cel shading ramp", 2D) = "white" {}
  _Color ("Tint", Color) = (1,1,1,1)
  [MaterialToggle] PixelSnap ("Pixel snap", Float) = 0


  Cull Off
  Lighting On
  ZWrite Off
  Fog { Mode Off }
  Blend SrcAlpha OneMinusSrcAlpha

  #pragma surface surf CustomLambert alpha vertex:vert
  #pragma multi_compile DUMMY PIXELSNAP_ON

  sampler2D _MainTex;
  sampler2D _NormalsTex;
  sampler2D _CelRamp;
  fixed4 _Color;

  struct Input
   float2 uv_MainTex;
   fixed4 color;
  half4 LightingCustomLambert (SurfaceOutput s, half3 lightDir, half3 viewDir, half atten) {
   half NdotL = dot (s.Normal, lightDir);
   half4 c;
   c.rgb = (s.Albedo * _LightColor0.rgb * (tex2D (_CelRamp, half2 (NdotL * 0.5 + 0.5, 0)))) * (atten * 2);
   c.a = s.Alpha;
   return c;
  void vert (inout appdata_full v, out Input o)
   #if defined(PIXELSNAP_ON) && !defined(SHADER_API_FLASH)
   v.vertex = UnityPixelSnap (v.vertex);
   v.normal = float3(0,0,-1);
   v.tangent = float4(-1, 0, 0, 1);
   o.color = _Color * v.color;

  void surf (Input IN, inout SurfaceOutput o)
   fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * IN.color;
   o.Albedo = c.rgb;
   o.Normal = UnpackNormal (tex2D (_NormalsTex, IN.uv_MainTex));
   o.Alpha = c.a;

 Fallback "Transparent/VertexLit"

First of all, we'll need to add new parameters for the shader, that will allow us to pass a normal map and a toon ramp for the shading.

_NormalsTex ("Sprite Normals", 2D) = "bump" {}
_CelRamp ("Cel shading ramp", 2D) = "white" {}

Also, in order for the lighting calculations to be passed to our shader, lighting must be enabled:

Lighting On

The actual surface shader is pretty much copied from the surface shader lighting examples provided by Unity. I opted by specifying a custom Lambertian lighting model where the lighting values are read from the provided toon ramp instead of procedurally computed. Note in the shader preprocessor directives I also specified I'd use my own vertex shader function. This vertex shader function is almost identical to the one in the default sprite shader, though in this case, normal and tangent values for each vertex have to be provided by this function, as Unity's sprite implementation doesn't provide them; normals are pointing towards camera and tangents are towards left. These vectors are in object-space, and are required for the lighting calculations and normals unpacking.

v.normal = float3(0,0,-1);
v.tangent = float4(-1, 0, 0, 1);

Last, but not least, we also extract the surface shader normal component from the normal map in the surface shader:

o.Normal = UnpackNormal (tex2D (_NormalsTex, IN.uv_MainTex));

That would be everything about the shader itself. Let's go onto using the shader in a sprite.

Using the shader

Unfortunately for us, Unity doesn't allow to provide a per-sprite value other than its main texture and its color. There's a way to "override" this and manually add the normal map and the toon ramp per-renderable instead of per-material, but last time I tried, it wasn't being serialized. Hence why in our shader definition _NormalsTex and _CelRamp aren't preceded by the new [PerRenderable] annotation. So, for now, we have to manually create one material for each normals map. Nothing difficult here: new material, use our Sprites/CelShadingSpriteShader shader, and pass the normals and toon ramp textures. Then, we just need to tell the sprite renderer to use this material instead of the default one and we're ready to go.

But, I've hand-drawn my sprites and don't know how to do a normals map for them!

Fear not. I've also done a small tool to generate this map in case you're an artist and don't know how a normals map works.

The sprite normals map generation tool

This is a basic tool to generate a sprite normals map from two "lighting" textures. Whereas the sprite base texture should contain the flat colors (and its details) but no lighting information, these two textures should contain only the lighting information for the sprite; one of them with the horizontal lighting, and the other with the vertical lighting. In most programs like Photoshop or GIMP these three textures would normally be drawn in three different layers: one for the base colors, and each of the other two blended on it using "add" or "screen" as the blending function. However, while normally an artist would paint the lighting only on the lit side, in this case the unlit side must also be shaded.

For instance: the horizontal lighting map is painted like the sun is at the left. Pixels that are completely facing towards the sun (left) are painted in white. Pixels completely opposite to the sun (facing right) are painted in black. Pixels facing towards the artist (neither facing left nor right), in 50% gray, and the rest, blended in between. The vertical lighting map is done the same way, as if the sun is right above the sprite.

Now, the tool itself:


using UnityEngine;
using UnityEditor;
using System.Collections.Generic;
using System.IO;
using System;

namespace Editor {
 public class SpriteNormalsToolWindow : EditorWindow {
  [MenuItem ("Window/Sprite utils/Normal map generator")]
  public static void ShowWindow () {
   EditorWindow win = EditorWindow.GetWindow<SpriteNormalsToolWindow> ("Sprite normals tool", true);
   win.minSize = new Vector2 (100, 100);
  public Texture2D horizontalNormals = null;
  public Texture2D VerticalNormals = null;
  public Texture2D OutputTexture = null;
  public void OnGUI () {
   EditorGUILayout.BeginVertical ();
    horizontalNormals = EditorGUILayout.ObjectField ("Horizontal normals", horizontalNormals, typeof (Texture2D), true) as Texture2D;
    VerticalNormals = EditorGUILayout.ObjectField ("Vertical normals", VerticalNormals, typeof (Texture2D), true) as Texture2D;
    EditorGUILayout.Space ();
    OutputTexture = EditorGUILayout.ObjectField ("Output texture", OutputTexture, typeof (Texture2D), true) as Texture2D;
    if (horizontalNormals == null || VerticalNormals == null)
     GUI.enabled = false;
    if (GUILayout.Button ("Generate normals"))
     GenerateNormalsMap (horizontalNormals, VerticalNormals);
    GUI.enabled = true;
   EditorGUILayout.EndVertical ();
  private void GenerateNormalsMap (Texture2D horizontals, Texture2D verticals) {
   // Create the asset if not existing yet
   if (OutputTexture == null) {
    OutputTexture = new Texture2D (Mathf.Max (horizontals.width, verticals.width), Mathf.Max (horizontals.height, verticals.height), TextureFormat.RGBA32, false);
   // Actual map generation
   Color[] outputPixels = new Color[OutputTexture.width * OutputTexture.height];
   float uNormalized = 0.0f, vNormalized = 0.0f;
   Color ch, cv;
   float r, g, b;
   Vector2 xyNormal;
   Vector3 normal;
   for (int v = 0; v < OutputTexture.height; ++v) {
    vNormalized = (float )v / OutputTexture.height;
    for (int u = 0; u < OutputTexture.width; ++u) {
     uNormalized = (float )u / OutputTexture.width;
     ch = horizontals.GetPixelBilinear (uNormalized, vNormalized);
     r = ch.r * 2.0f - 1.0f;
     cv = verticals.GetPixelBilinear (uNormalized, vNormalized);
     g = cv.g * 2.0f - 1.0f;
     xyNormal = Vector2.ClampMagnitude (new Vector2 (r, g), 0.999f);
     r = xyNormal.x;
     g = xyNormal.y;
     b = (float )Math.Sqrt (1.0f - (double )(r * r) - (double )(g * g)); // z = sqrt (1 - x^2 - y^2)
     normal = new Vector3 (r, g, b).normalized;
     outputPixels[u + v * OutputTexture.width] = new Color (normal.x * 0.5f + 0.5f, normal.y * 0.5f + 0.5f, normal.z * 0.5f + 0.5f, 1.0f);
   OutputTexture.SetPixels (outputPixels);
   OutputTexture.Apply ();
   File.WriteAllBytes (Path.GetDirectoryName (Application.dataPath) + "/Assets/SpriteNormals.png", OutputTexture.EncodeToPNG ());
   AssetDatabase.ImportAsset ("Assets/SpriteNormals.png");
   OutputTexture = AssetDatabase.LoadMainAssetAtPath ("Assets/SpriteNormals.png") as Texture2D;

Remember you must save this file in an editor subfolder inside the assets folder.

The tool will ask for a horizontal and a vertical lighting maps, and if no output texture is provided, will create one by itself.

How this tool works is quite simple. It maps the horizontals onto the red channel and the verticals onto the green channel. Normal maps contain the (normalized) normal vectors packed into color components (each vector component x, y and z range from -1 to 1, and are halved and shifted so they range between 0 and 1, for the r, g and b color components). The z component of these vectors can be reconstructed from the other two using some basic algebra and the pythagorean theorem:

sqrt (x^2 + y^2 + z^2) = 1;
z = sqrt (1 - x^2 - y^2)

Now, access the tool via "Window->Sprite utils->Normal map generator" and drop your lighting maps on their respective fields in the tool. I forgot to mention that, in order for the textures to be queried by the tool, both must be imported with the "Read/Write Enabled" flag enabled (advanced mode). Clicking the "Generate normals" button will create by default a new texture "SpriteNormals.png". This new texture must be imported as a normal map (not a regular texture), and if you plan to alter it later using this tool (for instance, making changes to the lighting maps), the "Read/Write Enabled" flag must be enabled for this one too (though can be disabled if no changes will be made later).

This is it so far. No big deal once you get the idea, as you can see. However, there are some caveats that must be considered.

The cons

These points have already been discussed in this entry at Unity Answers. Some of them have already been covered above. The rest...

  • As the normal maps can't be defined at each sprite, normal maps must be pre-packed manually and, as the texture coordinates between the spite textures and its normals must correlate directly in texture space, we must pack the sprites ourselves. In other words: Unity's built-in sprites atlassing won't work. Lay out the different sprites in a single sheet manually and use that texture to define the sprite fragments in Unity (using the three layers method I talked above is extremely useful for this!).
  • Sprites in Unity aren't meant to be lit, and there seems to be a bug in the lighting calculations. When using scales other than +/-(1, 1, 1) sometimes the sprites are detected as "out of light range" and the light contribution is discarded.
  • User Jessy pointed out that, if batching kicks in for several sprites, each sprites' object space is replaced by a new batch object space. This normally isn't a problem as long as no rotation or non-uniform scaling happens, but if a rotated or mirrored sprite gets batched, its tangents will be corrupted (as they're defined in the shader, not the geometry), and hence its lighting will be wrong too. So far I've had no problems mixing mirrored and non-mirrored sprites (they aren't batched together), but can't ensure you'll be lucky too.

Closing words

You can download the assets here. You're free to use them at will in your own projects, though it would be nice if instead of redistributing/copy-pasting them you would link this post (dropping a note for me to see how you used it or a mention in your game would be über-cool too! :D)

Hope you found it useful!