Raymarching with Swift and Metal for Shadertoy-like Graphics

Shadertoy is a well known website for creating and sharing shader driven graphics. It works in the browser for a tight feedback loop between changes and results. You use WebGL to write shader code. Can we do something similar with native Swift code for iOS and MacOS? Yes, at least in terms of creation.

Basics

It is remarkable easy to get a fully shader rendered app. (Actually it is more remarkable when you learn more and know how tedious many of the technologies are to set up normally.) Let's create a new Multi-platform App Swift project in Xcode. Replace the main view in ContentView.swift with:

import SwiftUI

struct ContentView: View {
  let startDate = Date()

  var body: some View {
    GeometryReader { gp in
      TimelineView(.animation) { ctx in
        Rectangle()
          .ignoresSafeArea()
          .colorEffect(ShaderLibrary.raymarch(
            .float2(gp.size.width, gp.size.height),
            .float(startDate.timeIntervalSinceNow)
          ))
      }
    }
  }
}

#Preview {
  ContentView()
}

That is basically all the Swift code we will write. Now it won't work yet as this ShaderLibrary.raymarch doesn't exist. Let's create a new Metal file by hitting cmd + n, e.g. RayMarch.metal and replace the code with:

#include <metal_stdlib>
#include <SwiftUI/SwiftUI_Metal.h>
using namespace metal;

[[ stitchable ]] half4 raymarch(float2 position, half4 currentColor, float2 size, float time) {
  float c = (1 + sin(time)) / 2;
  return half4(c,c,c,1);
}

Now go back to your Swift file. Ideally you should use Xcode's split view controls to get the Metal file alongside the Swift file. Start a preview with cmd + alt + enter and hit cmd + alt + p to restart if necessary. If everything is working it will should show a full screen going backwards and forwards between white and black.

Live Coding (sort of)

Now you should be able to make changes to the low level shader code and see the results in the preview pretty quickly. It isn't going to be quite as nice as Shadertoy, though Xcode will offer better code completion.

Okay but how did that code work?

Let's look at the Swift stuff first. It combines a few things, but each are individually quite simple.

• We use GeometryReader to get the size of the view.
• We use TimelineView(.animation) to redraw the view every frame.
• Rectangle is just a simple view that will take all available space offered. We use ignoresSafeArea to make sure it fills the whole screen.
• .colorEffect is a SwiftUI modifier that allows us to apply a shader to a view.
• ShaderLibrary.raymarch gives us access to our shader function. We pass in the data to it. One thing might seem a little odd, the .float2 and so on. This is because Metal expects certain data types which aren't the ones we'll have been using. float2 is just a 2D float vector (and so on for 3 and 4).

Now on the Shader side we do need to declare the parameters consistently. Metal is not Swift, indeed this can be pretty confusing as some things will look similar. It is a low level shader language. We will be writing fragment shaders, which are massively parallel programs that run for each pixel you on a screen.

We declare a matching function and signature. For colorEffect we actually get a position and currentColor argument out of box. We can then add further arguments.

[[ stitchable ]] half4 raymarch(float2 position, half4 currentColor, float2 size, float time) {

Now to draw something we first take the time and then take sin of it (this will give us a value between -1 and 1). We must return a 4d color (the half4). This is comprised of a red, green, blue and opacity value. For shaders we work with values between 0 and 1. To start with we just set all the color values to the same value. This will give us a greyscale color. And we set the opacity to 1 (fully opaque).

  float c = (1 + sin(time)) / 2;
  return half4(c,c,c,1);

Experiment with changing some of these c to specific values between 0 and 1.

Make it more interesting

Add one new line at the start of our shader. This gives us a uv value (like an x,y graph) that goes between -1 and 1 in vertical (and proportionately in horizontal).

  float2 uv = (position * 2.0 - size) / size.y;

We can now use this to add some gradients:

  float2 uv = (position * 2.0 - size) / size.y;
  float c = (1 + sin(time)) / 2;
  return half4(1 + half2(uv),c,1);

The uv values will include negatives and even go above 1. This is why the gradient is cut off. We can fix like:

  return half4(half2(1 + uv)/2,c,1);

One of the nice things we can do with shaders is combine vectors and scalars without needing to convert between them. uv is a 2d vector and 1 is a scalar. We can add them together and then divide by 2 to get a 2d vector again.

Raymarching

Raymarching is an awesome technique for creative rendering. It is actually quite simple yet can create complex, interesting visuals.

We basically think of an camera being at a particular point. We shoot out rays from that point through a 'screen'. For each ray we calculate the distance to the nearest object. We then move along the ray by that distance and repeat until we are very close to the object or we have gone very far away. Let's take that in 2 parts.

We call ro our 'origin' (as our location we imagine ourselves 3 units back in space). Then we shoot a ray through our screen. But we already have created a uv value for each pixel. So we can use that. A good enough approximation for now is to use uv as the x and y values of our ray direction. For z let's just use 1. We can then normalise (this means make the length of the vector 1) the vector.

  float3 ro = float3(0,0,-3);
  float3 rd = normalize(float3(uv, 1));

Okay so we have some rays. What about the distance? We need a (signed) distance function. To keep things simple let's just use a circle at (0,0,0) of radius 1. What is the distance from some point to this? Well it is actually trivial in Metal to get this. We can use length(p) - 1.

Okay let's actually do the raymarching. Setup a function to handle the distance:

float distanceToBall(float3 p) {
  return length(p) - 1;
}

And to our existing shader:

[[ stitchable ]] half4 raymarch(float2 position, half4 currentColor, float2 size, float time) {
  float2 uv = (position * 2.0 - size) / size.y;

  float3 ro = float3(0,0,-3);
  float3 rd = normalize(float3(uv, 1));

Let's do the raymarching. Start (remember this happens for every pixel) at t = 0. Now we iterate up to 80 times. We move forwards in the direction of our ray by the distance to the ball.

We repeat.

If we get very close we stop (d < 0.001). Or if very far off we give up.

  float t = 0;

  for(int i = 0; i < 80; i++) {
    float3 p = ro + rd * t;
    float d = distanceToBall(p);
    t += d;

    if(d < 0.001 || t > 100) break;
  }

Finally we need to return something from the shader to color the pixel. If t is high let's just use black or half4(0,0,0,1). If it is close let's try half4(sin(t),0,0,1) (this is a sneaky way of adding a little gradient based on t, but we'll see more interesting ways of picking colors later).

  if(t > 100) return half4(0,0,0,1);

  return half4(sin(t),0,0,1);
}

This gives us this. Which at first glance might seem trivial. But we are doing something pretty amazing. From very basic mathematics and in about 50 lines of code we are rendering a 3D object, pixel by pixel on our GPUs.

Let's animate it by changing the origin with time:

  float3 ro = float3(0,0,-3.1 + cos(time));

and make the colors more interesting:

  return half4(sin(t),cos(t),t,1);

Taking It Further

With not that much more code we can do something like this (video link):

Video version (note the actual rendering is much sharper than the 4k version of the video):

Let's update our Swift code to use ShaderLibrary.raymarchB and create a new shader in our existing file:

[[ stitchable ]] half4 raymarchB(float2 position, half4 currentColor, float2 size, float time) {
  float2 uv = (position * 2.0 - size) / size.y;

  float3 ro = float3(sin(time),0,-3);
  float3 rd = normalize(float3(uv, 1));

  float t = 0;

  int j = 2;
  for(int i = 0; i < 80; i++) {
    float3 p = ro + rd * t;
    float d = map(p, time);
    t += d;

    j = i;
    if(d < 0.001 || t > 10) break;
  }

  if(t > 100) return half4(0,0,0,1);

  return half4(palette(t * 0.9 + time * 0.01 + float(j) * 0.01, half3(0.5), half3(0.5), half3(1.0), half3(0., 0.2, 0.2)), 1);
}

This is similar to before except we have made our coloring fancier using Inigo Quilez's palette helper and reduced the t value at which we give up. We've also renamed the distance function to map and added a j value to keep track of how many iterations we've done (which we will also use in the coloring).

For the distance we do a little more. Instead of using p directly we take the the fractional part of if in all directions (this allows for an infinite repetition of our object). We also introduce a rotation of the shape to make it more dynamic (actually we rotate our ray, but the effect is the same; many raymarching approaches apply transformations to the ray rather than the actual object).

float map(float3 p, float time) {
  float3 q = float3(fract(p.xy) - 0.5, mod(p.z, 0.5) - 0.25);

  q.xy = q.xy * rot2D(time * 0.5 + floor(p.z));

   float box = sdBox(q, float3(0.1));
   return box;
}

We then take a distance. We use a box, which is a little more complex than a sphere, but not much (and there is a known standard way to formulate). You can learn more about this and many other 3D SDFs.

float sdBox(float3 p, float3 b) {
    float3 q = abs(p) - b;
    return length(max(q, 0.0)) + min(max(q.x, max(q.y, q.z)), 0.0);
}

The above depends on a couple of helpers; the first rotates in two dimensions and the second takes the modulo of two floats.

matrix<float, 2> rot2D(float angle) {
    float s = sin(angle);
    float c = cos(angle);
    return matrix<float, 2>(c, -s, s, c);
}


float mod(float a, float b) {
  return a - b * floor(a/b);
}

But the complete shader for this infinitely repeating, animated, psychedelic raymarching is only about 50 lines of code. And you could easily tie details of the shader rendering to your App's state (it could show loading progress, visualize music or any other dynamic data).

Source code

The entire, remarkably short code is in this Gist.

Learn More

In terms of Swift stuff

• Paul Hudson has several tutorials on shaders in Swift(UI), for example simple shaders and has a library of shaders you can apply to your apps.

There are vastly more resources on shaders using WebGL/GLSL than Metal, but often they are quite easy to translate.

• Inigo Quilez's articles are the definitive reference for Raymarching, SDFs and various other creative coding techniques.
• He also has some astonishing live coding videos of creating incredible scenes using similar techniques, for example this intricate, naturalistic landscape.
• kishimisu has a fantastic pair of video tutorials on shaders.