Generative Animated Videos with Remotion (and Solandra)

Getting started

Remotion is a React powered library for creating videos. Let's try it out with Solandra, a library for creating 2D graphics from code. We are going to make this:

We get started with yarn create video and choose the default. When that is done also install Solandra with yarn add solandra. That is the setup done! (Well, Remotion requires FFMPEG, see Remotion Docs.)

yarn start to start and take a look. We have something that looks like a video editor, but powered by React. Open ./src/Video.tsx to take a look at what is going on. We see a set of Compositions (videos). In the starter project the main one is ./src/HelloWorld.tsx. This is just a React component. It uses a couple of Remotions specific hooks:

const frame = useCurrentFrame()
const videoConfig = useVideoConfig()

Our first video

Let's try and draw something with Solandra. In the browser go to File, New Composition. And pick a suitable video size and name. This gives you code to copy(!) like:

<Composition
  id="HelloSolandra"
  component={HelloSolandra}
  durationInFrames={150}
  height={1080}
  width={1920}
  fps={24}
/>

Let's create a component and canvas and import:

import { useCurrentFrame, useVideoConfig } from "remotion"

export default function HelloSolandra() {
  const frame = useCurrentFrame()
  const videoConfig = useVideoConfig()

  return <canvas height={videoConfig.height} width={videoConfig.width} />
}

This is actually already working (kind of) but a blank video isn't what anybody wants. Let's at least draw the background.

import { useEffect, useRef } from "react"
import {
  continueRender,
  delayRender,
  useCurrentFrame,
  useVideoConfig,
} from "remotion"
import { SCanvas } from "solandra"

export default function HelloSolandra() {
  const frame = useCurrentFrame()
  const videoConfig = useVideoConfig()
  const delayHandle = delayRender()

  const ref = useRef<any>()
  useEffect(() => {
    const ctx = ref.current.getContext("2d")
    const s = new SCanvas(ctx, videoConfig)
    s.background(210, 80, 50)

    continueRender(delayHandle)
  })

  return (
    <canvas height={videoConfig.height} width={videoConfig.width} ref={ref} />
  )
}

We use a useEffect to setup a fresh SCanvas (Solandra's wrapper) after each render. We use delayRender and continueRender to ensure that Remotion doesn't attempt to immediately render the output. (I suspect in most cases this isn't necessary as puppeteer latency > render time(?) but let's try to be careful.)

Let's actually make a video. Let's use the current frame to change the colour and check we can render this. Just change one line of code (use the frame in the (hue) part of background colour):

s.background(frame + 150, 80, 50)

Let's update package.json to output our video on yarn build, i.e.

	"build": "remotion render src/index.tsx HelloSolandra out/sol-video-1.mp4",

And run it:

yarn build

A few seconds later (8.8 on this m1 Mac) we have our video. A very boring video. But we could draw absolutely anything on each frame and have it nicely assembled. We can also easily scrub the current time (to quickly preview things) and include other things like text in a very easy to compose and configure way.

A more interesting video

Watercolour example

A while ago I looked a simulating watercolours, adapted from Tyler Hobb's essay. I think this could be very nicely animated. Let's go. Again create a composition via the menu (or we could just copy and paste):

<Composition
  id="Watercolour"
  component={Watercolour}
  durationInFrames={600}
  height={1080}
  width={1920}
  fps={30}
/>

Setup

If some of the maths here seems tricky just skip on past it (you can read about more about the approach later).

And create a new component Watercolour by copying our previous code. To simulate the watercolour we want a couple of helpers. Firstly to create a start shape we generate a regular polygon and perturb each point a bit; Solandra has a method for this. Solandra also has the Point2D type for 2D points.

const startShape = (r: number = 0.2, magnitude: number = 0.2): Point2D[] => {
  const at = s.meta.center
  // A regular polygon is close to what we want but that would be closed (start = end) which we don't want here
  return s.build(s.aroundCircle, { at, r, n: 12 }, (pt) =>
    s.perturb({ at: pt, magnitude })
  )
}

We also want to be able to spread out our paint (simulating, crudely, paint flowing out). We do this by starting with two points (i.e. and edge of the original), then generating a new point from this, by taking the mid point, moving perpendicularly outwards:

const newPoint = (a: Point2D, b: Point2D): Point2D => {
  // Perpendicular outwards
  const beta = -Math.PI / 2
  const u = b[0] - a[0]
  const v = b[1] - a[1]
  const m = baseM * s.random()
  const d = s.random()
  const cb = m * Math.cos(beta)
  const sb = m * Math.sin(beta)

  return [a[0] + u * d + (cb * u - sb * v), a[1] + v * d + (sb * u + cb * v)]
}

We can then use this for spreading by applying to a set of points.

const spread = (points: Point2D[]): Point2D[] => {
  var spreaded: Point2D[] = []
  const l = points.length
  for (let i = 0; i < l; i++) {
    const p1 = points[i]
    const p2 = points[i > l - 2 ? 0 : i + 1]
    spreaded.push(p1)
    spreaded.push(newPoint(p1, p2))
  }
  return spreaded
}

Let's actually draw with it:

s.background(40, 70, 90)
s.setFillColor(5, 95, 60, 0.25)

let sh = startShape(0.3, 0.25)
sh = spread(spread(sh))
s.times(5, () => {
  sh = spread(sh)
  s.fill(SimplePath.withPoints(sh))
})

We also need to set a random seed for Solandra:

const s = new SCanvas(ctx, videoConfig, 1);

otherwise we get a new drawing on each frame.

Basic Watercolour

This already produces a decent effect, but by layering and using gradients we get something more compelling.

Making a video

Fancy Watercolour

Replace our drawing code with:

s.setFillGradient(
  new RadialGradient({
    start: s.meta.center,
    end: s.meta.center,
    rStart: 0,
    rEnd: 0.4,
    colors: [
      [0, { h: 5, s: 80, l: 60, a: 0.5 }],
      [1, { h: 45, s: 90, l: 50, a: 0.2 }],
    ],
  })
)

let shapes = [startShape(0.2), startShape(0.3), startShape(0.34)]
shapes = shapes.map((sh) => spread(spread(sh)))

s.times(5, () => {
  shapes = shapes.map((sh) => spread(sh))
  shapes.forEach((sh) => s.fill(SimplePath.withPoints(sh)))
})

We are using Solandra's RadialGradient to specify a nice gradient. The other code is similar, though we now have 3 layers which are each independently spread iteratively. Now let's animate it.

As we have a full programming language at our disposal we can very easily and precisely change anything about the rendering. Let's start with something simple: fading our watercolour in. The following lines control the gradient, let's animate their alpha (opacity) on each frame.

First let's get a relative time in the animation:

const t = frame / videoConfig.durationInFrames

Then use this to update the opacities, note the use of t*t for the outer one; this makes it non-linear, so the outside will take longer to appear painted.

colors: [
    [0, {h: 5, s: 80, l: 60, a: 0.5 * t}],
    [1, {h: 45, s: 90, l: 50, a: 0.2 * t * t}],
],

Let's render, add

"build:water": "remotion render src/index.tsx Watercolour out/watercolour.mp4",

to package.json and run it. You can see the full source code

solandra-remotion

To save on the little bit of boilerplate, here is a SolandraRemotion component. It does the obvious things and provides both a frame and time callback. I might publish it on npm at some point:

import { useEffect, useRef } from "react"
import {
  continueRender,
  delayRender,
  useCurrentFrame,
  useVideoConfig,
} from "remotion"
import { SCanvas } from "solandra"

export default function SolandraRemotion({
  draw,
}: {
  draw: (canvas: SCanvas, frame: number, t: number) => void
}) {
  const frame = useCurrentFrame()
  const videoConfig = useVideoConfig()
  const delayHandle = delayRender()

  const ref = useRef<any>()
  useEffect(() => {
    const ctx = ref.current.getContext("2d")
    const s = new SCanvas(ctx, videoConfig)
    draw(s, frame, frame / videoConfig.durationInFrames)
    continueRender(delayHandle)
  })

  return (
    <canvas height={videoConfig.height} width={videoConfig.width} ref={ref} />
  )
}

with this our first example becomes:

import SolandraRemotion from "./SolandraRemotion"

export default function HelloSolandra() {
  return (
    <SolandraRemotion
      draw={(s, frame) => {
        s.background(frame + 150, 80, 50)
      }}
    />
  )
}