CGWiki

Appearance

Javascript ​

Or the more honest title. 'React Three Fiber, and how little Javascript can you get away with learning'.

Overview ​

I'd avoided JavaScript for literally decades, buying all the internet hate that that it was messy, slow, best avoided.

Well it's not going away, and with my current job being more web focused, took the plunge. It's nowhere near as bad as I expected, just a couple of new symbols and idioms to learn. If you know enough vex to write a wrangle or two, JavaScript is fine.

My particular focus with it is on React Three Fiber, a React wrapper around Threejs. Seemed scary at first, but hasn't taken long to get mildly productive with it. React introduces a bunch of unfamiliar terms like useRef and useEffect and other things, but for the most part just pulling apart examples from other people is a good way to get started.

Codesandbox was the big revelation for me in terms of learning. It's a combined text editor like VScode, a web development framework like npm, and a browser-in-a-browser. There's several similar offerings available (jsfiddle, shadertoy, codepen etc ), but is the one favoured by the R3F scene. While its possible to use tools like npm and vscode natively on your machine to do dev (and as you get deeper into this stuff its a natural progression), but for just starting out, codesandbox is a no-brainer; find an example of something you want to do, fork it, code away.

Language tricks ​

I started this as a way to remember how to port my python knowledge over to javascript, but I've since realised it's just a summary of the great react docs at https://beta.reactjs.org/ . I'm going on the asssumption that people skimming this are looking to get into react three fiber, which means learning react, which means you may as well just jump over there and read their docs!

List comprehension is a map ​

My python coding is 99% list comprehensions, its the one bmx trick I know, I use it everywhere:

python
mylist = [1,2,3,4]
[ x*2 for x in mylist ]
## returns [2,4,6,8]
mylist = [1,2,3,4]
[ x*2 for x in mylist ]
## returns [2,4,6,8]

Javascript uses a similar thing, map.

javascript
mylist = [1,2,3,4]
mylist.map( x => x*2)
// returns [2,4,6,8]
mylist = [1,2,3,4]
mylist.map( x => x*2)
// returns [2,4,6,8]

This combines a couple of ideas into typically terse but practical js. Map lets you call a function on the elements of an array. The full form would look like this:

javascript
function doubleIt(x) {
    return x*2
}

mylist = [1,2,3,4]
mylist.map(doubleIt)

// returns [2,4,6,8]
function doubleIt(x) {
    return x*2
}

mylist = [1,2,3,4]
mylist.map(doubleIt)

// returns [2,4,6,8]

If the function is short, you can can use the arrow => operator to define an in-line function. This is a pretty standard thing in react.

You can write a longer function, you just have to remember to use curly braces and a return. Watch where the regular braces match (after map, end of the function), and the curly braces (after the arrow, before the final brace):

js
mylist = [1,2,3,4]
mylist.map(n => {
    n += 1
    n *= 2
    n = n + ' is the magic number'
    return n
})
// returns ['4 is the magic number', '6 is the magic number', '8 is the magic number', '10 is the magic number']
mylist = [1,2,3,4]
mylist.map(n => {
    n += 1
    n *= 2
    n = n + ' is the magic number'
    return n
})
// returns ['4 is the magic number', '6 is the magic number', '8 is the magic number', '10 is the magic number']

List comprehension is a filter ​

My other standard trick with list comprehensions is to add in some filter logic:

python
mylist = [1,2,3,4]
[ x * 2 for x in mylist if x>=3]
## returns [6,8]
mylist = [1,2,3,4]
[ x * 2 for x in mylist if x>=3]
## returns [6,8]

In javascript you run the filter and map as separate operations:

js
let mylist = [1,2,3,4]
mylist = mylist.filter(i => i>=3)
mylist = mylist.map(x => x*2 )
// returns [6,8]
let mylist = [1,2,3,4]
mylist = mylist.filter(i => i>=3)
mylist = mylist.map(x => x*2 )
// returns [6,8]

It's possible to concatenate these operations together, but readability suffers:

js
mylist = [1,2,3,4].filter(x => x>=3).map(x => x*2 )
// mylist is [6,8]
mylist = [1,2,3,4].filter(x => x>=3).map(x => x*2 )
// mylist is [6,8]

I've seen it written like this, which is a little better? I guess? Rising tone at end of sentence??

js
mylist = [1,2,3,4]
    .filter(x => x>=3)
    .map(x => x*2 )
// mylist is [6,8]
mylist = [1,2,3,4]
    .filter(x => x>=3)
    .map(x => x*2 )
// mylist is [6,8]

Map a dictionary or object ​

R3F will load glb's as an object of key:value pairs. These can't be directly mapped, but you can get a list of the keys, then map on that.

This is one way to do that, codesandbox here: https://codesandbox.io/s/glb-iterate-duqwyv?file=/src/Numbers.js

js
// get a list of keys
const keys = Object.keys(nodes)

// from here you could do a simple map of just return the keys:
keys.map( k => k)

//or use this key to retrieve the matching values:
keys.map( k => nodes[k])

// you can also get back an index if needed:
keys.map( (k,i) => 'index of '+k+' is '+i)

// Here's a full show off thing that builds a list of
// transformed nodes in a glb with a purple material:
const list = keys.map( (k,i) => {
return (
  <group key={i} position-x={i*0.05}>
    <primitive name={k} object={nodes[k]}>
      <meshStandardMaterial color={'mediumpurple'} />
    </primitive>
  </group>
)
})
// get a list of keys
const keys = Object.keys(nodes)

// from here you could do a simple map of just return the keys:
keys.map( k => k)

//or use this key to retrieve the matching values:
keys.map( k => nodes[k])

// you can also get back an index if needed:
keys.map( (k,i) => 'index of '+k+' is '+i)

// Here's a full show off thing that builds a list of
// transformed nodes in a glb with a purple material:
const list = keys.map( (k,i) => {
return (
  <group key={i} position-x={i*0.05}>
    <primitive name={k} object={nodes[k]}>
      <meshStandardMaterial color={'mediumpurple'} />
    </primitive>
  </group>
)
})

Spread operator or the 3 dots thing ​

I thought it was code snippets online implying 'and you fill in the rest', but no, its a thing. Several JS examples online will have function calls that look like

js
function(...args)
function(...args)

What on earth does that mean? Well imagine you had a particularly stupid function called add4, that took 4 explicit arguments:

js
function add4(a,b,c,d) {
  return a + b + c + d;
}

add4(3,2,5,3)
// returns 13
function add4(a,b,c,d) {
  return a + b + c + d;
}

add4(3,2,5,3)
// returns 13

Now say you had an array with 4 numbers, and you wanted to send that to add4. If you're being a good person you'd declare the arguments explicitly:

js
myvals = [1,2,3,4]
add4(myvals[0],myvals[1],myvals[2],myvals[3])
// returns 10
myvals = [1,2,3,4]
add4(myvals[0],myvals[1],myvals[2],myvals[3])
// returns 10

But what if you were lazy, and put trust in javascript's generally pretty loose and forgiving behaviour? You'd get this:

js
add4(myvals)
// returns '1,2,3,4undefinedundefinedundefined'
add4(myvals)
// returns '1,2,3,4undefinedundefinedundefined'

It its trying to add the array [1,2,3,4] to 3 undefined variables, its not happy.

The ... syntax does the implicit unpacking for you, 'spreading' the array elements into the matching slots. Very handy.

js
add4(...myvals)
// returns 10
add4(...myvals)
// returns 10

R3F ​

React Three Fiber is a pretty amazing React wrapper around ThreeJS. Lets you write web 3d almost as easy as writing html, with a ton of other nice things. That first link there goes straight to their collection of amazing examples, click on any of them to be taken to a codesandbox setup you can immediately customise, super cool, or scroll down a little further to the basic examples if you're after cleaner templates.

The Three JS Journey tutorial site is a great way to get started, well worth the money.

R3F terminology ​

The terms can be a little confusing at first, here's a glossary:

  • Threejs - a javascript library for doing 3d on the web. Its great, but a little low level for my tastes (like people who wanna vex everything)
  • React - a popular framework for the web, originally developed by facebook, now open source. One of its key aims is to simplify complex javascript development and be able to write clean easily accessed components, so that stuff that would otherwise be arcane javascript boilerplate can almost be treated like html. So in the way you use <b>bold</b> and <i>italic</tags>, you could have an image carousel reduced to a single <Carousel> call, or a twitch stream window just <Twitch>.
  • React Three Fiber - a React wrapper on Three. Combining the two above ideas, rather than using javascript box = new box() style coding, you write 3d in a html style way, so <Box>, and <camera> and <directionalLight> . Seem obtuse at first, but it actually makes a lot of creating 3d on the web much lighter and more fluid.
  • Drei - like Sidefx labs for R3F. The maintainers of R3F have strong ideas about what should be in the core R3F system, but are also aware there's lots of helpers and nice-to-haves that people want. All those things are pushed into the Drei library (drei is german for 'three', geddit?), and there's an incredible amount of cool stuff in it. You want sparkles? <Sparkles />. You want easy motion trails? <Trails />. You want to load a gltf? Environment maps? A premade presentation stage with soft lighting and an orbitable camera? It's all there.
  • Pmdrs - the artist and programmer collective behind both R3F and Drei.

Make colours match across Figma and R3F ​

This may be specific to our pipeline, but we had issues where 3d geometry that was derived from figma looked different in R3f:

  • textures looked desaturated and pale
  • colours thats should be identical in textures and vertex colours looked very different.

Turns out the vertex colours on our end required gamma correction, so running them through a gamma of 0.45 made them match the textures.

The bigger issue was the deafult tonemapper used by R3F. While it's filmic and great, my gut feeling is the shoulder of the tonemapper is too aggressive, and starts desaturating and softening off colours well before they should start getting that filmic overexposure vibe.

To turn it off and revert back to classic crappy srgb, add the flat={true} prop to the canvas:

html
<Canvas flat={true}>
<Canvas flat={true}>

This will cause colours to oversaturate and highlights to go weird, but diffuse surfaces will match their colour much better to the unlit textures, very important for logo work. In my case when using the Stage helper, I could bring everything down to a balanced level by adjusting the stage intensity:

html
<Stage intensity={0.5}  >
<Stage intensity={0.5}  >

Modifying glb materials ​

To debug the colour stuff, I needed to swap between standard and unlit materials ( meshStandardMaterial and meshBasicMaterial in threejs terms).

I was using the gltfjsx converter to get a tag representation of the glb, to tweak this to my requirements was easy enough; comment the material prop, make the mesh tag have open and close sections, and insert a material tag in the middle. So at a simplified level, this:

html
<mesh
   geometry={nodes.box.geometry}
   material={nodes.box.material}
/>
<mesh
   geometry={nodes.box.geometry}
   material={nodes.box.material}
/>

becomes

html
<mesh geometry={nodes.box.geometry}>
 <meshStandardMaterial color={'green'} />
</mesh>
<mesh geometry={nodes.box.geometry}>
 <meshStandardMaterial color={'green'} />
</mesh>

To make it use vertex colours from the mesh, swap color={} for 'vertexColors':

html
<mesh geometry={nodes.box.geometry} >
<meshStandardMaterial vertexColors />
</mesh>
<mesh geometry={nodes.box.geometry} >
<meshStandardMaterial vertexColors />
</mesh>

Textures are assigned to materials via the map={} prop. I console logged out the material, expanded out its tree to get the map, and directly referred to it:

html
<mesh geometry={nodes.logo.geometry} >
<meshStandardMaterial map={nodes.logo.material.map}/>
</mesh>
<mesh geometry={nodes.logo.geometry} >
<meshStandardMaterial map={nodes.logo.material.map}/>
</mesh>

finally if the image had an alpha channel, that can be enabled with the transparent prop:

html
<mesh geometry={nodes.logo.geometry}>
<meshStandardMaterial transparent={true} map={nodes.logo.material.map}/>
</mesh>
<mesh geometry={nodes.logo.geometry}>
<meshStandardMaterial transparent={true} map={nodes.logo.material.map}/>
</mesh>

Transparent images ​

Make sure to add

html
transparent={true}
transparent={true}

To any materials using texture with alpha, otherwise you'll get odd colours bleeding over where the alpha is meant to be 0.

Except you'd probably assume this would show the image where the alpha is 1, and reveal the plain untextured material where the alpha is 0 right. No. It will make the entire object transparent where the alpha is 0.

I was surprised that there's no built in mechanism to just treat the image like a decal. This leaves you with a few options:

  • Make a duplicate of your mesh that just has a plain untextured material. This is the easiest, but its inefficient. If you're only showing a handful of lightweight meshes, this isn't such a big deal.
  • Use Drei's Decal function, however this is for projecting a texture, not for working in the uv space of an image (and under the hood its also duplicating your mesh anyway)
  • Write your own shader to do the required alpha compositing. This is what people in native threejs are doing.
  • Use a layered shader builder like Lamina

As I said, I'm just duplicating meshes for now.

Occlusion Roughness Metalness ​

Codesandbox example: https://codesandbox.io/s/roughness-dq8ul6?file=/src/App.js

ORM maps appear to be hardcoded in threejs for MeshStandardMaterial, so don't try and be cute with material networks in Houdini, Three will ignore you.

So if you want to use occ/rough/metal, pack it so that

Red = Occlusion
Green = Roughness
Blue = Metalness

  • Occlusion assumes it's being treated as a lightmap bake, so wil only work if you have a second uv set. Otherwise it's ignored.
  • The map values will multiply against the roughness and metalness properties.
  • If roughness and metalness aren't specified then they go to a default of roughness=1 (fine) and metalness=0 (not fine, your map will never be seen!)
  • This means you can adjust the strength of your roughness/metalness by tweaking the matching property as an overall multiplier, handy.
  • You can use seperate occ/rough/metal maps, thats fine, but its a waste of resources if three is just loading a single channel from each map anyway.

Here's how it looks if you're defining by hand:

jsx
const orm = useTexture('./test_orm.png')
const col = useTexture('./col.png')
return (   
   <mesh>
     <boxGeometry args={[1, 1, 1]} />
     <meshStandardMaterial
        map={col}
        roughness={0.7}
        metalness={0.2}
        aoMap={orm}
        roughnessMap={orm}
        metalnessMap={orm}
      />
</mesh>
)
const orm = useTexture('./test_orm.png')
const col = useTexture('./col.png')
return (   
   <mesh>
     <boxGeometry args={[1, 1, 1]} />
     <meshStandardMaterial
        map={col}
        roughness={0.7}
        metalness={0.2}
        aoMap={orm}
        roughnessMap={orm}
        metalnessMap={orm}
      />
</mesh>
)

Flipped UVs on GLB meshes ​

Annoyingly for some reason uv's will be upside down for glbs in R3F. I was going back to Houdini and flipping uv's there, only to find after the fact that there's a built in threejs option, which you can access from R3F. Just add map-flipY={true} :

jsx
<meshStandardMaterial map={texture} map-flipY={true} />
<meshStandardMaterial map={texture} map-flipY={true} />

Load and playback a GLB ​

I always forget the basic version, here's a template:

jsx
import  {useGLTF, useAnimations } from '@react-three/drei'

export const MyGlb = () => {
  const url = '/mycool.glb'
  const { scene, animations } = useGLTF(url)
  const { actions, names } = useAnimations(animations)

   useEffect(() => {
    const anim = actions[names[0]]
    anim.play()
  })

    return (
     <primitive object={scene} />
     )
}

// and then over in your canvas element
<MyGlb />
import  {useGLTF, useAnimations } from '@react-three/drei'

export const MyGlb = () => {
  const url = '/mycool.glb'
  const { scene, animations } = useGLTF(url)
  const { actions, names } = useAnimations(animations)

   useEffect(() => {
    const anim = actions[names[0]]
    anim.play()
  })

    return (
     <primitive object={scene} />
     )
}

// and then over in your canvas element
<MyGlb />

Control GLB playback timing from a time source ​

Clever Mike Lyndon to thank here. Normally you'd just go theanim.play() and let it run its own thing, but what if you need the playback locked to an external source, like the global frame variable provided by remotion?

javascript
useEffect(() => {
    const anim = actions[names[0]]
    anim.play().paused = true
    anim.time = 0.25+(frame-10) / 25.0
})
useEffect(() => {
    const anim = actions[names[0]]
    anim.play().paused = true
    anim.time = 0.25+(frame-10) / 25.0
})

Clone a glb with animation ​

R3F doesn't like when you refer to a glb twice. The drei Clone function lets you do what it says and clone whatever you need, but when you do this, you lose the connection to the original animation.

Paul from pmndrs pointed to the glb example they provide that has a replicated glb playing different animations, I slimmed it down a little to just the bits I needed. Some tricky stuff involving three functions and mystery helpers, but as I understand it the core of it is you have to make a deep clone of the full scene early on, and be able to identify the copies with ref's.

The other esoterica here is the skinnedMesh call. I'm using a houdini generated glb, where I had a top level joint called 'root', and a skinnedmesh called 'mesh'. Lots of head scratching while looking at the other codesandbox reference, in the end you need to point to use both a <primitive> call that points to 'nodes.root', and in skinnedmesh refer to 'nodes.mesh.geometry' and 'nodes.mesh.skeleton':

javascript
import { useGLTF, useAnimations } from '@react-three/drei'
import { useEffect, useMemo} from 'react'
import { useGraph } from "@react-three/fiber"
import { useCurrentFrame } from 'remotion'
import { SkeletonUtils } from "three-stdlib"

export const MultiGLB = () => {
  const url = '/theglb.glb'
  const { scene, animations } = useGLTF(url)
  const clone = useMemo(() => SkeletonUtils.clone(scene), [scene])
  const { nodes } = useGraph(clone)
  const { ref, actions, names } = useAnimations(animations)

  useEffect(() => {
    const anim = actions[names[0]]
    anim.play()
  })

  return (
    <group ref={ref}  >
        <primitive object={nodes.root}  />
        <skinnedMesh
            geometry={nodes.mesh.geometry}
            skeleton={nodes.mesh.skeleton}
        />
    </group>
  )
}
import { useGLTF, useAnimations } from '@react-three/drei'
import { useEffect, useMemo} from 'react'
import { useGraph } from "@react-three/fiber"
import { useCurrentFrame } from 'remotion'
import { SkeletonUtils } from "three-stdlib"

export const MultiGLB = () => {
  const url = '/theglb.glb'
  const { scene, animations } = useGLTF(url)
  const clone = useMemo(() => SkeletonUtils.clone(scene), [scene])
  const { nodes } = useGraph(clone)
  const { ref, actions, names } = useAnimations(animations)

  useEffect(() => {
    const anim = actions[names[0]]
    anim.play()
  })

  return (
    <group ref={ref}  >
        <primitive object={nodes.root}  />
        <skinnedMesh
            geometry={nodes.mesh.geometry}
            skeleton={nodes.mesh.skeleton}
        />
    </group>
  )
}

useHelper ​

edit: well dur, this is the first thing covered on the hooks documentation for R3f. https://docs.pmnd.rs/react-three-fiber/api/hooks

Three has the abilty to show gizmos on lights/cameras/objects like a 3d app would, R3F wraps that up with the userHelper hooks. I struggled with this, each time I tried to use it I'd have R3F shout at me that I was using it in the wrong place.

I think what I was doing wrong is that it needs to be called from within the canvas component. I kept calling it from the function that was returning a canvas component, but that seems to be one level too far away.

The fix is to wrap whatever you're wanting a useHelper for in a component, then call that in your canvas. Ie, this won't work:

js
import { useHelper } from "@react-three/drei"
import * as THREE from 'three'

export const App = () => {
  const camRef = useRef()
  useHelper(camRef, THREE.CameraHelper, 1) // outside the canvas, dur
  return (
    <Canvas>
      <orthographicCamera ref={camRef}  />
    </Canvas>
  )
}
import { useHelper } from "@react-three/drei"
import * as THREE from 'three'

export const App = () => {
  const camRef = useRef()
  useHelper(camRef, THREE.CameraHelper, 1) // outside the canvas, dur
  return (
    <Canvas>
      <orthographicCamera ref={camRef}  />
    </Canvas>
  )
}

but if you create a component that wraps both the camera and the helper, and call that in the canvas, its happy:

js
import { useHelper } from "@react-three/drei";
import * as THREE from 'three'

const CamWithHelper = () => { // wrap camera and helper together
  const camRef = useRef()
  useHelper(camRef, THREE.CameraHelper, 1)
  return (
    <orthographicCamera ref={camRef}  />
  )
}

export  const App = () => {
  return (
    <Canvas >
        <CamWithHelper /> // helper and camera in the correct scope, woo
    </Canvas>
  )
}
import { useHelper } from "@react-three/drei";
import * as THREE from 'three'

const CamWithHelper = () => { // wrap camera and helper together
  const camRef = useRef()
  useHelper(camRef, THREE.CameraHelper, 1)
  return (
    <orthographicCamera ref={camRef}  />
  )
}

export  const App = () => {
  return (
    <Canvas >
        <CamWithHelper /> // helper and camera in the correct scope, woo
    </Canvas>
  )
}

Remotion ​

Template with javascript ​

Remotion is an amazing react system that lets you control other react components from a timeline, including r3f. It defaults to typescript, but there's a hello world js template available in the demo pack. Run

npm init video

as per usual, choose the second demo.

NPM ​

If you've dabbled a bit in html and then asked 'whats npm?', it'd be the equivalent of someone asking 'i used povray in 1995, whats houdini?'. Fair bit of ground to cover, also I'm an idiot, but these are the high level strokes:

Despite everyone hating javascript, google and others invested a crazy amount of work into making it super fast. As it got more fancy on the browser, someone realised 'maybe we could use javascript for more general things?', and so some wise folk made the javascript engine a standalone application, like a python interpreter but for javascript.

People started experimenting with this, found that it was good. If you're writing a bunch of javascript for the browser, why change languages for the server side? Why not make a web server in javascript? This morphed into its own platform, Node.js.

Node.js become so popular that it required a way to manage all the things people were making for it, plugins, downloadable modules and things for both server and browser. This is called the node package manager, or NPM

Npm can manage packages, start and stop them, getting them installed, get things initialised, all sorts, its a combined compiler/installer/query thing.

If you've used pip install, npm is kind of the same thing, but more general purpose, and has become a very handy one-command instruction for getting all sorts of web dev (and other things) done. You've heard of React? The usual way to get started is to build a react project with NPM. Threejs and react three fibre? Build them with NPM. Tools for graphing? Making nice gradient colour palettes? Generating random pet names? Reading csv's? Getting yahoo stocks? Invariably there's a pre-made npm module to do that.

So if you hear about something like vitepress, they'll usually have something like this on their getting started page:

bash
npm add vitepress
npx vitepress init
npm run docs:dev
npm add vitepress
npx vitepress init
npm run docs:dev

The first line is installing vitepress into your local node modules.

The second starts a text based wizard to ask you the name of your project, and sets up a little folder and file structure on disk.

The third starts a local web server, so you can be coding live and see the results in the browser.

Under the hood vitepress will likely require other packges, it quietly works out what it needs and gets them for you. If you get other plugins for vitepress itself, its just another npm command to get them downloaded and installed. If your project requires 'compiliing' into a more packed format and uploading to a certain location, its likely npm can do that too.

If you've been used to fighting with python, or houdini's annoying package json stuff, npm is an amazing example of how good a software ecosystem can be.