CGWiki

Appearance

HDA's ​

Quick blat of ideas while this is fresh in my mind...

I've avoided HDAs for a long time, I'm still avoiding them actually, but latest job requires I get my head into it.

Like a lot of Houdini things its a good idea hidden by a slightly obscure interface, this'll try and clear it all up. I compare this a lot to maya references as that's the only comparison point I have, and most people in 3d understand them. If you don't, well.. tough. 😃

What is an HDA anyway? ​

A Houdini Digital Asset is like a maya reference. You export a chunk of a hip to an external file (eg c:/projects/library/coolthing.hda), now those nodes live somewhere else on disk, and you can bring them into your hip like a reference.

They have a few features that make them a little more complicated than maya references:

  • You can't just go File -> Export on a bunch of nodes, you need to use the HDA workflows to export.
  • Saving and updating is kind of magical and in-place, needs a mild leap of faith to trust its working.
  • As well as dumping a bunch of nodes to an external location, you can also put parameters on the top of the hda to control nodes within it, which is a trick you can't do with maya references
  • There's no 'maya reference editor' style thing to get a central overview of all HDAs used in the hip. When you edit HDAs, you do it per node, one at a time.
  • HDAs are supposed to be saved in a shared central location if you're in a big studio, meaning pipeline environment variables and all that. There's workarounds and tricks, but thats the ideal scenario.
  • HDAs when defined will appear in your tab menu rather than expecting you to go File -> Load reference, which is neat.

Once you realise that all an HDA is a bunch of nodes saved together on disk, it all becomes a lot easier to understand. It's also worth noting how many native Houdini nodes are HDAs. The clue is to look for the padlock icon next to the node. If you see it, that's an HDA, and you can dive inside to see how the sausage gets made.

Creating an HDA ​

Again several ways, but here's how its most often done:

  • Select the nodes you want to HDA-ify
  • Put them in a subnet
  • R.click on the subnet, Create Digital Asset
  • Set the name and location to save it. The default is to prefix with your username, I usually remove that. Also be clear about the save location, if its just for you, fine, put it in your home folder, but you probably want to put it somewhere more public if its to share with others.
  • Houdini shows you the Edit operator type dialog. This is for you to customise the interface. The simple trick here is to put the dialog to one side, dive into your hda subnet, select nodes, and drag and drop parameters you want to control directly into the 'existing parameters' region. They'll be automatically channel referenced onto the top of the hda, super handy.
  • Hit accept, you're done.

Kiran points out an even easier method than drag and dropping parameters is to hold down alt + middle click the parameter and it will automatically add to the type properties. Handy!

Lock states, editing and saving and reverting hdas ​

Now you have an HDA, you can do a few things with it:

  • make modifications to the hda in the hip (the equivalent of maya reference edits)
  • push those changes out of your hip and into the hda on disk (the equivalent of opening a second maya, opening the reference, editing and saving it)
  • reverting any changes (the maya equivalent of deleting all reference edits).

Next to the hda is a padlock icon. Red means its unlocked and you might've made a local change, gray means its locked and matches the definition on disk.

In maya reference terms red means you might've made reference edits, gray means you've made no changes. If you dive inside the hda when the padlock is gray you'll see the nodes inside are dimmed; you're literally locked out of making edits.

If you want to make changes you can go back to the top of the hda, r.click and choose Allow editing of contents. The padlock goes red, dive inside and it no longer dimmed, you can edit away. Any changes you make are stored in the hip, the equivalent of making reference edits in maya. Like maya, the original hda (the reference) is untouched. If you were to bring in another copy of the hda, it wouldn't have those edits (the equivalent of referencing the same file twice, the second won't have the reference edits of the first).

To save your changes to the hda on disk, right click on the hda and choose Save node type. That's it. Those changes are now in the hda on disk. That's the equivalent in Maya of opening a second maya, loading the reference, copying your changes into there and saving.

Your hda will still be in the red and unlocked state. You don't want to risk editing and saving changes to the hip do you?

To do the equivalent of a 'delete reference edits', ie make it match exactly to the state of the hda on disk, r.click again and choose Match current definition. The padlock goes gray and locked, if you go inside you can see its all dimmed, but should now show that the changes you saved are now part of the hda rather than of the hip.

This becomes a bit of a pattern. You'll make an hda, give it a nice UI, share it around. Someone suggests an improvement. You'll:

  • 'Allow editing of contents' to unlock it, dive inside to fix it, make sure it works in that edited state.
  • 'Save node type' to push those changes to the HDA on disk
  • 'Match current definition' to lock it, see how the hda will look to others.

Load $HIP/hda on hip load via hou.session ​

Download hip: hda_from_hip.zip

Hacky. But I like hacky.

My current job has a couple of interesting contraints. Hip files need to be run In The Cloud, and ideally run as standalone as possible. That means minimal environment variables, minimal fixed locations for libraries or assets, just the hip and what the hip might need in subfolders.

We also need multiple people working on a single file at once, using github to manage our files. That meant monolithic hip files were becoming an issue, because really only one person can work on it at once. We've tried using the diff tools, but they fell apart on big files, especially if you have things like stash nodes and frozen nodes.

Splitting work up into HDA's would be ideal, but the assumption is you load all your HDAs at houdini startup, and use environment variables to say where those HDAs will be found, which breaks our first requirement.

Enter this trick.

The amazing Chris Gardner shared a python script he wrote to hot load HDAs. Give it a folder, it'll scan the folder and load any HDAs it finds. That script lives here:

https://gist.github.com/chris-gardner/c9daf34a668c5dddda94b9f6276d8cb8

But how can we call that from a hip if we're trying to not load any external scripts or dependencies? Hou.session, that's how!

Hou.session is a python module that is saved as part of a hip, and is loaded (and therefore executed) on hip load. It can be accessed via Window -> Python Source Editor.

As such I copied Chris's script into hou.session, and I run

loadHdaLibrary(hou.getenv('HIP')+'/hda')

At the bottom of the script. Now I can make a hda folder next to the hip, save HDAs in there, and they'll be loaded when the hip is loaded.

Obviously you'd want to move stuff to a more solid pipeline infrastructure over time, but this'll do in the short term.

Best practices ​

It's easy to get carried away and start HDA-ing all the things. But it's good to have a think first, and as always the comparison to maya references come in handy.

  • Don't use HDAs to be tidy. - If you're just wanting to collapse a few nodes together to clean up the network, then just use a subnet, thats what it's for. Or the new collapsable network boxes. Or just leave all the nodes in place and colour them to group them, and give nice names.
  • Will a controller null do? - If you're after a clean central 'control panel' for your setup, you can do the drag and drop thing with a null, just be sure to uncheck the 'forbid links from this same network level' toggle that sidefx annoyingly insist on. HDAs are more for reusable assets than tidy UIs or tidy nodes.
  • Don't make HDAs too small - If you're just wrapping up 2 nodes, why? You're really not gaining much, maybe drag nodes onto the shelf or into a gallery instead.
  • Don't make HDAs too big. - If you're wrapping up 200 nodes, and every time you use it you have to unlock it to dive inside to add nodes and tweak parameters, then maybe you can break that up into smaller, more modular units.
  • Think of the collaborative workflows. - Once a HDA is on disk and away from hip files, it can be worked on in isolation by one artist, while its being used in many hips. Have many HDAs in many hips, then multiple people can be editing different HDAs in parallel, which can be a huge productivity boost. 'Will this be useful to many people in several contexts?' is a good reason to make an HDA.
  • OTL and HDA are the same thing. - For all practical purposes anyway. OTL was the old name, short for 'Operator Type Library'. Nodes in Houdini are SOPs, DOPs, POPs etc, ie Surface Operators, Dynamics Operators, Particle Operators.... Ops, or Operators in summary. You have different kinds of Operators, ie different types, and you could save many operators into a single file, making a library of operator types. OTL. Get it? Wisely someone at sidefx realised one day 'thats a terrible name', and they were changed to Houdini Digital Assets instead. A few of the tools still use the old name, like the hotl command, so don't get confused, they're the same thing.
  • An HDA file can contain many assets. - As I said the old name OTL better implied these files could contain multiple assets. These days people generally save one asset per HDA, as its easier to find stuff when the name of the node in the houdini graph matches to the name of a HDA file on disk, but you can absolutely store as many assets as you want in one hda file.
  • You can save HDAs in your hip - If you want to be super portable and have an all-in-one solution, you can choose to save hdas 'embedded', ie, as part of the hip. Be careful with this though, as it kind of defeats the purpose of using HDAs in the first place. Would you save a maya reference into the working file? You'd need a very good reason, and people would look at you funny.

HDA updates ​

Say I have a TD modifying an HDA, and an artist using that HDA in a shot. How are those updates pushed from TD to artist?

To simulate this I have 2 houdini's open; the one on the left is editing a hda, the one on the right is using it in a shot. You can see that I've changed some of the parameters in the right, but its still a locked hda, so the innards are the same as the hda on disk.

Now if I unlock and change nodes in the left, say swapping the box for a sphere, it doesn't affect the right one at all, not until I save it.

Once I r.click on the hda and choose 'Save node type', what happens to the right side?

As soon as I give the right a nudge, it detects a change and updates.

But what if the artist has unlocked the HDA and made changes? If the TD updates the HDA and saves, will that change be inherited?

No, the right no longer updates. Once you unlock an HDA in a shot it loses its connection to the HDA on disk. If you want to get back in sync you r.click on the hda and choose 'match current definition'.

But what if the artist wants to update the HDA and keep the changes they've made? Unfortunately you can't. There's no built in diff/merge style workflows presented to the end user, so if you go off and do your own thing to an hda, the only way to inherits to the hde on disk is to reset any change you have made.

HDAs in HDAs ​

Once you start getting tricky you might start making bigger HDAs out of other HDAs. Be careful! If you update those smaller HDAs on disk, they won't be automatically updated in the bigger HDAs. You need to manually open the bigger HDA, update those components inside, save.

HDA editor ​

I mentioned before that there's no equivalent to the Maya Reference Editor window. Well that's a lie, of course there's something, but it serves a different purpose, and helps to point out an important difference between Maya references and HDAs.

Window -> Asset Manager will bring up a list of HDAs, but note that its not just HDAs in the current hip, its all the HDAs Houdini knows about. All the ones as part of the sidefx install, ones in your home folder, ones embedded in the hip, ones in your pipeline, EVERYTHING.

This implies that HDAs aren't really just 'external .ma's linked to the current .ma' like references are. A better analogy is Maya's plugin editor; deformers and tools that are part of your session to get stuff done.

Another interesting feature of the HDA editor is that you can have multiple versions of the same HDA in different places, you can locate them and swap between them. If you want to swap between them more directly you can expose a dropdown on HDAs to choose between definitions. From that Asset Manager move to the 'Configuration' tab, and at the bottom for Asset Bar choose Display Menu of All Definitions. This can be really handy when you're trying to track down versions of files, or which HDA you're using.

Updating default parameter values ​

So you've dragged in all your parameters, but realise the default extrude depth should be 0.2 for future use, and the colour should be red and not green?

The intuition is to unlock the HDA, make some changes to the parameters, save node type, match definition. But if you create a new instance of your HDA, the default values are as before. Boo.

Remi Pierre had the answer here. Unlock and change values as before, but now r.click on the HDA and select 'Type Properties'. Now select all the parameters in the view, r.click, 'Copy defaults from node'. Accept, save node type, match definition, try making a new HDA, the values will be correct. Thanks Remi!

30 seconds later

The joys of knowing clever people. No sooner had I posted this than Jeffy from Sidefx suggested an even easier way; the gear menu in the same interface has the same option, but will implicitly work for all the parameters, so you don't need to select them all. Cheers Jeffy!

Using font sops in hdas ​

If you try and promote the font selector menu to the top of an HDA, you'll find its broken.

Luckily Henry Foster has a fix!

Go into the type properties for the HDA, select that promoted font parameter, jump across to the 'Menu' tab.

You should see that it's using an opmenu script to populate the menu, but the path to the font sop is incorrect. When you drag the parameter into the HDA it will bring most of the stuff over, but it doesn't fix this path.

Edit it so that it points to the correct relative path in within your HDA, all should be good again.

Unlocked vs locked on creation ​

There's cases where the default locked state, or even the fact that the nodes are referenced back to an hda on disk can get in the way. Say you make a bunch of nodes to define a shot template with camera, lights, rops etc. You can wrap this up as a hda. Not only should it be unlocked on creation, because you expect artists to immediately dive in, you don't want them to accidentally run 'match current definition' and destroy all their work.

On the save tab of the hda parameter are options to control this behavior:

Save content as locked, when disabled, will not look like a hda at all, its as if you made some nodes, put them in a subnet, promoted some parameters. While it still knows its an hda (the asset bar will still know where it comes from), you can't match definition, its more of a template setup.

Unlock new nodes on creation is less aggressive, and just puts down the hda with the open padlock mode engaged. You can still match definition on this if required.

A third option which sidefx use more and more is dive targets. You can mark a subnet within your hda as available for editing, but leave the rest of it locked. Things like the vellum sop solver do this, so you have a region where you can put your own custom forces, but the boilerplate of setting up the sim is locked and safe.

Getting stats from your hda library ​

This was an interesting deep dive. We wanted to get a sense of how many hda's we had, how many nodes in each hda, what types etc etc.

My first tack was to use python. It's possible to query the children nodes of a node with allSubChildren(). But there's a recent change to how hda's get loaded; sidefx found that having all hda's load all their nodes by default could slow down scene loading massively. Now hdas work in a delayed load format, and only instantiate their internals on cook.

This of course would ruin any attempts for me to run stats; I can't predict what inputs each hda need, nor do I want to wait for a potentially lengthy cook per node.

There's a fix of course via an envvar:

export HOUDINI_DELAYSYNC_HDA=0

Restart houdini, create an hda, all the nodes are in there. Nice.

That said, I started to look into how you'd query a package for its contents, each hda in the package, each tool within the hda file... it was all starting to look a little boring.

Change of approach time.

I'd heard people talk about hotl, a command line tool to let you manipulate hda's. One of its options is to unpack a hda, leaving you with an interesting folder structure of hscript, python, embedded assets and whatnot. I ran this on one of our hdas:

hotl -X /tmp/unpackHda /path/to/mycoolhda.hda

I searched around the subfolders looking for interesting info. There's a 'contents.dir' folder, inside was several files that represented each node within the hda. Looking closer still a box sop would have a box1.init file, which contained text saying what kind of node it was (box in this case, obviously). One folder up was a .init file for the hda itself, that contained the full hda name, version, user who created it etc.

Parsing text files seemed a much easier way to get the data I needed, and could be run in parallel with tops. Thus armed, I created a tops graph that would:

  1. Filepattern a package folder recursively
  2. Filter to .hda files
  3. hotl -X every hda to /tmp/`@hdaname`
  4. Parse the .init files to get the hda details, and all the children node details
  5. Count the number of child nodes via partitioning on the hda name
  6. Write it all out to a csv

This feels like a pretty solid way to generate stats, easy to tweak, fast to run on the farm. Hooray!

Todo ​