Authenticity and Artifice

AI Art was all the buzz a couple of weeks ago. That chatter has died off somewhat, perhaps as people got tired of the shiny new toys like Midjourney and Dall-E 2, but it’s a Thing that will only get more technically impressive and practically useful as time goes on. The pros and cons of that can certainly be debated, but I don’t think that we’re going to see that genie go back into the bottle. Like “Machine Learning”, which improves things that the Money Men care about in production, like schedule and headcount, using AI in art is a tool and a toy that is too useful to go away. At least, until the inevitable meltdown of society and technology, and we’re back to drawing on stone cave walls with charcoal-tipped sticks, but that’s tangential.

“steampunk floating island apocalypse” via NightCafe

This particular bit of buzz is of interest to me both in the abstract and professionally. I worked in video game development for a decade, and I’m working in film at the moment. I haven’t had occasion to use these particular tools for anything more intense than helping my kids with homework, but I do use Houdini, which is built on “proceduralism“, which is more or less the engine that drives AI art.

I’m already using a tool that takes inputs, runs simulations and variations, then spits out something that I can sort-of art direct. The computer does the heavy lifting of calculating all the bits and bobs bouncing about, and if I’ve set up the parameters for the procedure correctly, that calculation comes up with something usable. My job is then mostly about setting up the system for success, and inevitably wrangling things when the computer mangles them somehow. I’m not drawing and painting frames, like I grew up wanting to do, watching the Nine Old Men work their magic. No, I’m a desk-jockey cowboy-mage, desperately trying to harness eldritch powers in a digital wilderness, hoping to produce something that the art director will be happy with.

I’m using a tool to produce effects. It’s not the same as using a ballpoint pen on paper, which I can do, as seen here with my Dwarven Tinkerer, but it’s still a tool. It’s a tool with a bit of a mind of its own, and a black box heart that I hope I can channel to great effect. Sometimes it does as predicted, but sometimes it gets a bit flipped somewhere, or an assumption inverted, and things go awry. This, to me, is the most irksome part of using such tools from a production standpoint. Yes, the simulations get faster and faster every year, the results cleaner and more useful… but sometimes I just don’t have the control that I have with much less ambitious (and much more time consuming) tools.

Maybe I can have the spiffy AI system generate 200 different trees, all variations on a theme based on growth rules and parameters, but none of them are what I actually want to use for a “hero” tree. They can be good for fillers to back up the Potemkin Villages that games and films build as part of their magical facades, but for things that get the spotlight, that Uncanny Valley effect where computers still don’t quite get reality is still a hurdle.

We’ve known this for a long time in film; that’s part of why filmmakers can get away with matte painted backgrounds and greenscreen tricks, even as they spend an inordinate amount of time on actors and their makeup and lighting. Backgrounds can be simpler, counting on viewer assumptions and interpolations to gloss over imperfections. We also see a similar “audience interpretation” filling in the gaps when we look at concept art. Even masters like Daniel Dociu, for all their incredible skill and intricate detailing, still don’t work out and carefully render every little detail when they produce concept art. Zoom in on something like his “Tectonic Dystopia” piece…

…and note that even as he bombards the viewer with detail, it doesn’t always bear heavy scrutiny. He’s put in a lot of work, but a detail like a single road is largely a suggestion, a brushstroke or two, maybe a few blobs or smudges, and the viewer’s assumptions of what a city looks like at scale fills in the mental gaps. It’s a fine dance between just enough detail to be plausible without having so much detail that it triggers our sense of wrongness if something’s not perfect.

Leveraging the viewer’s imagination and interpretation is indeed part of Dociu’s mastery of his craft, and while I may sound disparaging, I recognize and am impressed by his genuine skill in performing such feats. Sometimes, we want to be fooled. Art movements and forms of entertainment have been built on this sort of shenanigan, tricking the viewer’s eye, like pointillism, impressionism, or the mental assault of cubism and lesser imitators in more modern art, bluffing with balderdash to give the impression of depth.

The principles at play, then, those of dazzling with detail, or overloading with obfuscation, well, those are age-old fine art traditions. When it comes to AI, though, it’s still learning. It’s only as good as the material it’s trained with, and the assumptions built into the generation systems. Those assumptions aren’t always built with fine art principles in mind, or are built to function first, rather than consider fripperies like composition, emotional appeal or verisimilitude, much less photoreality. Perhaps such considerations will continue to be folded into the frameworks of these tools, but for now, there is a lot of room to grow.

Deep Fake videos are one branch of the technology that is getting particularly interesting and potentially troublesome. Sure, being able to fake Tom Cruise or Harrison Ford is a humoresque parlor trick, but more nefarious uses abound in an era of political disarray and general lack of fidelity to truth. There’s a moral dimension to art, and there always has been, so it’s wise to be aware of how technology can engender trust when it is not warranted. Again, sometimes people want to be fooled, though, for better and worse.

Similarly, there are revolutions in animation brewing. Motion is especially tricky, and much more likely to faceplant into the Uncanny Valley. The technology keeps improving, however, as noted over here, and here. This will definitely make some production faster, especially for midground and background crowds and such. It will be interesting to see how well it fares in the foreground. I’m not convinced yet that it will work as well as some would like, but there are already real consequences for production pipelines.

In the meantime, however, I’ve found that I increasingly value authenticity. From OK GO‘s oddball music videos that bank on their intense efforts in production to Wintergatan’s fascinating machine, from anachronistically authentic YouTube gamers (the older gentleman known as TinFoilChef just played Minecraft the way he wanted to and built an audience that loved his affable curmudgeonly ways) to hand-carved woodworking, I find value in things that appear to me to be genuine and honest. I still carry pens and a sketchbook most places I go, after all, and I’m almost always drawing something, even if it’s just odd designs to keep myself focused. There is value in things that have been made by hand, though whether that value can translate into a career is certainly always a question.

The sky isn’t falling. New tools mean more ways to fool people, with all its attendant implications for an increasingly dysfunctional humanity. “All is vanity“, though, and we must always consider truth and our own decisions. It was ever thus. My profession is definitely impacted, and my personal interests in creative endeavors will be perturbed somewhat, so I’m not neutral on this. I simply see yet another set of shenanigans. Artists have always borne responsibility to be uplifting and useful, since their tools are inherently not honest, as mere representations of reality. Far too many fail miserably at this, and new tools will not compensate for moral failures. Those of us in the audience will always have to be wary, or at least, we’ll have to choose which artifice we want to accept as authentic.

Read Full Post »

Avatar: Aang and Korra

I really like Avatar: The Last Airbender.  Aang’s story is solid, and the core cast all get great moments like Katara’s “The Painted Lady”, Toph’s “The Blind Bandit” and Sokka’s “Sokka’s Master”.  Even Zuko’s arc works really well.  It’s not a perfect show, and it has its weak episodes like many shows, but for every beach party episode, there’s a gem like “The Boiling Rock”.  For every facepalming moment like the musical hippies, there are great character moments like Sokka reflexively covering Toph on a crashing airship or Iroh counseling a mugger.  There are even really great subtle worldbuilding touches like the trains in Ba Sing Se.  (And oh, Iroh’s Tale.)

It’s one of the very few series that I have on DVD, in the august company of DuckTales, Stargate SG-1 and MacGyver (classics, all, and though also cursed with the occasional stinker episode, the good far outweighs the bad).

So, when it came to the sequel Avatar series, Legend of Korra, I was really looking forward to seeing some cool new ‘bending tricks or even combo uses (like Chrono Trigger’s team-up Tech attacks) for the magic in that world.  I thought it would be great to see how technological advancements might happen in a world with magic, and how the two might compete and cooperate.  The world really has a ton of rich potential.

Unfortunately, I wound up disappointed with the characters, writing and meta manipulations of the series creators.  The animation is often great, and the flashback two-parter in Book 2 where we see the origin of the Avatar is really, really fine work.  I’ve had a hard time putting my finger on a lot of what bothered me, though, until I stumbled across this Tumblr:

Poorly Written, Poorly Executed

It’s true that there’s no shortage of criticism and nitpickery online, but almost every single entry I’ve read over there resonates with concerns I have with the series.  This one is perhaps the best place to start, though, looking at it through my perspective as an aspiring writer and experienced artist, since it underlines the backbone of the trouble; the creative staff.  (See also: the Star Wars “prequel” films, and how a rein-free Lucas squandered some of his goodwill credit.)

My single biggest complaint with Korra is that there is so much wasted potential.  The bad writing and character assassinations throughout are like a persistent cough, but the lost opportunities are really what bug me.  The series really could have tackled things ranging from social considerations of magic (one of the stronger themes in the series, but undermined by Amon’s big reveal), the importance of spirituality in the Avatar world, liberty vs. the State (OK, Book 3’s baddies were mostly well done), nascent dictators and assorted other considerations in a world where magic is relatively common but with wildly differing power levels.

There’s so much there to mine, but no, we got relationship drama, disjointed storytelling, burned bridges, relationship drama, character assassination, pointless drama, character assassination of the previous series’ characters, and relationship drama.  There were certainly high points, so the series isn’t devoid of value, but it could have been so much more.

One other big thing that bothers me is something that seems to be a concern in a lot of popular media.  Once the creators start engaging an audience, things can often go sour.  Far too many little beats in Korra came across as either fanservice or trolling that it became less of a legend and more of a performance art experience.  Maybe that’s fine for what some people want, but I think that the Avatar world deserved better.  There’s certainly a good reason to be aware of your audience, but I think there needs to be a barrier there, or else the creative work suffers.

I think I can still recommend Book 1 of Korra’s Legend, but really, it’s best just to go watch Aang’s Airbender stories.

Edited to add:  I ran into another Tumblr that reminded me that the group dynamic was better in Season 1, Bolin was better, and man, they really needed to make a team work like Aang’s crew.  They never really did that well at all in Korra.  They could have, but failed.

Korra’s Team

Man… that’s really the downer of all this.  It could have been so much better.

Edited again to add:  This article from Larry Correia is a great fisking of a monumentally dumb article that shows some of the mentality that I think pervaded Korra’s story.  Maybe not at first, but it seeped in as the authors engaged with the most rabid segments of the audience and catered to sociopolitical quirks of today instead of staying true to their own story.  To be sure, every work is a “product of its time” in ways small and great, but when a work of fiction becomes less about its own interesting story and more about preaching a Message, it falls apart.  This was the problem with Heinlein’s Stranger in a Strange Land‘s third act, for example.

Aang’s journey was internally consistent (mostly, and notably, the Message episodes were the weakest), Korra’s was a mess of Messages and fourth-wall pressures, suffering from undue external influences.  It’s a good case study in what not to do, in many ways.

Read Full Post »

Animation and Writing

I’m an animator.  I’m a writer.  I’m an artist.  I’m a math and science geek.  I’m a gamer and a game designer.  I do a lot of creative things, and always wish I could do more.  My interests are varied and my skillset rather “MacGyverish”, and I work at a fairly small company, so I don’t really focus on animation, but it’s what I earned my college degree in.  En route, I took many classes that required many papers to be written, and a handful of creative writing courses.  Much of what I ran into there was either dry and boring or trippy hippy artsy fartsy nonsense, but it was at least good practice.

So… this NaNoWriMo thing.  I’ve known about it for years, but I’m always too busy.  This year, I’m actually even more busy than I’ve ever been.  Still, I have a lot of novel ideas rattling around in my skull, and some of them really need to escape and see if they can’t spread their wings a little.  I’m sort of not really committing to anything, but I’m going to spend a bit of time writing a novel skeleton, if not a novel itself.  The thing is, there’s an interesting effect that I’ve noticed in my own writing that correlates really well to my animation.  I think that the animator’s Illusion of Life can apply to writing as well.

There’s a difference between “straight ahead” animation and “frame to frame” animation, or keyframe animation.  I’ve done traditional hand animation and computer animation.  I specialize in the latter, but enjoy both.  In both, straight ahead animation tends to produce a more lively, chaotic sort of feel, where the characters and action builds on itself and inertia carries the day.  Keyframe animation is much better when certain story beats or timing points need to be honored, and it’s especially useful for things like walk cycles and other sort of motions that game development uses (I presently work in games, though I’d love to animate for feature films).  Keyframing is also one of the major things that computer animation can leverage, since the computer can calculate the interim frames between keys, instead of needing an army of inbetweeners, as hand-drawn animation needs.  There’s still artistry in making the bezier-like animation curves carry weight and timing well, since computer interpolation is pretty dry and mathematical, so it’s not really a magic “Animate Awesome” button, but computer animation uses keyframing very frequently, simply because it’s good at it.  (And looking at those curves should give you an idea of how knowing math and physics are important to animation.)

Anyway, I’ve been thinking in similar terms for the novels I’d like to write.  There are “story beats” that I’d like to hit, character moments I’ve written mentally that I’d like to work in, and other assorted vignettes that I’ve worked more on than others.  It’s a sort of mental tapestry of ideas, themes, events and moments that I’d like to commit to paper.  It’s not so much a bullet-pointed outline as it is a sketch.  A sort of “concept art” for the story I’d like to write, a rough mental image that can be built into something stronger.  I’ve tightened the art a bit here and there, and left some other pieces loose so that they can be reworked as the whole thing comes into focus.

Interestingly, there’s a bit of what I wrote about here going on, too, where certain bits and bobs of detail can intimate other details, and ultimately, there really is a lot I can leave up to the reader.  It’s very much like a painting, in a way, where the novel has to carry enough detail and interest to let the reader fill in the gaps.  The interesting thing is that I think this applies in the creative process, too, where I hit the high points, the key frames, if you will, of my story, and then go back and fill in the gaps as necessary, but find ways to leave other gaps open for the reader.

As I’ve been writing this, then, in bits and pieces over the years, I do parts, the vignettes, in “straight ahead writing”, but I use those vignettes, those “fixed points in time“, as key frames to hang the larger story on.  It’s a relatively fluid approach, not unlike working from a sketch to a fully realized painting.  Like figure drawing, It’s important to nail down a good sketch, understanding the skeletal structure, musculature, physics and such, before going on to finish a piece.  Some elements can be done in a relatively straightforward manner, and other things might need to be left in the air, and in many ways, the whole piece gets attention over time, instead of just drawing a head, polishing it to a shine, then doing shoulders and so on.  More than once, I’ve seen students do that sort of thing in figure drawing sessions, and then they are surprised when they run out of room at the bottom of the paper for the legs of the model.  The piece really works best when considered as a whole from the outset, even if some of the process winds up being really straightforward, like rendering a face or a hand.

I know, it’s a bit of a stretch, writing about how visual art creative principles inform my writing, and all without using much in the way of visuals to underscore the idea (though the links I’ve included have good visuals).  Still, I thought it might prove useful to try to illustrate how these principles of creativity can bleed between disciplines, and how learning in one sphere can enhance another.  I’ve long believed that, like the basic physiology of neurons, creativity and intelligence grow as you start making connections between individual building blocks of your palette.  If you want to have a rich imagination, you really should be inquisitive and explore.  Learn as much as you can.  Find things that allow you to reframe an idea in a way that you haven’t looked at it before, and it will help you understand what you’re looking at.  Look for the connections and look for the different perspectives.

And then commit it to paper.  I sketch most often with a ballpoint pen.  It forces me to either roll with the mistakes or do it right in the first place.  It’s an emboldening process, ultimately, even though mistakes are inevitable.  Funny how often that’s true.

Read Full Post »

Filling in the Gaps

I’m just ruminating a bit, spurred by a pair of excellent game design posts I read last week.

First, there’s Syl’s post about Why Storytelling in MMOs is Overrated.  I love her article, and I’ve wished for a long time now that MMO devs would ease off the reins and let players tell the story.  (Tangentially, Brian “Psychochild” Green’s work on Storybricks looks like a good step in that direction.)  The developer-driven narrative in these MMO things is a mismatch for the game design from the conception, and the devs seem to cling to their sense of authorship too much.  I can understand that, as a creative sort.  I’ve done a bit of Game Master work in tabletop RPGs in my day, though, and ultimately, the game always seems to run better when the players feel like they are in control.  The GM has to keep everything together, but player agency is the heart of games.  Even if it means they do things the GM doesn’t anticipate or even desire.

Second, there’s this gem from The Rampant Coyote, From Whom Much is Given, Much Is Demanded.  The discussion there about graphics and how cutting edge technology tends to create absurd demands rings true to my experience both in games and when I got my college degree in computer animation.

Today, I stumbled across this interesting tech demo from Activision.  It’s, well… creepy.  It’s very impressive, but it’s still not quite right.  Here it is on YouTube:

That Uncanny Valley looms large.  This is one of the huge dangers of chasing the tech edge.  Yes, in theory, with enough money, processing power and artistry, it’s possible to make artificial life that can pass for the real thing.  The cost is huge, though, and that Uncanny Valley is big.

Also, most importantly, it’s relatively easy these days to make artificial life look good in a still frame, but the real test is when it moves.  Motion is ridiculously hard to make, and exceptionally easy to break.  We have an instinctive understanding of how living things are supposed to move and behave, from physics to biology to exceedingly subtle emotional cues.  (See: Lie To Me, Sherlock Holmes, psychopaths, etc.)

This, perhaps more than anything, is what I really dug into when I was in college.  It’s at the heart of the Disney films I always wanted to make, The Illusion of Life that really makes animation work.  (By the way, I highly recommend that book if you have any interest in animation, along with a more recent tome, The Animator’s Survival Kit.  If you can only digest those two books, you’ll be a long way to understanding the core of animation.)  Ultimately, it’s possible for a skilled animator to make a broom or sack of flour (or even a paper airplane) seem more alive than the latest Final Fantasy CGI characters.  Or, as I noted over at Syl’s place, animators try to be conscious of the silhouette, making sure it’s readable at all times.  You can get a lot of mileage out of just the silhouette, as the XBox LIVE Game LIMBO shows:

And really, a lot of what gets communicated has to do with what isn’t seen.  (For a funny riff on this, there’s this take on what LIMBO might play like when you can see more information… but again, selective reveals are what sell the humor; it’s the juxtaposition of what you expect vs. what is “really” there that makes it humorous/scary.)

If you haven’t seen Paperman, go watch it.  Seriously, go watch it and then come back.  (Or watch the embedded one, sure.)

And then watch this, a video about some of the tech behind it.

So, for a relatively simple-looking bit of animation, there’s a lot of tech under the hood.  Some of it is obviously CG, at least to me, having spent as much time as I have watching and producing art and animation, both traditional and computer-assisted.  Still, there’s a lot of work going into this… and it’s all to make a stylized bit of art.  As with the style of The Incredibles, stylization goes a long way to making something play well.  It short-circuits our instinctive evaluation systems, and the errors in animation that pop up are kind of fudged away, filed in mental gaps that we don’t wind up caring about, largely because we have already internalized that these characters are not real, and we don’t expect them to be.

This is how we perceive motion in film and animation in the first place, per the Persistence of Vision theory.  The 24 or 30 frames per second that flicker by don’t cover the infinitely reducible time frames that reality can be split into, but they happen fast enough that our brain accepts them as continuous enough to be believable.  In fact, sometimes less information works better, as evidenced by some of the kerfluffle around the new-fangled 48FPS The Hobbit movie.  All we really need to know is enough to fool our brain into accepting something as real or believable, and then let our imagination and subconscious do the rest of the work.  Perhaps we could call it a “Persistence of Cognition” theory when it comes to storytelling and lore; the reader/viewer invests headspace in imagining the fictional world and how it works, or how they could work within it.  It’s all about leveraging the strengths of the end viewer/reader/player, making them a partner in the experience.

This is why a lot of the high end stuff fails.  It tries to do too much.  Our brain takes it at its word, holds it to a higher standard, and finds it lacking.

Most of the time, especially with art, story and anything that really hinges on the viewer getting emotionally involved and engaging the imagination, less, to a certain degree, actually is more, simply because you’re letting the viewer breathe and take a bit of ownership, which tends to be a multiplying factor in the efficacy of a presentation.  It’s part of that “willing suspension of disbelief” that’s so important to get people to buy into what you’re doing.  There really are reasons not to go into obsessive hyperdetail, not only because it’s a time and money sink, but because it’s also less effective.

Artists tend to understand this instinctively after some practice, since it’s entirely possible to put too much into a piece of art and thereby ruin it.  Hinting at detail is often far more effective than rendering it.  Even Daniel Dociu’s incredible art, which tends to look really complex, is largely suggestive, relying on the viewer to infer a ton of detail that really isn’t there.  Just look at the actual brushstrokes in one of his pieces and compare it to what you thought was there at a glance.  Dociu is a master at implying complexity.  He’s making your brain do the heavy lifting.

Similarly, as any avid reader can tell you, “head canon” and “mental visualization” of words on the page can never compare to a moviemaker’s craft.  They simply function differently.  That’s a good thing, and creative types really need to leverage the supercomputers in viewers’ brains to do a lot of the creative work for them.  It takes trust, and knowing just what to imply and what to make explicit… but there’s a lot of strength in letting the viewer in on the process, even if it’s only on a subconscious level.

They will fill the gaps, if you can learn what to leave up to them.

Read Full Post »

Bluth and Bricks: Storybricks

I’ve noted the Storybricks project before, but I wanted to make mention of it specifically instead of in passing.

Psychochild’s recent article on it has a good summary of what’s afoot, so I’ll first point you to that:

Precise terminology, or: a game vs. a toolset

Then there’s the Kickstarter page for Storybricks.  If you’ve been itching for more control in storytelling in the MMO space, this is a great way to put your money where your mouth is.  I’ve pledged to support it, since this is just the sort of tool I would love to see.  I’ll admit, I don’t have all that much time these days to really go play with the Unity-based existing playable alpha, but this is something I want to see work, so I’m on board with what the team is doing.

They actually contacted me a while back regarding Storybricks, and at the time, my concern was about the nonverbal parts of storytelling and interpersonal communication.  My specialty in college was animation, so naturally I figured that such would be crucial to really nailing the sense of character that will sell a story well.  They were aware of this, and have addressed it in a few articles… here’s one via Massively, and here’s another also via Massively, but I know there are others somewhere.

Further, recently they announced that animation legends Don Bluth and Gary Goldman signed on to advise the team on animation concerns.

How in the world did you guys do that, by the way?

Anyway, while I’m more a fan of Glen Keane‘s style, Bluth and Goldman understand animation, so the Storybricks team is in good hands on that front.

If you’re interested in making MMOs more interesting, giving NPCs more character and empowering players to tell stories in the MMO venue, please check out Storybricks.  I think you’ll find a lot to like.

Read Full Post »

Without Words

Read Full Post »

9 UP

I just watched 9 and UP back to back in theaters a mile or so apart.  It has been an interesting few hours.

One is about soulless machines and puppets with pieces of a soul, the other lifts your soul if there’s a piece of it left to be found.

One is a post-apocalyptic nightmare, the other is a whimsical dream.

One is a study in browns and fire, the other is all about color and clouds.

One embraces gritty realism and pseudoscientific magic, the other throws realism out the window and works its own magic.

One is at its best when it’s loudest, the other is at its best when it’s quiet.

One viewing of one will last me a lifetime, the other will be a cherished DVD that I view many times.

One is bitter and freaky, the other is bittersweet and weepy.

One tries to hammer a Message home, the other unabashedly embraces emotion, often about a home.

I hesitatingly recommend one, and heartily recommend the other.

Both are visually excellent, deeply creative and fiercely unique.

Each is a master work in its own way, and well worth seeing if you have any interest whatsoever in the subject matter.

Each, in its own way, embraces the message of looking forward and living life while learning from the past, even as you let go.

Funny how that works out.

Read Full Post »