News aggregator

Sune Peterson a.k.a. MOTORSAW sends in this live AV collaboration with Schmid. It’s a beautifully constructed half-hour set, building from simple downbeat with minimal feedback to a frenetic, camera-shaking climax.

Live at home from Sune Petersen on Vimeo.

Sune also had some great thoughts in his email about performance, which I’d like to share:

This year I have begun a collaboration with a friend of mine, who is making music, to try to see how to make visuals and music as part of the same project, and now I have bumped into a wall.
How to go about doing that?
The technical aspects of linking my VVVV-patches to his ableton arrangements is not the problem, the problem is that we are both used to thinking in a way where music is first and then there is usually
some sort of inspiration coming when listening to the music and then some visuals can be created.

Fact is that the way your visuals are being made makes the “sound” of your visuals, it is possible to see what software is being used to make the visuals, just like you can hear that people are playing a guitar (remember arkaos tunnels?). To me that is not really a problem as long as the “tune” you are playing is good.

In preparation to this collaborative project, we have begun making some DJ/VJ sets where we don’t link our computers… nor do I have an audio input since I find that most audio responsiveness is too straight forward and often too “jumpy”.
We have done this lately and with the great flexibility of VVVV the performance can easily scale from one projection up untill so far 3 projections but more is on its way.

This DJ/VJ setup has made me realize that what I really want to achieve is to create an instrument that I can play, just like a guitar player in a band plays his guitar. This introduces the risk of being
off beat and making errors and I definitely need to improve my sense of rhythm but I feel that I am much better connected to the performance when I play the instrument instead of “programming” it. I have some times felt that I was just a spectator of some of the too automated performances I have previously made. This way of performing is really engaging. That is also why I am on stage with the DJ when performing.

What now remains is to continue developing this instrument, I have this “rule” that whenever I make a new performance I make one new feature, this way the performance evolves over time.

I hope I did not lose you in this long email, I suppose that when you write about something about which you are passionate, long texts appear.

I hope you don’t mind me “spamming” you this way.

One new feature per performance is a rule I can definitely get behind. Likewise for VJs getting out there on stage!

We all complain from time to time that the visual community isn’t getting the pay, props, or profile of the eardrum-botherers. Getting on stage and making audiences aware that there’s an artist in control of their retinas is a great way to improve the lot of visualists everywhere.

Finally, I really hope that nobody out there considers sending their work to us spamming! We do receive a lot of email, so Peter and I don’t always post everything that hits our inbox. I’m sure that most of you would agree that we could probably work on a slightly increased posting frequency though. We’re working on that, but we do need your help, so if you’re out there and making awesome stuff, then tell us about it.

MonoMeet Fall 2009 – Tehn from Chili on Vimeo.

I’m headed to Boston next week for a workshop and evening performance – and that’s the perfect time to share some of the good folks I get to work with. You can catch them in person if you’re in the greater Boston area, and expect more on what they’re doing here online following the event for the rest of the planet.

The monome is an undifferentiated array of light-up buttons. Nothing about it really has to control music – it could be used for visuals, or orchestrating an army of small robots, or triggering pyrotechnics. Kedaar Kumar is one of the multi-disciplinary performers who crosses between music and motion. The video at top shows a little peek at his generative work, behind monome inventor tehn (Brian Crabtree). He also has one of the best housings I’ve seen for the monome, in the form of an Etch-a-Sketch case.

For more of Kedaar’s work:
www.soundcloud.com/kedaar
Vimeo playlist
See also the collective of Harmonix employees of which Kedaar is a part (yes, the Harmonix of Rock Band and original Guitar Hero fame). Their album is available free: www.oxytocinmusic.com

It can be a little tough to wrap your head around it at first, but the monome is a wonderfully powerful concept. (There are two phases I went through. The first phase: “Huh? This is just a bunch of buttons! There’s no velocity sensitivity!” Second phase: “Wait – the simplicity of this is completely genius.”) As an idea, you don’t actually need the hardware itself – really. Adam Ribaudo, the creator of the insanely awesome 7up software for monome, has executed a virtual monome that runs on a powerful multitouch Java framework. Non-physical monomes slide with elaborate backgrounds like Japanese screens, both instrument and visual representation. You can check out all the software at Adam’s site (under the name makingthenoise / mtn):

http://makingthenoise.com/software.html

Incidentally, I’m working on my own quasi-emulator which I’m writing in Processing. I’m doing it from scratch partly because I want to use it as a teaching tool, talking about arrays, objects, and imagining other monome shapes (trinome?)

Boston Events

If you are near Boston, design organization AIGA Boston is hosting me for both an introduction to Processing I’ll teach during the day and a free party by night.

Thursday, November 12
9:30a – 1:30p Visual Programming with Processing workshop at The Cloud Foundation – registration required
6p – 9p Social Mixer + Visual Lounge at The Enormous Room (which is a room, though not enormous) – FREE

Full details of both:
Boston AIGA: Visual Programming with Processing + Social Mixer

It’s been a while since I’ve banged on about my experiments in cheap, fast music video production, but that doesn’t mean I’ve slowed down.

Recently I got a group of my favorite things together: A projector, a $4 Ikea rear projection screen, my Vixid mixer, Korg Entrancer, video camera, some home made pizzas, and Canadian Whiskey.

What followed was a couple of hours of partying, playing with shadows and video feedback, a bit of editing, and a subsequent music video release, for Edward Guglielmino’s Settle Down With Me:

Edward Guglielmino – Settle Down With Me from Jaymis on Vimeo.

I’ve been waiting for a while to combine the Vixid’s powerful feedback capabilities with rear-projection. The plan for this clip was to do a simple, cheap, fun experiment to help launch the single, and create something which could be revisited with more detail down the track if we decided to give it more budget.

The setup was reasonably simple:

• Projection screen made of Ikea Saxan curtain, taped tight across a double-width doorway between rooms.
• HD Camera shooting the screen (”analogue” video feedback), fed to the mixer.
• Kaoss Pad Entrancer with Time Delay effect used on the top feedback loop, to control strobing.
• Two additional Vixid feedback loops used for colour inversion (zebra-stripe) and layer rotate effects, and to enhance colour.
• Projector behind the screen.

The shoot really was about experimentation. The band came over, we made a batch of pizzas, had some drinks, and then spent the night playing around with different shooting methods and effects. In front of and behind the screen, with and without lighting, solo and in groups.

As often happens with these experimental projects, I only really found the effect I was happy with towards the end of the shoot. By the time Matt the guitarist had arrived, I’d decided that the cleanest results were coming when the talent was behind the screen, and I’d tweaked the Vixid setup to perfection. This provided some really beautiful, colourful, organic looking effects. I would have happily made the clip a single-shot of the guitarist, but apparently the plebs want more excitement and variation.

They’ve got their cheap thrills now though, so I can reveal my Directors Cut here on CDMo:

Edward Guglielmino – Settle Down With Me (Director’s Cut) from Jaymis on Vimeo.

I’m immensely pleased with how this ended up. It took a remarkable amount of on-set tweaking to get the projector and camera alignment correct, and the feedback and effects loops just right, but when it came together, the final product required minimal post-production – just a crop and a levels punch to get bright colours and pure black and white.

The VJX16-4 mixer really is the glue which made this whole thing possible. That’s not to say that the technical capabilities for this kind of thing only exist in the Vixid box. Several other mixers, and most software visualists are using could create similar, or more complex, effects. It’s the tactile immediacy which makes it so powerful. There’s no fiddling around, you just plug things together in the appropriate order, and start creating.

I think this is the point people are missing when they argue (on CDMu comments last week) about whether hardware gear is “better” than software. They can achieve precisely the same things, if they’re programmed to, and a “software” system is infinitely more flexible, because you can run other programs and join them all together. What people actually love about hardware – placebos aside – is the limitations.

The in-built barriers of a hardware device, and the comparatively narrow set of parameters in which it’s designed to work, force you to play inside a little walled garden of creativity. There’s less scope to waste time configuring, installing other software, trying new combinations of controllers and tools, building new automation techniques, checking your email… A hardware device doesn’t do any of that. You’re forced to focus on two things: What am I capable of doing, and what do I want to make with those capabilities.

In a world where our creative capabilities are effectively infinite, (and getting more infinite every day,) it’s liberating to spend some time in a place where there’s less choice. This is why music producers love their MPCs, and why house tracks still get produced on drum machines with a dozen hardwired samples. Too much choice can be paralyzing.

This is also why, after owning it for over a year, I still haven’t been able to give you a Vixid review. I must have sat down to write it every two months since I got the thing, and every time I do so, I get distracted by discovering more techniques which let me do awesome stuff.

So, while I’m espousing the benefits of limited hardware boxes, let me give you the executive review, in case my opinion wasn’t already clear: Of all the boxes I’ve ever worked with, the Vixid is the least-limited, the most flexible and configurable. Yet it still gives you enough of a walled-creativity-garden that you’re not paralyzed by choice. It’s an inspiring tool to own, and to work with.

If you’ve got the need for a limited hardware mixing box, and the budget with which to acquire one, there really is no other choice. If you don’t, then you can put together something cheaper, and much more flexible, using the computer you may have lying around your home.

Just don’t let it paralyze you.

I’ve written a few posts about alternative reality projects before, such as animal superpowers, avatar machines & alternative perspective summary.

Rebecca Lyddon (fairly recent graduate from London College of Communication) in response to a Design Cities brief from the Design Museum:

“It asked us to design a city guide for London breaking away from the norm (showing landmarks and hotspots.)”. Her concept…

“My response explores the way that we learn and gain a personal understanding from our surroundings. By looking at the idea of perception and the way that we look at things I have made a pack which contains 12 masks (flat pack) where the viewer chooses a mask to wear in a location. It is their personal experience which creates a unique view for the viewer to experience.”

View masks. An interesting analogue approach.

Qeve is a promising-looking, open-source visual performance tool built in visual patching environment Pure Data (Pd). It was built primarily on Ubuntu Linux but should also run with some adjustment on Mac. (Pd itself runs on Windows, but some of the visual dependencies are not available on that platform. I’d still recommend Linux.)

Aside from being free and open, and a set of patches you can go in and modify, there are some nice-looking features here:

• Audiovisual mixing and step sequencing, transitions
• Video browsing
• Beat sync and master clock
• 3 layers, supporting multiple formats (video, 3D, photos, text, paint), 3D/2D graphics
• Full OpenSoundControl support
• Audio analysis
• FreeFrame plug-in support
• Live video streaming
• MIDI configuration
• Control with Wiimote, Nintendo DS, and (open source iPhone OSC app) MrMr

The project comes from the development team Estereotips (Italian/English/Spanish site).

Project page:
http://www.estereotips.net/qeve

vdmo Kstati tells us all about it, in a complete English-language blog post:
Qeve: Free tool for Live Performance and VJing

The current documentation PDF is in English.

I’m giving this a shot on my Ubuntu box; stay tuned.

And here’s what happens when the software mixes with TUIO and an open multitouch interactive visual mixing table. It’s the MesaQ project, by the same team.

When Thanksgiving rolls around, OpenCV will be one of the gifts for which I’ll be thankful. OpenCV is a “computer vision” library, capable of tracking motion and analyzing images, but generally useful as a pixel-crunching video library for many tasks. It’s a native, C/C++ library, accessible from many other languages, including Java and Python. Experienced C++ programmers and artists dabbling in Processing alike can use it. (See our Processing tutorial.) It’s also a real success story for open source code, first developed by Intel but heavily developed by researchers and artists ever since. And it’s a success story for the permissive BSD License, used in everything from academic work to commercial projects.

I’m happy to have a few bits of really good news on the OpenCV front.

OpenCV 2.0: Vision improved. OpenCV itself has taken a massive leap forward this fall with version 2.0. The library is smarter, faster, leaner, and easier to use. You can program with Python instead of C++. And even if your programming skills are moderate, this should mean lots of new features that will filter into other OpenCV-based libraries. Among the 2.0 features already rolling out in pre-release versions:

• Loads of new object detectors and descriptors, including a 3D feature descriptor, self-similarity, new abilities to detect objects and people
• Faster and smater algorithms, and more samples to learn how to use them
• A new Python interface
• All-new, cleaner C++ interface
• Easier installation, easier builds on different platforms, and easier integration with IDEs like Xcode, Eclipse, and CodeBlocks
• Subversion for access to code
• Stereo camera support on Linux

Check out the changelog. And to get started, learn how to use the thing, and download OpenCV, see the wiki:

http://opencv.willowgarage.com/wiki/

Computer vision on Android: The other news is that, thanks to Google’s growing support for native code (finally), OpenCV is coming to Android. With OpenCV doing the geeky pixel crunching behind the scenes, you’re free to write some fairly straightforward Java to make vision-enabled mobile apps – or simply use OpenCV for high-performance pixel processing.

You can check out the project on GitHub:
http://github.com/billmccord/OpenCV-Android

This is an early build, so expect some wrinkles. But there are already some handy tools for getting started. There’s a video emulator, so if you don’t yet own an Android because you’re waiting for the better devices coming out in the next few months (smart), you can still get going. And there’s even some basic sample code for testing with face tracking.

If I’ve fired up your appetite, and you’re not sure where to begin, there is an excellent book on the topic, called – aptly enough – Learning OpenCV. It came out last fall, and so doesn’t include all the 2.0 features, but the basic functionality is the same.

So, would anyone like to start a little user group to work on this stuff? Say so in comments, leave your real email (it shows up for CDM admins, but not publicly), and we’ll get something rolling.

Tutorials from CDM on OpenCV:

Processing Tutorials: Getting Started with Video Processing via OpenCV
OpenCV Motion Tracking, Face Recognition with Processing: I’m Forever Popping Bubbles

…and because Linux is a terrific development platform you can boot into on any Mac, PC, or netbook, check out my rundown of getting OpenCV working on Ubuntu.

It’s striking how audiovisualism has broken into maximalist and minimalist streams, toward DJ-style cut-ups and mash-ups of video at one end of the spectrum and geometric generative animation at the other. Firmly planted in the maximalist, Emergency Broadcast Network-descended sliced-and-scratched popular culture, we have Eclectic Method. The duo does audiovisual pop-fragmentation better than anyone right now. They’ve appeared in a number of interviews, but Babelgum focuses right in on what they’re about, why those shards of mainstream politics and culture matter, and how a future generation will be able to receive multiple streams of information at once, jacked in Serial Experiments Lain-style.

Oh, yeah, and you get a brief glimpse of what happens when you mash with Fraggle Rock.

They’re indeed an audiovisual supergroup, and when your eyeballs and eardrums are ready to rock out, they deliver.

Thanks, Shawn Faherty, for the tip!

Speaking of live production and projection, French audiovisual festival Visionsonic 2009 starts today, kicking off with a live music/foley performance of The Odyssey of Rick the Cube.

L’Odyssée de Rick le cube (extraits du spectacle) from Jesse Lucas on Vimeo.

This year’s festival has an awesome international lineup, streaming for 20 hours in the next 3 days, including cutup artist Tasman Richardson (Canada), RVB en Alpha (France), Origamibiro and The Joy of Box (UK), The Incredible Hexidecibels (NZ), UpDate (France, hopefully with their cool projection mapped laptops), and many more.

You can stream the show to your favorite media player with this address. The stream goes live in about 12 hours, plenty of time to plug in your projector and fire up VLC.

Yoo sends along this charming animation from CalArts Animation program student Miwa Matreyek, entitled “Dreaming of Lucid Living.” Merging digital effects with a silhouetted character, Miwa creates elegant shadow play. It’s not hard to imagine this kind of technique being applied to live performance – just look at the centuries-old tradition of wayang shadow puppet theater in Indonesia.

Over at the Vixid blog I’ve been looking at the work of French “video scenographer” Perrick Sorin, who integrates projection, live video, and chromakeying into stage performance.

/Jaymis.

I have a pathological dislike of lists masquerading as blog posts. Too many “bloggers” spend a couple of minutes googling around on a random subject, find a couple of links, and then decide that they’ve collated the “Top 10 Best Navel Lint Collections“.

That said, I’m all about the community-voted lists of awesome stuff, especially music videos. I always look forward to the Triple J (Australia’s publicly funded “youth” radio station) Hottest 100, as it generally heralds a weekend of fantastic music video programming.

Videos.Antville is one of my favorite music video sites. The community posts an eclectic mix of great stuff, both new and old, and the group of filmclip-heads often lock horns in the lively comment threads.

Inspired (or perhaps disillusioned) by the Pitchfork “Top 50 Videos of the 2000s”, they got together, nominated and voted for the “‘Ville.2K 101 Best Music Videos of the Decade“.

Just as interesting as the end result, I think, is the list of nominations. This really has been the decade in which video production and distribution escaped from the big studios, and it’s great to see low budget pieces like Chairlift – Evident Utensil, OK GO – Here It Goes Again and Architecture in Helsinki – That Beep are inhabiting a similarly cherished space to money-burning productions like, well, Cochise.

This decade has really been about advancing tools and capabilities. Those advances will never stop, but realistically, there’s no longer an exponential gap in capabilities between “indie producer in a home studio” and “big budget juggernaut”.

The next decade is going to be mesmerizing.

APPARATI EFFIMERI Cosmopolitan Opening from Apparati Effimeri on Vimeo.

The Drive-In Movie Theater may be all but dead, but using facades of buildings as the new, architectural visual surface is very much alive. So, at the risk of this site becoming Create Digital Projection Mapping, here’s another beautiful work from our friends at Apparati Effimeri.

Cosmopolitan Hotel 03/10/2009
Bologna (IT)

Apparati Effimeri:
Federico Bigi
Marco Grassivaro
Roberto Fazio

Sound Design:
BeInvisibleNow

Now, if only there were indoor spaces with permanent projection systems where this stuff started to happen. Instead of clubs with a few dinky projection screens, imagine having a rotating roster of visualists redefining the interior of a space. I guess we just need a new economic boom?

blink (hc gilje 2009) from hc gilje on Vimeo.

HC Gilje sends a wonderful new installation he has just opened this month in Norway. Here, mapping projections into the space is a kind of improvisation on the architectural volume, a way of playing with the space itself. Even without any particular projection mapping tricks, the “flat” projection” does take on depth in three dimensions. HC also tells us a Kindergarten class got to explore the space, and evidently had a ball fooling around in the light. (Kids are always the best audiences.)

As he describes the work:

Combining mapped projection with reflections from the floor onto the walls of the space, and the mirroring of images from the walls onto the floor, to create abstract light paintings.

On Friday 30th I’m looking forward to talking at Playful in London:

“Playful is a one-day event all about games and play – in all their manifestations, throughout the contemporary media landscape. It’s a conference for architects, artists, designers, developers, geeks, gurus, gamers, tinkerers, thinkerers, bloggers, joggers, and philosophers. We look at what PLAY means both creatively and culturally, and put speakers on the stage who offer different perspectives on where we are currently, where we’ve been, and where we’re going. We want people walking away talking about the nature of games…what they mean to different people inside, on the periphery, outside or miles away from the industry.”

Theres a great lineup of speakers, including:

James Bridle, Robin Burkinshaw, Rexbox (LittleBigPlanet), Russell Davies, Alfie Dennen & Paula Le Dieu (Bus-tops), Kareem Ettouney (Art Directore Media Molecule), Nicholas Felton, Duncan Gough, Leila Johnston, Katy Lindemann, Simon Oliver (Rolando / HandCircus), Molly Range, Roo Reynolds, Daniel Soltis, Tassos Stevens & Tim Wright.

You can buy tickets here, hope to see you there!

I have just posted some documentation of my latest work, Hand from Above.

Hand From Above encourages us to question our normal routine when we often find ourselves rushing from one destination to another. Inspired by Land of the Giants and Goliath, we are reminded of mythical stories by mischievously unleashing a giant hand from the BBC Big Screen. Passers by will be playfully transformed. What if humans weren’t on top of the food chain?

Unsuspecting pedestrians will be tickled, stretched, flicked or removed entirely in real-time by a giant deity.

Hand from Above is a joint co-commission between FACT: Foundation for Art & Creative Technology and Liverpool City Council for BBC Big Screen Liverpool and the Live Sites Network. It premiered during the inaugural Abandon Normal Devices Festival.

Written using openFrameworks & openCV.

Watch video

Quite a few people have sent me The fun theory, a viral campaign from ad agency DDB Stockholm for Volkswagen.

“This site is dedicated to the thought that something as simple as fun is the easiest way to change people’s behaviour for the better. Be it for yourself, for the environment, or something entirely different, the only thing that matters is that it’s change for the better.”

I really like the concept of creating these playful experiments for the good of using our bodies & the environment.

The World’s Deepest Bin

“To throw rubbish in the bin instead of onto the floor shouldn’t really be so hard. Many people still fail to do so. Can we get more people to throw rubbish into the bin, rather than onto the ground, by making it fun to do?”

I like this one best out of the two, particularly as its a new idea. Drop the litter in the bin and a sound plays making it seem like the rubbish is falling a very long distance to the bottom of a pit, encouraging you to put more rubbish in to the bin (or perhaps throwing away things that aren’t rubbish to hear the noise again, eek).

Piano Staircase

“Take the stairs instead of the escalator or elevator and feel better is something we often hear or read in the Sunday papers. Few people actually follow that advice. Can we get more people to take the stairs over the escalator by making it fun to do?”

Its nice, but the idea has been done many times before. This is the first time I’ve seen the stairs be made to look like white & black piano keys though, like the big piano toy.

1) Back in 2006 I wrote about Tuned Stairs…

2) Also by the Gainer team for the NTT ICC in Tokyo.

3) And at the Museum of Science in Boston (watch videos).

4) Also in the Science Museum of Minnesota (watch video)

5) And finally also at the Sony Centre in Tokyo (watch video)

In true viral nature you will be able to upload your own video ideas on the theme to win a prize, so it’ll be interesting to see what concepts the people come up with.

The Youth Music Box, designed by Silent Studios & I, has now moved to the next leg of the UK tour to Bristol.

It will be on at Colston Hall, Bristol until 18th October. Go and have a play if you are in the area.

Watch video

[...] all manner of distractions » Blog Archive » Moving forward http://www.flight404.com/blog/?p=347 – view page – cached I wouldn’t be surprised if many of you had no idea I was a Sculpture major. Well, originally, I was a Painting major, then Illustration, then Industrial Design, before finally settling on Sculpture in the second semester of my sophomore year. I spent the next two and a half years learning how to cut wood, bend metal, blow glass, weld steel, shave plaster, cast wax. It was all very exciting to me and I loved every minute of it. Well, not EVERY minute. The sound an oxy-acetylene torch makes when it backfires is comparable to a gunshot. Oh, and did you know if someone cuts themselves on a bandsaw, you are not supposed to clean up their blood? I ended up sprinkling saw dust on the small puddle next to me because, frankly, it was — From the page [...]

This is brilliant. 2 Hidden Cameras, A bunch of Kids, 1 Marshmallow each : The Marshmallow Test

[update]
Here is a video of the scientific study Joachim de Posada talking at TED Talks. Plus an article in the New Yorker.

Three London based studios regularly featured here on Pixelsumo have taken time out for good causes. Artwise and the World Wildlife Fund present Pandamonium…

“Raising awareness of climate change and of the natural world, Pandamonium invited 16 artists to deconstruct, manipulate and interact with WWF’s iconic panda-shaped collection units. The striking result is 16 original artworks reflecting today’s global environment.

Since the 1960s, WWF’s iconic panda collecting units used to be a familiar sight in shops, post offices and waiting rooms, silently imploring the passing public to donate loose change to the charity. Over many years the pandas raised crucial funds for the international conservation organisation, before being decommissioned in 2007. Having realised that the pandas were becoming collector’s items in their own right, WWF-UK teamed up with Artwise Curators to challenge a group of leading British artists with a creative brief to bring a group of decommissioned pandas out of storage and into the limelight once more.

These works will be on display for two months occupying the windows to the west of the store (on the corner of Oxford Street and Orchard Street) and within the Selfridges Concept Store. The works will be auctioned to raise funds for WWF-UK at a special event on 12th October.” (from Artwise)

Troika, Surrogate

Mixed media
H 80 x L 120 cm
Estimated value: £1,000 - £1,500

For this effect, Troika enhanced the baby panda’s eyes with glowing lasers – creating an ‘android’ (or should that be ‘pandroid’?) that becomes a symbol for people’s capacity to destroy themselves and nature with technology.

“If we fast-forward into the not-so-distant future, we could imagine a world where nature and not only the panda but the entire animal world have been pushed aside by bleak man-made landscapes.”

Watch video

Jason Bruges Studio, Panda Eyes

Pandas, perspex, paint, plywood, thermal camera, servo motors, custom DMX platform
H120 x L180 x W 180 cm (including plinth)
Estimated value: £15,000 - £20,000

These 100 rotating pandas detect a viewer’s presence, and track human movement in unison, in a slightly unnerving way – their confrontational stare is slightly unnerving and urges viewers to consider their impact upon the environment.

“Jason Bruges Studio keenly supports the work and intentions of WWF, specifically in relation to environmental science and technological innovation … Using WWF’s world famous panda emblem has given Jason Bruges Studio a platform through which to join the organisation in delivering its important messages.”

Watch video

United Visual Artists, Frozen in Time

C-type print
H 91 x L 71 cm (image size) framed
Estimated value: £1,000 - £1,500

UVA recreated the panda in ice, then allowed it to melt in their studio. This single image remains – the only record of its existence.

“… representative of our lack of understanding of the world we inhabit … the ice melted faster than predicted and once this image was created the piece slipped, fell and smashed into thousands of shards of ice, which melted and flooded our studio.”

Watch video

I pretty much always use cameras and computer vision in my work, so I am always keeping a close eye on game technology developments in this area. Once these devices come out, we are then able to hack and use them for our own artworks.

I wanted to take a closer look Natal (Sony controlled and Ubisoft ‘Your Body’ coming another time), in regards to its origins, whats real, what is marketing and whats exciting.

In case someone from Microsoft is readings this post, I am not being negative, I am super impressed by the software behind it and the accuracy of the tracking algorithms, I am looking forward to getting my hands on it as a consumer & hopefully developer.

If you haven’t seen Project Natal, read here and go watch this video…

First of all, its not everything shown in the video real. There are a few instances where the tracking wouldn’t work (hands over skateboard, dad getting in the way of daughter etc). The clue is the text shown clearly at the beginning “Product vision: actual features and functionality may vary”.

This video shows a demo on stage at E3 is the live demo…

Very good estimation of body points, works very fast too. Microsoft Research you have some very clever people.

This is a concept of what the hardware *could* look like come release…

(was white before e3)

But right now it doesn’t (at least publicly). The camera has an rgb sensor just like your webcam, but also a depth perception, in the form of an infrared sensor. Infrared light needs to be projected out from the hardware, it hits the player and the room and is recorded by an infrared imaging sensor. This is called time of flight. The software gets a grayscale image, with the brightest pixels being closest to the camera, before being analyzed by the software.

I was wondering where the infrared emitters are on that hardware image above? The green xbox coloured circle on the left?

This is a closer look at the live demo. It looks like probably 3 devices, rgb cam, infrared emitter and infrared sensor…

Engadget said…
“The first thing to note is that Microsoft is very protective of the actual technology right now, so they weren’t letting us film or photograph any of the box itself, though what they had was an extremely rough version of what the device will look like (not at all like the press shot above). It consisted of a small, black box aimed out into the room — about the size of a Roku Player — with sensors along the front. It almost looked a bit like a mid-size (between pico and full size) projector.”

That means that the Milo interaction demo shown here from Lionhead is pre-recorded, naughty naughty :) (p.s funny parody video)

Of course its real though, hands on writeups from eurogamer, kotaku and wired.

“The Milo demo was partially being manipulated by a developer who was sitting nearby, and I couldn’t tell if he was merely calibrating the game or how much he was pulling its strings.” - kotaku

These videos show the camera hardware blurred out even :)

Ok so the hardware isn’t finished, fair enough. The magic really is in the software. How good will the tracking work under different lighting conditions in my house?

Its all about getting a good clean depth image, unlike normal webcam games that are based on solely movement detection or background subtraction (nightmare). So how big will the infrared emitter need to be?

This is the zcam by 3DV Systems…. (pic shows depth image and hardware)

Watch video, see how it works.

Microsoft bought 3DV Systems. I instantly thought they were going to use this camera and their own sdk after seeing the demos, however Eurogamer reports:

“Aaron Greenberg was even more direct. Asked whether Natal was derived from 3DV technology, he told Eurogamer: ‘No, we built this in house.’”

“Kim admitted ‘it’s a combination of partners and our own software’, and some have theorised that acquisitions like 3DV’s were designed to insure the company against similar patents. ‘You have to be very aware,’ Kim said. ‘We want to ensure that we have great intellectual property protection. You have to have a strong legal approach, and this is not easy stuff. It has to be all buttoned up, legally. We have had a very concerted focus on this.’ ”

An interview on Venture Beat…

“VB: Some people are very curious about the patents in the gesture control technology. Is there freedom to innovate here, or do you have to be very aware of the patents out there?

SK: You have to be very aware. We want to ensure that we have great intellectual property protection. You have to have a strong legal approach, and this is not easy stuff. It has to be all buttoned up, legally. We have had a very concerted focus on this.”

Eurogamer reports today… “All of the key image-processing is done by Natal’s in-built silicon, leaving the Xbox 360 free to power the game itself”.

There are rumors that Microsoft are working with & licensing patents from Prime Sense, another Israel company like 3DV making such technology. This image shows their hardware, which the Natal model looks much similar to…

Funnily enough, both joystiq and n4g reported in 2006 that Prime Sense might have been working with Sony Eyetoy for the ps3 for 3d depth sensing :)

I don’t know if that is true or not, but there is a very detailed video on google of the history of eyetoy with tech demos shown to university students, along with demos Sony were working on using the 3DV Zcam, shown below (1 hour in to video) with full skeletal estimation and ball batting games for PS2.

Its a competitive market out there.

Er, anyway. This video from Eurogamer came out today, a hands on with Natal. No camera, but notice the large light source in the corner (*probably* normal light with filter on to only let through infrared spectrum, maybe to help ensure good coverage for the press).

I feel the hardware or who got it to release first isn’t as important as the software tracking. This is fairly interesting…

“Tsunoda made the point that Natal will continue to work even if someone walks in front of a player because it knows how the human body works. So, if a player had his or her arms blocked, but Natal’s cameras could still see part of their arm, it can fill in the rest based on algorithms that tell it how that arm should look.”

This splat video is great. At 3:05 it shows how the depth image threshold adjusts to a 2nd person entering the space. See at 3:31 he adjusts his mic and the avatar does the same :)

For me, Johnny Chung Lee has hit the nail on the head here…

“The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I’ve seen in academic research, never mind a consumer product.” (source)

Its exciting times ahead for the consumer level of computer vision, can’t wait.

p.s small disclaimer. Everything posted above is publicly available by google, and is just speculation.