In 2010, when Microsoft unveiled the Kinect, it pitched the camera as a revolutionary new gaming device. Swing an imaginary lightsaber and that would be translated onscreen. Throw a football and it would be caught on your TV. Fifteen years later, we know the Kinect as an expensive failure. Microsoft overestimated the demand for playing games with your body. But the Kinect did still turn out to be revolutionary \u2014 just not for gaming.<\/p>\n
Now, we understand the Kinect is anything but a gaming device<\/a>. It became a robotics game changer<\/a>, enjoyed a brief dalliance with pornography<\/a>, and is now upsold as a ghost hunting toy<\/a>. None of which would have been possible had a community of hackers not come together to fashion open source drivers for the Kinect, freeing it from the limitations of being locked to the Xbox 360 and opening new frontiers of experimentation, creative expression, and commercial advancement.<\/p>\n \u201cTechnically, nothing the Kinect did was entirely new,\u201d says Memo Akten, an artist working with code, data, and AI<\/a> and an assistant professor at the University of California, San Diego. The small camera projected a grid of infrared dots and read deformities in that pattern to discern depth. In an early example of machine learning, it recognized human limbs and gestures. \u201cThose capabilities existed in research and industrial systems for many years,\u201d he adds. Those systems cost in the region of $5,000 to $12,000. Here was Microsoft selling a variation of the technology for $150.<\/p>\n \u201cWhat had previously required very expensive equipment and\/or complex multi-camera setups with manual alignments, calibration, and correspondence was now available off the shelf,\u201d Akten continues.<\/p>\n Kyle Machulis, CEO of Nonpolynomial and founder of buttplug.io<\/a> \u2014 an open source project for controlling sex toys \u2014 was working on $250,000 mapping systems not dissimilar to the Kinect in 2010. He quickly recognized the peripheral as an opportunity to \u201cdemocratize that technology.\u201d<\/p>\n He recalls heading out on November 4th to pick up a Kinect to reverse engineer. An hour later, New York-based DIY electronics producer Adafruit announced OpenKinect: a bounty of $1,000<\/a> \u2014 a prize that it would raise to $3,000 \u2014 for whomever offered evidence of the Kinect working on any operating system.<\/p>\n \u201cImagine being able to use this off the shelf camera for Xbox for Mac, Linux, Win, embedded systems, robotics, etc.,\u201d Adafruit wrote in its announcement. \u201cWe know Microsoft isn\u2019t developing this device for FIRST Robotics, but we could! Let\u2019s reverse engineer this together, get the RGB and distance out of it and make cool stuff!\u201d<\/p>\n Doing so was not a simple case of taking the Kinect apart or plugging it in. Though it could connect to a PC via USB, the way they communicated was unknown and the only way to get at it was to watch the Kinect and Xbox 360 speaking to one another. \u201cSince the Kinect didn\u2019t have PC drivers, we needed this piece of hardware called a USB sniffer,\u201d Machulis tells The Verge. A colloquial term for a protocol analyzer, a USB sniffer is a tool that could record the data passed between the Kinect and Xbox 360. In 2010, that cost $1,200 and, Machulis says, \u201cI really didn\u2019t want to buy it.\u201d<\/p>\n Some information could be gleaned by simply connecting the Kinect to the PC, but it was mostly unhelpful \u2014 power consumption, packet sizes, and confirming the Kinect is, in fact, a camera. Hackers could start sending random packets and possibly work something out, but it was just as liable to brick the Kinect completely.<\/p>\n Hackers and reverse engineers around the world were raring to go. But it appeared that whoever got their hands on a sniffer would win the bounty almost by default. That race wasn\u2019t just for the money, however, but also the cachet of being the first to hack such a high-profile device. With the community stalled over the massive expense \u2014 almost half the bounty \u2014 it opened the door for someone outside the community to potentially snatch the glory away.<\/p>\n To keep the contest equitable \u2014 and, perhaps, to try and maintain the bounty\u2019s and the company\u2019s momentum in the press, Adafruit took on that expense, ordering a sniffer to then release the logs to the community. But while everyone waited for the device to ship to Brooklyn, it appeared the worst had already happened and someone had beaten them to the punch.<\/p>\n \u201cAlexP\u201d released a video the next day demonstrating control of the Kinect\u2019s motor<\/a> on PC. It prompted a panicked response<\/a> from Microsoft denying the Kinect could be hacked while threatening to explore legal options. Microsoft quickly U-turned when it became clear no one was trying to hack consumers\u2019 cameras. But as the community reeled from potential litigation, AlexP returned with a second video. This time, he showed off depth and RGB images on PC<\/a>. The contest was over before it had begun.<\/p>\n AlexP didn\u2019t claim the bounty, however. Instead, his company, Code Laboratories, opened a $10,000 \u201cfund,\u201d upon payment of which Code Laboratories would release the source code to the open source community. For some in the OpenKinect community, this wasn\u2019t so far removed from what Adafruit \u2014 which was already benefitting from significant press \u2014 was doing. Even if Adafruit was looking to open source the Kinect drivers and Code Laboratories to sell the drivers as it had with the PlayStation Eye before. For others, it was tantamount to a ransom, withholding code that could make the Kinect more accessible and unleashing its potential easier. \u201cBut that was great motivation for the community to just be like: Let\u2019s take $10,000 away from you, actually,\u201d Machulis says.<\/p>\n A competitive edge shouldn\u2019t be surprising in a contest for a bounty. According to Machulis, however, there was more to it. \u201cThat\u2019s the thing about reverse engineering; It\u2019s who gets their name on it first and loudest.\u201d<\/p>\n On the evening of November 9th, Adafruit finally uploaded the logs collected by its sniffer, and the community began to pore over them. They were searching for the protocols that controlled the Kinect, exploring packets that might turn on a light, enable a camera, or operate the motor. It was incremental, tedious, and exhausting work.<\/p>\n As US hackers dropped off in the early hours, the clock ticked over to the Kinect\u2019s European release date. Like his American peers, 20-year-old Hector \u201cmarcan\u201d Martin purchased a Kinect and, armed with Adafruit\u2019s logs, went through packet by packet to divine the Kinect\u2019s protocols.<\/p>\n When the US woke, it did so to the fruits of his dogged examination: a video of Martin demonstrating RGB and depth on Linux<\/a>. It had taken six days for OpenKinect to hack the Kinect from its release \u2014 really, once the logs became available, it had taken Martin less than 24 hours.<\/p>\n That was far from the end, however. \u201cHector certainly did most of the hard, technical work in terms of getting the initial packets set up,\u201d Machulis says. \u201cThen everyone realized this shit is gonna get big.\u201d The bounty claimed, the community set its sights on more drivers.<\/p>\n Theo Watson spent three weeks at his computer working on OpenKinect \u2014 every day, 10 hours a day. The Kinect revolutionized how Design I\/O<\/a>, which Watson co-founded, developed interactive installations, and he still uses it today<\/a>. In 2010, however, he was 30 years old, recently transplanted to the US, isolating limb data from infrared cameras.<\/p>\n \u201cI saw that time off as an R&D investment,\u201d he says, a way to open more efficient avenues to bigger and better interactive experiences. \u201cI really wanted to be the first person to get the Kinect running on a Mac.\u201d<\/p>\n The OpenKinect community had coalesced around a remote nucleus of reverse engineers from different countries, generations, and demographic groups. There was drama, frayed nerves, but also a common goal, as Watson discovered while trawling through the process of running the Kinect on OSX.<\/p>\n \u201cI think it really helped because it felt like you were part of a team effort,\u201d he says. \u201cIf people were running into problems, it\u2019s like having a collective brain. The Kinect basically needs everything to be perfect. If you\u2019re off by one little thing, you don\u2019t get anything. Then, suddenly, someone notices something and it works.\u201d<\/p>\n Driven by the community, with a bit of help from Martin, Watson had the Kinect talking to Mac by November 12th. More drivers and code filled the community\u2019s GitHub, and hacking efforts wound down. \u201cThere\u2019s only so much to extract and then you have to be able to do something with that data,\u201d Machulis says. \u201cThose are two fairly different skillsets.\u201d<\/p>\n Instead, people like Akten stepped in. The first open source drivers, libfreenect, didn\u2019t include body tracking \u2014 Microsoft released its own skeletal SDK in 2011, its hand forced by OpenKinect \u2014 only granting access to raw depth data. \u201cWe could still do a lot with that,\u201d Akten says. \u201cFor one thing, we had the 3D data, which allowed all kinds of creative, playful interpretations.\u201d<\/p>\n The community chat flooded with experiments, many Akten\u2019s. He explored drawing in 3D space<\/a>, later moving on to develop machine learning algorithms to detect poses<\/a>, and even controlling drones<\/a>. The Kinect was open, bringing with it a host of explorations of how to exploit it creatively. Suddenly, an affordable way for robots to detect obstacles and map environments in real time became available<\/a>, surgeons explored examining scans contactless<\/a>, rapid 3D models of rooms and objects<\/a> became a real possibility, teachers used the Kinect as an interactive learning device<\/a>, and, if you really wanted, someone could now control a sex toy<\/a> over a video call.<\/p>\n \u201cThis thing on the front camera,\u201d Watson says, pointing to the black bar at the top of his iPhone\u2019s screen, \u201cthat, I think, is a miniature Kinect.\u201d<\/p>\n He\u2019s almost wistful. Apple purchased PrimeSense<\/a>, the Israeli company behind the Kinect\u2019s sensor technology, in 2013. \u201cI was so disappointed,\u201d he says, \u201cbecause I just knew that was the end of the Kinect technology.\u201d<\/p>\n The sale prompted Microsoft to explore a new system for its next Kinect<\/a> \u2014 OpenKinect went and hacked that one too \u2014 discontinuing Kinect for Windows shortly after its release in 2014 and shutting down manufacturing for the Kinect in 2017<\/a> as sales diminished and it focused on the Kinect 2 and development of the third-generation Kinect Azure<\/a>. Yet, the technology has lived on<\/a>, incorporated into countless Apple devices as part of its facial recognition and 3D mapping<\/a> to the point of being ubiquitous.<\/p>\n That sense of loss extends, in part, to the internet from which OpenKinect emerged. \u201cIt was way more punk rock!\u201d Watson laughs. \u201cNo one had really established the rules.\u201d<\/p>\n