If there is one thing people have learned from programs like Draw Something, it's that artistic ability varies greatly from one person to another.
Though computers are capable of image recognition, the sketches of the average person are usually too abstract for a computer to process. But thanks to a new program, computers can recognize drawings with 56 percent accuracy, allowing them to process artwork by even the most amateur of artists.
James Hays, assistant professor of computer science, developed the program along with researchers from the Technical University of Berlin, Marc Alexa and Mathias Eitz.
Eitz led the research and started out by compiling a list of 250 everyday objects, such as a comb or a camel. Then, through a crowdsourcing program called Mechanical Turk, he and his team collected 20,000 sketches by offering one or two pennies for sketches of the objects.
Originally, Eitz collected thousands of hand-drawn sketches from students at his university to scan into a computer, but the method was too messy and time-consuming.
But the crowdsourcing method also presented a number of challenges. For instance, because participants were using a tablet or computer to create the drawings, their drawings were slightly different than if they had been drawn by hand.
Additionally, many sketchers the researchers employed were not motivated to help for the sake of the research itself. As a result, the team had to sort through the drawings to remove the profane sketches and those created by internet bots for financial gain. But there were fewer unusable sketches than Hays expected based on other experiences with crowdsourcing, he said. The team only had to get rid of 1 percent of the sketches.
"For whatever reason, it was a lot cleaner here," Hays said. "I think because people were enjoying it to some degree."
After sorting through the sketches, the team had a comprehensive database of drawings with 80 pictures for each of the 250 categories.
"The cool thing about this project is that Mathias (Eitz) collected the first database of how humans sketch things," Hays said, "As far as we know, that's never been done before."
In addition, the program uses new ways to recognize drawings. Most programs that recognize imagery only process the geometric shapes of the image, while this program is the first to use category-based recognition.
As users draw an image on a tablet or computer, the program suggests categories that the image may fall under by comparing the sketch to the images in its database. When the drawing is completed, the program displays possible categories from most to least likely.
"We gave (the computer) the high-level knowledge that these 80 images are the same semantic identity versus another sketch-based program that focuses on geometric (identity)," Hays said. "We were surprised to find that that hadn't been done before - it's a fairly common programming tool. ... Nobody had applied that pipeline to sketches."
Hays believes that someday image-based communication with computers may facilitate access to computers in countries where the literacy rate is not high enough for interaction with computers, which are almost entirely language-based.
Hays is also excited by the sociological and anthropological implications of the program. For instance, the program distills each category down to the most common method for representing that object.
"The data is pretty interesting from a sociological standpoint," Hays said. "It says something about ourselves and our cultures, like how Americans draw bread differently from Europeans."
He added that almost all people are bad at drawing lobsters.
The team will leave it up to Eitz to decide if it will expand the program, but Hays said that it would not be hard to add more categories, and they might even move on to more abstract concepts such as emotion. In addition, he thinks accuracy might be improved if the program could monitor the timing during a sketch and the order of the strokes. For now, Eitz has introduced the program in an application for iPhones and iPads, allowing the researchers to collect even more data for their database.
