Got Data? 200+ Crunch Numbers for Duke DataFest

Photos by Rita Lo; Writing by Robin Smith

While many students’ eyes were on the NCAA Tournament this weekend, a different kind of tournament was taking place at the Edge. Students from Duke and five other area schools set up camp amidst a jumble of laptops and power cords and white boards for DataFest, a 48-hour stats competition with real-world data. Now in its fourth year at Duke, the event has grown from roughly two dozen students to more than 220 participants.

Teams of two to five students had 48 hours to make sense of a single data set. The data was kept secret until the start of the competition Friday night. Consisting of visitor info from a popular comparison shopping site, it was spread across five tables and several million rows.

“The size and complexity of the data set took me by surprise,” said junior David Clancy.

For many, it was their first experience with real-world data. “In most courses, the problems are guided and it is very clear what you need to accomplish and how,” said Duke junior Tori Hall. “DataFest is much more like the real world, where you’re given data and have to find your own way to produce something meaningful.”

“I didn’t expect the challenge to be so open-ended,” said Duke junior Greg Poore. “The stakeholder literally ended their ‘pitch’ to the participants with the company’s goals and let us loose from there.”

As they began exploring the data, the Poke.R team discovered that 1 in 4 customers spend more than they planned. The team then set about finding ways of helping the company identify these “dream customers” ahead of time based on their demographics and web browsing behavior — findings that won them first place in the “best insight” category.

“On Saturday afternoon, after 24 hours of working, we found all the models we tried failed miserably,” said team member Hong Xu. “But we didn’t give up and brainstormed and discussed our problems with the VIP consultants. They gave us invaluable insights and suggestions.”

Consultants from businesses and area schools stayed on hand until midnight on both Friday and Saturday to answer questions. Finally, on Sunday afternoon the teams presented their ideas to the judges.

Seniors Matt Tyler and Justin Yu of the Type 3 Errors team combined the assigned data set with outside data on political preferences to find out if people from red or blue cities were more likely to buy eco-friendly products.

“I particularly enjoyed DataFest because it encouraged interdisciplinary collaboration, not only between members from fields such as statistics, math, and engineering, but it also economics, sociology, and, in our case, political science,” Yu said.

The Bayes’ Anatomy team won the best visualization category by illustrating trends in customer preferences with a flow diagram and a network graph aimed at improving the company’s targeting advertising.

“I was just very happily surprised to win!” said team member and Duke junior Michael Lin.

Mapping Science: The Power of Visualization

By Lyndsey Garcia

Mobile Landscapes: Using Location Data from Cell Phones for Urban Analysis

Mobile Landscapes: Using Location Data from Cell Phones for Urban Analysis

We are constantly surrounded by visuals: television, advertisements and posters. Humans have been using visuals such as cartographic maps of the physical world to help guide our exploration and serve as a reminder of what we have already learned.

But as research has moved into more abstract environments that are becoming more difficult to interact with or visualize, the art of science mapping has emerged to serve as a looking glass to allow us to effectively interpret the data and discern apparent outliers, clusters and trends.

Now on display from from January 12 to April 10, 2015, the exhibit Places & Spaces: Mapping Science serves as a fascinating and intriguing example of the features and importance of science mapping.

The end result of a ten-year effort with ten new maps added each year, all one hundred maps are present at Duke at three different locations: the Edge in Bostock Library, the third floor of Gross Hall, and the second floor of Bay 11 in Smith Warehouse.

Visualizing Bible Cross-References

Visualizing Bible Cross-References

Science maps take abstract concepts of science and make them more visible, concrete, and tangible. The scope of the exhibit is broad, including science maps of the internet, emerging pandemics in the developing world, even the mood of the U.S. based on an analysis of millions of public tweets. Science mapping is not limited to the natural or technological sciences. Several maps visualize social science data such as Visualizing Bible Cross Connections and Similarities Throughout the Bible, where the axis represents the books of the Bible and the arches portray connections or similar phraseology between the books.

Angela Zoss, the exhibit ambassador who brought the project to Duke, comments, “The visualization helps at multiple phases of the research process. It helps the researcher communicate the data and understand his or her data better. When we try to summarize things with equations or summary statistics, such as the average, the mean, or the median, we could be glossing over very important patterns or trends in the data. With visualization, we can often visualize every single point in space for small data sets. One might be able to detect a pattern that you would never have been lost in simple summary statistics.”

The physical exhibit holds importance to the Places & Spaces project due to the physical printing of the maps. Some of the details on the maps are so intricate that they require an in-person viewing of the map in order to appreciate and understand the information portrayed. Such as, A Chart Illustrating Some of the Relations Between the Branches of Natural Science and Technology, is a hand-drawn map from 1948 showing the relationships between the branches of natural sciences and technology by using a distance-similarity metaphor, in which objects more similar to each other are more proximate in space.

A Chart Illustrating Some of the Relations between the Branches of Natural Science and Technology. Used by permission of the Royal Society

The maps look more like works of art in a museum than a collection of maps to interpret data. Angela Zoss explains her love of visualization as, “Visual graphics can inspire an emotion and excitement in people. It can encourage people to feel for information that would otherwise seem dry or intangible. The exhibit heightens those emotions even more because you see so many wonderful examples from so many different viewpoints. Every visualizing person is going to make a different choice in the details they want represented. Being able to see that variety gives people a better idea of how much more is possible.”

3D Storytelling of Livia’s Villa

by Anika Radiya-Dixit


Eva Pietroni is in charge of the 3D modeling project, “Livia’s Villa Reloaded”

Have you ever pondered upon how 3D virtual realities are constructed? Or the potential to use them to tell stories about architectural masterpieces built millenniums ago?

The 5th International Conference on Remote Sensing in Archaeology held in the Fitzpatrick Center this weekend explored new technologies such as remote sensing, 3D reconstruction, and 3D printing used by the various facets of archaeology.

In her talk about a virtual archeology project called “Livia’s Villa Reloaded,” Eva Pietroni, art historian and co-director of the Virtual Heritage Lab in Italy, explored ways to integrate 3D modeling techniques into a virtual reality to best describe the history behind the reconstruction of the villa. The project is dedicated to the Villa Ad Gallinas Albas, which Livia Drusilla took as dowry when she married Emperor Augustus in the first century B.C.

The archeological landscape and the actual site have been modeled with 3D scenes in a Virtual Reality application with guides situated around the area to explain to tourists details of the reconstruction. The model combined images from the currently observable landscape and the potential ancient landscape — derived from both hypotheses and historical references. Many parts of the model have been implemented in the Duke Immersive Virtual Environment (DiVE).

Instead of using simple 3D characters to talk to the public, the team decided to try using real actors who talked in front of a small virtual set in front of a green screen. They used a specialized cinematic camera and played around with lighting and filtering effects to obtain the best shots of the actor that would later be put into the virtual environment. Pietroni expressed her excitement at the numerous feats the team was able to accomplish especially since they were not limited by rudimentary technology such as joysticks and push buttons. As a result, the 3D scenes have been implemented by testing the “grammar of gesture” — or in other words, the interactivity of the actor performing mid-air gestures — in a virtual environment. Hearteningly, the public has been “attracted by this possibility,” encouraging the team to work on better enhancing the detailed functionalities that the virtual character is able to perform. In her video demonstration, Pietroni showed the audience the Livia’s villa being reconstructed in real time with cinematographic paradigms and virtual set practices. It was extremely fascinating to watch as the video moved smoothly over the virtual reality, giving a helicopter view of the reconstruction.


Screen Shot 2014-10-14 at 9.24.55 PM

Helicopter view of the villa

One important point that Pietroni emphasized was testing how much freedom of exploration to give to the user. Currently, the exploration mode — indicated by the red dots hovering over the bird in the bottom left corner of the virtual reality — has a predefined camera animation path, since the site is very large, to prevent the user from getting lost. At the same time, the user has the ability to interrupt this automated navigation to look around and rotate the arm to explore the area. As a result, the effect achieved is a combination of a “movie and a free exploration” that keeps the audience engaged for the most optimal length of time.

Another feature provided in the menu options allows the user to navigate to a closer view of a specific part of the villa. Here, the user can walk through different areas of the villa, through kitchens and gardens, with guides located in specific areas that activate once the user has entered the desired region. This virtual storytelling is extremely important in being able to give the user a vicarious thrill in understanding the life and perspective of people living in ancient times. For example, a guide dressed in a toga in a kitchen explained the traditions held during mealtimes, and another guide in the private gardens detailed the family’s sleeping habits. The virtual details of the private garden were spectacular and beautiful, each leaf realistically swaying in the wind, each flower so well created that one could almost feel the texture of the petals as they strolled past.


Screen Shot 2014-10-14 at 9.24.39 PM

Guide talking about a kitchen in the villa

Screen Shot 2014-10-14 at 9.25.15 PM

Strolling through the gardens

The novelty of the “Livia’s Villa Reloaded” project is especially remarkable because the team was able to incorporate new archeological findings about the villa, rather than simply creating a system from old data without ever updating the visual aspects. Sometimes, as the speaker noted, this required the team to entirely reconfigure the lighting of a certain part of the villa when new data came in, so unfortunately, the project is not yet automatic. Of course, to ultimately improve the application, the team often queries the public on specific aspects they liked and disliked, and perhaps in the future, the virtual scenes of the villa may be developed to a perfection that they will be confused with reality itself.


See details about the conference at:

Artistic Anatomy: An Exploration of the Spine

By Olivia Zhu

How many times have you acted out the shape of a vertebra with your body? How many times have you even imagined what each of your vertebrae looks like?

On Wednesday, October 1, Kate Trammell and Sharon Babcock held a workshop on the spine as part of the series, Namely Muscles. In the interactive session, they pushed their audience members to gain a greater awareness of their spines.

Participants assemble vertebrae and discs of the spine

Participants assemble vertebrae and discs of the spine

Trammell and Babcock aim to revolutionize the teaching of anatomy by combining art, mainly through dance, and science. They imagine that a more active, participatory learning style will allow students from all backgrounds to learn and retain anatomy information much better. Babcock, who received her Ph.D. in anatomy from Duke, emphasized how her collaboration with Trammell, a dancer and choreographer, allowed her to truly internalize her study of anatomy. The workshop participants, who included dancers and scientists alike, also reflected a fusion of art and science.

Trammell observes the living sculptures of thoracic vertebrae

Trammell observes the living sculptures of thoracic vertebrae

To begin the exploration of the spine, Trammell and Babcock had participants close their eyes and feel models of individual vertebrae to gain tactile perception. Trammell and Babcock then instructed participants to make the shape of the vertebrae they felt with their bodies, creating a living sculpture garden of various interpretations of vertebrae–they pointed out key aspects of vertebrae as they walked through the sculptures.

Finally, Trammell and Babcock taught movement: in small groups, people played the roles of muscles, vertebrae, and spinal discs. They worked on interacting with accurate movements (for example, muscles only pull; they cannot push) to illustrate different movements of the spine.

Interactive illustration of a muscle pulling vertebrae

Interactive illustration of a muscle pulling vertebrae




To complete the series, Trammell performed Namely, Muscles, choreographed by Claire Porter, on October 4th  at the Ark.

Mathematical Restoration of Renaissance Masterpieces

Screen Shot 2014-09-21 at 10.41.22 PM

The Ghissi Masterpiece, missing the ninth panel

By Olivia Zhu

Ninth panel of the Ghissi masterpiece, as reconstructed by Charlotte Caspers

Ninth panel of the Ghissi masterpiece, as reconstructed by Charlotte Caspers

What do Renaissance masterpieces and modern medical images have in common?

The same mathematical technique, “oriented elongated filters,” originally developed to detect blood vessels in medical images can actually be used to detect cracks in digital images of antiquated Renaissance paintings.

On September 19, Henry Yan, Rowena Gan, and Ethan Levine, three undergraduate students at Duke, presented their work on oriented elongated filters and many other techniques to the Math Department. Yan, Gan, and Levine performed summer research to detect and correct cracks in the digitized Ghissi masterpiece, an altarpiece done by 14-century Italian painter Francescuccio di Cecco Ghissi. The altarpiece originally consisted of nine panels, but one was lost in the annals of history and has been recently reconstructed by artist and art historian Charlotte Caspers.

The role of the three undergrads was to digitally rejuvenate the panels of the Ghissi masterpiece, which had faded and accumulated cracks in paint layers because of weathering factors like pressure and temperature. Using various mathematical analysis techniques based in Matlab, including oriented elongated filters, linear combinations of 2-D

Henry Yan's K-SVD analysis to detect cracks in the image at left

Henry Yan’s K-SVD analysis to detect cracks in the image at left

Gaussian kernels (which essentially create directional filters), K-SVD (which updates atoms to better fit an image), and multi-scale top-hat (which extracts small elements and details from an image), the research group created a “crack map,” which they overlaid on the original image.

Then they instructed the computer to fill in the cracks with the colors directly adjacent to the cracks, thereby creating a smoother, crack-free image—this method is called inpainting.

In the future, Yan, Gan, and Levine hope to optimize the procedures they have already developed to accomplish color remapping to digitally age or refurbish images so that they look contemporary to their historical period, and to digitally restore gilding, the presence of gold leaf on paintings.