the university essay

Exploring urban social media:

Selfiecity and On Broadway

Lev Manovich

Abstract

User-generated visual media such as images and video shared on Instagram, YouTube,

Sino Weibo, VK, Flickr and other popular social media services open up amazing

opportunities for the study of contemporary visual culture and urban environments. By

analyzing media shared by millions of users today, we can understand what people

around the world imagine and create; how people represent themselves and others; what

topics, styles and visual techniques are most popular and most unique, and how these

topics and techniques differ between locations, genders, ages, and many other

demographic characteristics. In a number of projects completed between 2012 and 2015,

we analysed large number of images shared on Instagram by people in urban areas. This

article discusses two of these projects: Selfiecity (2014) and On Broadway (2015). In

Selfiecity, we compared patterns in self-representations using a collection of “selfie”

photos shared on Instagram by people in five global cities. In On Broadway, we focused

on a single street in NYC – part of Broadway running through Manhattan for 13 miles –

and analysed images shared along Broadway on Instagram and Twitter, Foursquare

check-ins, taxi rides, and selected economic and social indicators using U.S. Census data.

The article presents our methods, findings, and unique interactive interfaces for

explorations of the collected data we constructed for each project.

User-generated visual media such as images and video shared on Instagram, YouTube, Sino

Weibo, VK, Flickr and other popular social media services open up amazing opportunities for

the study of contemporary visual culture. By analysing media shared by millions of users today,

we can understand what people around the world imagine and create; how people represent

themselves and others; what topics, styles and visual techniques are most popular and most

unique, and how these topics and techniques differ between locations, genders, ages, and many

other demographic characteristics.

In 2005 I coined the term “cultural analytics” to refer to the “analysis of massive cultural data

sets and flows using computational and visualization techniques” and 2007 we set a research lab

(Software Studies Initiative, softwarestudies.com) to begin concrete research. Having developed

and tested our techniques and software tools on variety of smaller datasets such as 4535 covers

of Time magazine from 1923 to 2009, in 2012 we started working on social media data.

In a number of projects completed since then, we analysed large number of images shared on

Instagram by people in urban areas. Starting with a general comparison between 2.3 million

images shared by hundreds of thousands of people in 13 global cities (Phototrails, 2013,

http://phototrails.net/ ), we consequently focused on more specific types of images, filtered by

type of content (self-portraits in Selfiecity, 2014, http://selfiecity.net) or a specific city area (13

miles of Broadway in Manhattan in On Broadway, 2015, http://on-broadway.net).

Given that all users of Instagram app are presented the same interface, same filters, and even

same square image size, how much variance between the cities do we find? Are networked apps

and their tools such as Instagram creating a new global visual language, an equivalent of visual

modernism a hundred years earlier? Does the ease of capturing, editing and sharing photos lead

to more or less aesthetic diversity? Do software and networks result in more repetition,

uniformity and visual social mimicry, as food, cats, selfies and other popular subjects seem

appear to drown everything else?

Use of large samples of social media, and computational and visualization tools allows us to

investigate such questions quantitatively. Our analysis in Phototrails revealed strong similarity

between the cities in terms of basic visual characteristics – such as tonality and colours of images

– and also the use of filters. But these findings were partly an artefact of the method we used. We

disregarded the content of photos, the differences in compositions and other aspects of

photographic aesthetics, the relative popularity of various photo types and many other possible

dimensions of difference. Instead, we considered the photos only as assemblages of colour

pixels.

Figure 1.

50,000 Instagram photos shared in Tokyo in 2012, organized by brightness mean (distance to the center)

and hue mean (angle). http://phototrails.net/instagram-cities/

Figure 2. Top: 50,000 Instagram images in NYC over a number of consecutive days, organized by upload date and

time. Bottom: 50,000 Instagram images in Tokyo over a number of consecutive days, organized by

upload date and time. Both samples are from early 2012. http://phototrails.net/instagram-cities/

To compensate for some of the limitations of this first project, in 2013 we started a new project

Selfiecity (http://selfiecity.net). Rather than using an arbitrary sample of social media images

with any content, we focused on only one kind – the popular selfies (self-portraits captured with

mobile phone’s cameras). In the next part of this text I will discuss how we assembled the selfie

dataset, our research methods, the presentation of the work via visualizations and a website, and

some of our findings.

1. Selfiecity

Making Selfiecity

The Project Team. To work on Selfiecity, we assembled a large multidisciplinary team.

The team includes media theorists, an art historian, data scientists, visual designers and

programmers who work between New York, Germany and California. The project was

coordinated by Manovich, while Moritz Stefaner was responsible for creative direction and

visualizations.

The project presentation online combines Findings about the demographics of people taking

selfies and their poses and expressions; a number of media visualizations (Imageplots) which

assemble thousands of photos together; and an interactive application (Selfiexploratory) which

allows visitors to explore the whole set of 3,200 selfie photos, sorting and filtering it to find new

patterns. It addition, the website selfiecity.net also includes three essays about the history of

photography and the selfie phenomenon, the functions of images in social media, and media

visualization method.

Data Collection. The first stage in working on this project was the creation of a selfie

dataset. This required many steps. When you browse Instagram, at first it looks as though it

contains a large proportion of selfies. A closer examination reveals that many of them are not

selfies, but photos taken by other people. For our project, we wanted to use only single-person

‘true selfies’.

The team partnered with Gnip, a third party company which at that time was the world’s largest

provider of social data (gnip.com). After developing software that interfaces with the Gnip

service, in September 2013 we started to collect Instagram photos in different locations. After

many tests, we focused on central areas in five cities located in North America, Europe, Asia and

South America. The size of an area used for Instagram images collection was the same in every

city.

We wanted to collect images and data under the same conditions, so we selected a particular

week (5–11 December 2013) for the project. Listed below are the numbers of photos shared on

Instagram inside the chosen areas of our five cities during this week, according to Instagram data

provided by Gnip (sorted by size, and rounded to nearest thousand):

New York City – 207,000

Bangkok – 162,000

Moscow – 140,000

Sao Paolo – 123,000

Berlin – 24,0000

Total: 656,000 photos.

For our next step, we randomly selected 140,000 photos (20,000 or 30,000 photos per city) from

the total of 656,000 photos. We then used Amazon Mechanical Turk service to select selfie

photos from this set. Each of 140,000 photos was tagged by between two and four Amazon

Mechanical Workers. We experimented with different forms of a question the workers had to

answer, and found that the simplest form – “Does this photo show a single selfie?” – produced

best results.

We then selected the top 1,000 photos for each city (i.e. photos which at least two workers

tagged as a single-person selfie). We submitted these photos to Mechanical Turk again, asking

the three ‘master workers’ not only to verify that a photo showed a single selfie, but also to tag

gender and guess the age of a person.

As the final step, at least one member of the project team examined all these photos manually.

While most photos were tagged correctly (apparently every Mechanical Turk workers knew what

a selfie was), we found some mistakes. We wanted to keep the data sets size the same to make

analysis and visualizations comparable, and therefore our final set contains 640 selfie photos for

every city (eliminating the mistakes), for a total of 3,200 photos.

Computer analysis. This sample set of 3,200 selfie photos was analysed using state-of-

the-art face analysis software rekognition.com. The software analysed the faces in the photos,

generating over 20 measurements, including face size, orientation, emotion, presence of glasses,

presence of smile, and whether eyes are closed or open, and others.

We have used these measurements in two ways. We compared the measured face characteristics

between cities, genders and ages. We also included some of the measurements in the

Selfiexploratory interactive application, to allow website visitors to filter the selfies database by

any combination of selected characteristics.

The software also estimated the gender and age of a person in each photo. We found that both

gender and the age estimates were generally consistent with the guesses of Mechanical Turk

workers.

Visualizing the selfie photos

Typically, a data visualization shows simple data such as numbers. However, a single number

cannot fully everything a photo contains. “A single photo is not a ‘data point’ but a whole world,

rich in meanings, emotions and visual patterns” (Moritz Stefaner, artistic director and

visualization designer of Selfiecity). This is why showing all photos in the visualizations (along

with the graphs or by themselves) is the key strategy of the project. We call this approach “media

visualization.” As Moritz Stefaner explained “Showing the high level patterns in the data – the

big picture – as well as the individual images has been an important theme in our project. How

can we find summarizations of big data collections, which still respect the individuals, and don’t

strip away all the interesting details? This has become a quite central question to us, not only

with respect to selfies”.

Stefaner created a few different types of visualizations for the project, described below.

Blended Video Montages (http://vimeo.com/moritzstefaner/selfiecity-five-cities). Each video

presents 640 selfies from each city. It runs through all the images, but not in a simple sequence.

Instead, a few selfies are superimposed on the screen at a time, with new ones fading on top of

the old ones. The faces are aligned with respect to eye position and sorted by the head tilt angle.

This visual strategy is designed to create a tension between individual selfie photos and patterns

across many images. We do not show each face by itself. But we also do not superimpose all

faces together – which would only produce a generic face template, the same for every city.

Instead, we show something else: a pattern and individual details at the same time.

Imageplots. Manual inspection of photos one by one can reveal many interesting details, but it is

difficult to quantify the patterns observed. We created histogram-type visualizations that show

distributions of genders, ages and smiles in different cities. Like normal data visualization, they

allow you to immediately see patterns expressed in the shapes of the graphs. Bu, because these

graphs are composed of individual photos, they also provide a different way to explore the

interplay between the particular and the general.

Selfiexploratory. This is the key part of the project. It is the interactive visualization app, which

allows website visitors to explore the selfie dataset in many ways. Visitors can filter the photos

by city, gender, age and a number of face measurements extracted by face analysis software.

Figure 3.

Imageplot showing distribution of selfie photos in five cities according to gender (vertical axis) and degree of smile (horizontal axis). The degree of smile was measured by face analysis software; it can take

any value between 0 (no smile) and 100 (strong smile). http://selfiecity.net/#imageplots

Figure 4. A screen shot from Selfiexploratory application. The user selected some of the youngest selfies from our

data of 3200 selfies using Age graph (left column, second row). (http://selfiecity.net/selfiexploratory/)

The application allows visitors to explore the photos using data from both human judgements

and computer measurements – two ways of seeing the photos. The gender and age graphs on the

left use human tags and guesses (from Amazon’s Mechanical Turk workers). All other graphs to

the right use software face measurements. Whenever a selection is made, the graphs are updated

in real time, and the bottom area displays all photos that match the selection. The result is an

innovative, fluid method of browsing and spotting patterns in a large media collection.

In addition to presenting the selfie dataset though visualizations, videos and the interactive

selfiexploratory application, we also decided to present selected findings in a more conventional

format as statistics. Out of a larger set of findings, we selected and presented the following:

1) Depending on the city, only 3–5% of images we analysed were actually selfies.

2) In every city we analysed, there were significantly more female than male selfies (from 1.3

times as many in Bangkok to 1.9 times more in Berlin). Moscow is a strong outlier – here, we

have 4.6 times more female than male selfies. (While we do not have this data for other

countries, in the US the ratio of female to male Instagram users is close to 1:1, according to a

Pew Internet survey).

3) Most people in our photos are pretty young (estimated median age 23.7). Bangkok is the

youngest city (21.0), whereas New York City is the oldest (25.3). Men’s average age is higher

than that of women in every city. Surprisingly, more older men (30+) than women post selfies on

Instagram.

4) Computational face analysis revealed that you can find lots of smiling faces in Bangkok (0.68

average smile score) and Sao Paulo (0.64). People taking selfies in Moscow smile the least (only

0.53 on the smile score scale).

5) Women’s selfies have more expressive poses; for instance, the average amount of head tilt is

50% higher than for men (12.3° vs. 8.2°). Sao Paulo is most extreme – there, the average head

tilt for females is 16.9°!

These findings present only some of the patterns we found. In general, reviewing all the patterns,

we discovered that each of our five cities is an outlier in a unique way (on patterns, see Berry

2015, this volume). Depending on which dimension we choose, one of the cities usually stands

out. However, when we combine many dimensions together, Moscow and Bangkok stand out

from other cities.

Perhaps our overall most interesting finding is the following. Even though people use same

photo app and service (Instagram) that also allows them to easily see how others photographs

themselves around the world, selfie photos we analysed have significant local specificity. The

types of poses change from city to city, and between genders and ages. So while Instagram

maybe contributing to the emergence of a uniform “global visual language,” at the same time it

still reveals cultural and social differences in how different groups of people represent

themselves.

2. On Broadway

In Phototrails, we compared photos from 13 global cities, without filtering them by type or

location. In Selfiecity, we filtered photos to only compare single type photos (selfies) also across

multiple cities. For our next project On Broadway, we decided to zoom in closer into the

universe of social media by focusing on the posts along a single city street. At the same time, we

expanded our data sources, going beyond Instagram and adding Twitter, Foursquare, Google

Street View, taxi pickups and drop-offs, and economic indicators from US Census Bureau.

Figure 5.

Data and image layers used to create the interface to navigating a city street in On Broadway project. http://on-broadway.net

Figure 6. Screenshot from On Broadway application, showing a zoomed-in view centered on Time Square.

http://on-broadway.net

Figure 7. Screenshot from On Broadway application, showing full zoomed-out view – all 13 miles of Broadway in

Manhattan. http://on-broadway.net

Figure 8.

Interaction with On Broadway installation at Public Eye exhibition in New York Public Library (2014- 2016). http://on-broadway.net

Representing The City

Modern writers, painters, photographers, filmmakers and digital artists have created many

fascinating representations of the city life. Paintings of Paris boulevards and cafés by Pissarro

and Renoir, photomontages by Berlin Dada artists, Broadway Boogie-Woogie by Piet Mondrian,

Spider-Man comics (Stan Lee and Steve Ditko), Playtime by Jacques Tati, and Locals & Tourists

data maps by Eric Fischer are some of the classic examples of artists encountering the city. The

artwork that directly inspired our project is Every Building on the Sunset Strip by Edward

Ruscha (1996). It is an artist book that unfolds to 25 feet (8.33 meters) to show continuous

photographic views of both sides of a 1.5-mile long section of Sunset Boulevard.

Today, a city “talks” to us in data. Many cities make available datasets and sponsor hackathons

to encourage creation of useful apps using their data. (For example, NYC Mayor Office’s

sponsored NYC Open Data website offers over 1,200 datasets covering everything from the trees

in the city to bike data.) Locals and tourists share massive amounts of visual geo-coded media

using Twitter, Instagram and other networks. Services such as Foursquare tell us where people

go and what kind of venues they frequent.

How can we represent the 21st century using such rich data and image sources? Is there a

different way to visualize the city besides using graphs, numbers, or maps?

Constructing Broadway The first step in our project was to precisely define the area to analyze, and assemble the data

form this area. Like a spine in a human body, Broadway runs through the middle of Manhattan

Island curving along its way. We wanted to include a slightly wider area than the street itself so

we can capture also the activities nearby. To define this area, we selected points at 30-meter

intervals going through the center of Broadway, and defined 100-meter wide rectangles centered

on every point. The result is a spin-like shape that is 21,390 meters (13,5 miles) long and 100

meters wide.

We used the coordinates of this shape to filter Instagram, Twitter, Foursquare, Google Street

View, taxi and economic data. In the following I describe the details of our datasets.

Instagram. Using the services provided by Gnip, we downloaded all geo-coded Instagram

images publicly shared in larger NYC area between February 26 and August 3, 2014. The dataset

contains 10,624,543 images, out of which 661,809 are from Broadway area.

Twitter. As a part of Twitter Data Grant awarded to Software Studies Initiative, we received all

publically shared tweets with images around the world during 2011-2014. We filtered this

dataset, leaving only tweets shared inside Broadway area during the same time period as we used

for Instagram (158 days in 2014).

Foursquare. We downloaded Foursquare data for March 2009 - March 2014 (1826 days)

through the Foursquare API. Overall, we counted 8,527,198 check-ins along Broadway.

Google Street View images. We experimented with our own video and photo captures moving

along Broadway, but at the end our results did not look as good as Google Street View images.

So we decided to include these images as another data source. We wrote a script and used it to

download Google Street View images (one image for each of our 713 points along Broadway),

looking in three directions: east, west and up. The first two views show buildings on both sides

of the streets. The view up is particularly interesting, since it shows the amount of sky visible

between buildings to Google wide angle lens. In Downtown and Midtown areas, most of the

images in these views are taken by high-rise building, and only a small part of the sky is visible.

But in the northern part of Broadway, buildings are lower, and this is reflected in larger parts of

sky visible in the images.

Taxi. Chris Whong obtained 2013 taxi pickups and drop-offs data from NYC Taxi and

Limousine Commission (TLC). He describes how he was able to get the data here

http://chriswhong.com/open-data/foil_nyc_taxi/.) In 2013 there have been 140 million trips in

Manhattan. Filtering this dataset using Broadway coordinates left us with 22 million trips

(10,077,789 drop-off and 12,391,809 pickup locations).

Economic indicators. We used the latest data available American Community Service (ACS).

It is a yearly survey of the sample of the US population by US Census Bureau. ACS reports the

data summarized by census tracks. These are areas that are much larger than 30 x 100 meter

rectangles we use to define Broadway area. Our Broadway consists from 713 rectangles that

cross 73 larger US Census tracks. Because of these two different scales, any Census population

statistics available will only approximately apply to the smaller Broadway parts. Given this, we

decided to only use a single economic indicator from ACS - estimated average household

income. This data was shown as one of the layers in the application.

Navigating the Data Street, without Maps

We have spent months experimenting with different possible ways to present all these data using

a visual interactive interface. The result of our explorations is a visually rich image-centric

interface, where numbers play only a secondary role, and no maps are used.

The project proposes a new visual metaphor for thinking about the city: a vertical stack of image

and data layers. There are 13 such layers in the project, all aligned to locations along Broadway.

As you move along the street, you see a selection of Instagram photos from each area, left, right,

and top Google Street View images and extracted top colours from these image sources. We also

show average numbers of taxi pickups and drop-offs, Twitter posts with images, and average

family income for the parts of the city crossed by Broadway. To help with navigation, we added

additional layers showing names of Manhattan neighbourhoods crossed by Broadway, cross-

streets and landmarks.

This interactive interface is available online as part of the project website (on-broadway.nyc).

We also showed it on a 46-inch interactive touch screen as part of the exhibition Public Eye at

New Your Public Library (12/2014-1/2016). Since the exhibition was free and open every day to

the public, with dozens of people inside at any given time, we were able to see how ordinary

New Yorkers and city tourists were interacting with the interface. It became clear that focusing

on the visual layers – Instagram photos and Google Street View images – was the key in making

the interface meaningful and useful to the public. We saw many times how visitors would

immediately navigate and zoom in a particular block of the city meaningful to them: perhaps a

place where they were born, or lived for a long time.

This personalization of the “big data” was one of our main goals. We wanted to let citizens see

how many types of urban data relate to each other, and let them relate massive and sometime

abstract datasets to their personal experiences - places where they live or visit.

Conclusion: Aesthetics vs. Politics of Big Data

Today companies, government agencies and other organizations collect massive data about the

cities. This data is used in many ways invisible to us. At the same time, as I already mentioned,

many cities make available some of their datasets and sponsor competitions to encourage

creation of useful apps using this data.

But these two activities – collection of data, and release of the data to the public - are not

symmetrical. The data released by cities only covers what city administers and controls –parks

and streets, infrastructure repairs, parking tickets, etc. This is the data about the city as an entity,

not about particular individuals or detailed patterns of their activities. In contrast, the data

collected and analyzed by social media services, surveillance camera networks, telecom

companies, banks, and their commercial clients (or government agencies if they were able to get

access to parts of this data) is about the individuals: their patterns of movement, communication

with other people, expressed opinions, financial transactions.

Some of the data from social media services is easily available via API to anybody with a basic

knowledge of computer programming. This data is used in numerous free and commercial apps.

(For example, when I use Buffer to schedule my posts to Twitter and Facebook, Buffer interacts

with them via their APIs to place these posts at particular times on my account pages). The same

data has already been used in hundreds of thousands of computer science papers and conference

talks. Numerous students in computer and design science classes also routinely download,

analyze and visualize social media data as part of their assignments. But ordinary people are not

aware that the tweets, comments, images, and video they share are easily accessible to anybody

via these free API tools. While articles in popular media often note that individuals’ data is

collected, aggregated and used for variety of purposes, including surveillance or customization of

advertising, they typically don’t explain that this data is also available to individual researchers,

artists or students.

Artists can certainly play their role in “educating the public” about the access and use of people

data. In our project websites, we have carefully explained how we obtained the data for

Phototrails, Selfiecity and On Broadway, and how we used it. But our main goal was “aesthetic

education” as opposed to “political education.”

“Big data” including visual social media is our new artistic medium, and the projects discussed

here investigate its possibilities. In fact, we wanted to combine aesthetic questions and research

questions: not only what we can learn from social media, but how we use it to create aesthetic

representations and experiences? How should we imagine our cities and ourselves in the era of

massive data collection and its algorithmic analysis? How can visualizations of such data

combine bigger patterns and individual details? What alternative interfaces for exploring and

relating to this data are possible, in addition to linearly organized “walls”, maps, timelines, and

rectangular grids of images and video in Facebook, Twitter, YouTube and other social media

service? In short: how we can see differently – not only the world around us (this was the key

question of modern art) but also our new “data reality”?

Acknowledgements

Each of the projects described in this article was created by a team:

Phototrails: Nadav Hochman, Lev Manovich, Jay Chow.

Selfiecity: Lev Manovich, Moritz Stefaner, Dominicus Baur, Daniel Goddemeyer, Alise

Tifentale, Nadav Hochman, Jay Chow.

On Broadway: Daniel Goddemeyer, Moritz Stefaner, Dominikus Baur, and Lev Manovich.

Contributors: Mehrdad Yazdani, Jay Chow, Nadav Hochman, Brynn Shepherd and Leah

Meisterlin; PhD students at The Graduate Center, City University of New York (CUNY):

Agustin Indaco (Economics), Michelle Morales (Computational Linguistics), Emanuel Moss

(Anthropology), Alise Tifentale (Art History).

The development of Phototrails, Selfiecity and On Broadway was supported by The Graduate

Center, City University of New York (CUY), California Institute for Telecommunication and

Information (Calit2), and The Andrew W. Mellon Foundation. We are grateful to Gnip for their

help with Instagram data collection. The part of this article about Selfiecity project was adapted

from Alise Alise Tifentale and Lev Manovich, “Selfiecity: Exploring Photography and Self-

Fashioning in Social Media,” Postdigital Aesthetics: Art, Computation and Design, ed. David

Berry (Palgrave Macmillan, forthcoming).