In my dissertation experiment, I asked ~60 people from two different graduate schools (or “communities”) on campus to label and organize a set of short documents into a hierarchy (tree structure). They used a web-based interface created specifically for the experiment, that closely resembled the file-and-folder metaphor everybody is used to in Microsoft Windows and MacOS.
Each person was instructed to organize the documents with a different “target audience” in mind: for themselves, for somebody in the same graduate program, and for somebody in the other graduate program. Twenty people were randomly assigned to each “target audience” group, ~10 from each community, and each person organized the documents once, for a single target audience. This resulted in the creation of 6 different “types” of file-and-folder hierarchies, by PRODUCER and AUDIENCE; the N in the chart below represents both the number of participants and the number of hierarchies created, by type:
I have been exploring different ways to analyze the hierarchies participants created, and I am starting to think there are three types of measures:
- vocabulary — word-level measures, like label agreement, average word rank, number of unique words, length of labels
- “topology” — structural measures, like number and size of folders, average path length, etc.
- semantics — this one is a little harder to measure than the others. i wanted to know whether the conceptual groupings of files might look different based on the community of the hierarchy creator, and the target audience
I used multidimensional scaling (MDS), which I wrote about a few days ago, which seemed to show that there were indeed meaningful patterns in the way documents were grouped together. But, I lost too much information with this technique — the MDS showed three distinct conceptual groups, but it was hard to determine whether structure existed within those groups.
Based on previous categorization research (Rosch et al. 1976), I expected that students from CS would create more nuanced conceptual structures for the CS-related documents, and MSI students would do the same for the information-science-related documents. but MDS was not the right technique to use for this — so I used hierarchical cluster analysis instead.
Below are two dendrograms, one that represents the clustering based on data from all of the CS students, and the other that represents data from all of the MSI students. The same three groups from the MDS are also represented here: CS, Information Science, and Security.
In the aggregate MSI student dendrogram, the Information Science cluster is broken into two parts:
In the aggregate CS student dendrogram, the same documents that make up Info Sci 1 and 2 above are merged into one cluster, while the same documents that make up the CS cluster above are broken into two groups:
My next analysis steps will be to figure out how to use this information to systematically examine all of the hierarchies for evidence of these clusters. Ideally, I would like some kind of quantitative measure that indicates to what extent individual participants created structures with these same kind of patterns — but I’m not sure how to do that yet. My ultimate goal is to be able to compare hierarchies along all three dimensions mentioned in this post: vocabulary, topology, and semantics, and find out whether differences exist according to common ground and audience design factors.
