There are many important limitations to our computational research. Modern tools are not well-suited for old and incomplete data sets. In addition, our lack of knowledge about early modern English makes many textual analysis techniques difficult.
- The algorithms are extremely slow. Ideally, we could classify all 30K texts from 1660-1700 using topic modeling to show the distribution of texts across our four categories, religio, publica, medica, and altera. However, our most recent attempt as this task took over two hours to sort a couple hundred texts. This limitation stems from the depleting resources on our virtual machine, since the it took less than 20 minutes to sort 500 texts initially on our VM.
- Our dataset lacks punctuation and capitalization, which can cause ambiguities. See the EIC Word Clouds page for an example.
- Spellings are not regularized and cannot be effectively corrected by software. See the About Our Dataset page for an overview of how we dealt with archaic and abnormal spellings.
- Our lexicons for topic modeling and word embedding were constructed by hand, so they are not comprehensive and must be updated for future iterations of the algorithms.
- Due to our incomplete knowledge of early modern ethics and language, we cannot build an effective sentiment dictionary for sentiment analysis, a technique that rates text on a scale that corresponds to negative, neutral, or positive sentiment. Modern English sentiment dictionaries are built for social media and contemporary language, which means that they do not accurately reflect what people thought three hundred years ago.
- We are limited to finding correlations, not causations. Therefore, none of the results from our word embeddings are conclusive about the relationships between individual words or bigrams. We can only formulate further questions and find answers from close readings of relevant texts. Because the machine learning algorithms we use are non-deterministic, these correlations also differ in each run of an algorithm.
- In fact, the problem of reliability seems prominent even in the work of distinguished digital humanities scholars. See the Research in the Digital Humanities page for an overview of well-known scholarship in this discipline. While the expositions and case studies of distant reading that Ted Underwood discusses are fascinating, many of his examples feature weak correlation among variables. For example, in chapter 4, he asserts that “[b]ooks reviewed in prestigious venues are increasingly less likely to describe their characters in conventionally gendered ways” even though the Pearson correlation coefficient is only -.151 (Underwood 129). In Underwood’s own words, “absolute values of r around .1 are called small effects; those around .3 are medium-sized effects; those greater than .5 are large effects” (Underwood 27). We are skeptical that such a small effect can be used to assume a historical pattern.
- In applications of statistical inquiry to the humanities, many results appear much more tenuous than insights drawn from traditional archival research. By looking at some past research and thinking about what has been done that may not be entirely reliable, we were able to bring a more critical eye to the validity of our results and an understanding of the limitations of our work.
What we’ve gained from our computational work are not conclusions, but hints to subsets of years that require more investigation, the texts that are most relevant for further close reading, and an increased understanding of 17th century English vocabulary.
