Big Data in Theory-Driven Research: New Directions in Data Collection and Measurement
Dr. Galit Gordoni / School of Management and Economics
Rising interest in the use of Big Data in the behavioral sciences is evident. New types of data and methods of data collection are increasingly applied in studies focused on human behavior. It is still not clear, however, whether user-generated data can serve as a valuable data source in theory-driven empirical studies. This study tests the usability of user-generated text data (posts) in open-source web discussions for assessing research questions guided by the theory of planned behavior (TPB). The study focuses on integrating web scraping (i.e., an automated tool for finding and extracting data from on-line sources) in the initial stage of a study applying the TPB for identifying key beliefs underlying the adoption of Big Data in Israel. Discussion boards on the topic, generated between June and August 2018, were identified and scraped. Content analysis was conducted followed by a comparison between the beliefs identified via web scraping with representative surveys on Big Data adoption. Initial results support the usefulness of using web scraping as an observational data collection method in the first stages of identifying key beliefs underlying specific behaviors for a theory-driven belief-scale development.