This week I first reviewed my first round coding results and rethought the data quality mechanisms, then finished created the first version of citizen science project list.
My teammate Heejun and I coded the first round coding separately to test the objectivity of the data quality mechanisms and subjectivity of human raters (i.e., Heejun and me). I compared Heejun’s and my coding results. Among total 56 papers we found in Scopus database, for identifying the papers’ disciplines, we have agreement on 52 papers (93%); for whether we should code or not code a paper, we have agreement on 46 papers (82%); for 17 data quality mechanisms, we have 46% completely same coding results. Most differences between Heejun’s and my coding results can be used to supplement each other’s results (e.g., Heejun found a mechanism used by a citizen science project mentioned in one paper, while I missed to identify it in the paper, or vice versa, so we have different coding results for this paper. When putting our coding results together, we helped each other to fill the hole) Our disagreement, or I should say uncertainty which is a more appropriate word to use here, focuses on one mechanism: Data Normalization. We need more discussion on the definition of this mechanism.
To summarize, so far, we have documented data quality mechanisms of 20-24 citizen science projects by coding 56 papers found in Scopus. Among 20-24 citizen science projects, half of them are related to Volunteer Geographic Information (VGI). All 17 mechanisms has been found in those papers. There are new mechanisms appeared, such as Trust development / Quality Metrics / Trust Model, in the papers, which need our further discussion.
When creating the citizen science project list, I collected citizen science projects from four resources: (1) citizen science central, (2) SciStarter, (3) a citizen science project list provided by researchers from University of Washington who are studying biodiversity citizen science project data quality; (4) citizen science project cases studies in [1]. After put all the project in the excel file manually and cleaned all the repeated project, there are total 975 projects in the project list.
In the next week, I will mainly work on exploring the possibility of documenting data quality mechanisms from the 975 citizen science projects. Also I will work with my teammate and mentor on achieving better agreement on coding results for the 56 papers.
References:
[1] Roy, H.E., Pocock, M.J.O., Preston, C.D., Roy, D.B., Savage, J., Tweddle, J.C. & Robinson, L.D. (2012) Understanding Citizen Science & Environmental Monitoring. Final Report on behalf of UK-EOF. NERC Centre for Ecology & Hydrology and Natural History Museum.