This continues our series of student reflections and analysis authored by our research team.
On tPP’s 3-step Verification Process
Izzy Bielamowicz
More-so than in the past, this semester I have come to realize the importance of tPP’s verification process. Throughout my time as a member of tPP, I always viewed multi-person verification as useful, but it was not until this semester, when I became a member of the Steering Committee that I finally understood the cruciality of the 3-step process to its full extent.
Arguably one of the most critical methodological approaches of tPP is that before any completed case is moved into the official dataset, it is checked for accuracy by at least 3 separate coders. First, a case is “claimed” by a coding pair in the team spreadsheet, which is accessible to all team members, by coloring its row green. Once the coding pair has claimed a case, they begin the coding process exclusive of one another. Coding a case individually allows for each coder to process and interpret the facts of the case on their own accord based on the guidelines of the codebook.
Once the coders have singly coded a case, they meet to discuss their rationale behind each variable. If the coders have arrived at the same conclusions for every variable based on the facts of the case and the guidelines of the codebook, it is likely that they have correctly coded the case. Should the coders run into a variable which they have coded differently, there is opportunity for valuable discussion. In my personal experience, it is often the case that one of the coders has missed information within the source files or that they have misinterpreted the codebook. In other instances; however, I have experienced dissenting opinions based on inconsistent understanding of the material. Often, variables which offer issue for dissent are ideological target, ideological affiliation, and tactic. This may be attributed to different assumptions gathered from the known facts of the case, and the inevitable ambiguity of certain cases regarding ideology and plot. Should this disagreement occur, the coders reanalyze the case and agree upon the most suitable code. Once the coding pair has completed their case – should dissent arise or not – they move the completed case back into the team spreadsheet, coloring it white.
Once a case is completed by a coding pair and moved back into the team spreadsheet, it is reviewed again by a member of the Steering Committee, the Auditor. Going through the review process for the first time this semester was the experience which I needed to fully grasp the importance of tPP’s verification process. Assigned 10 cases to review, I began by opening the source files for each case and cross-checking the information I was gleaning with the coding which had already been done. As I checked, I realized that mistakes had been made by the initial coding pair. I proceeded to correct the errors and transfer the updated cases to the official dataset. Without having a third review of the cases I had been assigned, there would have been inaccurate codes in tPP’s dataset, decreasing the validity of the project as a whole.
Ultimately, the 3-step verification process tPP follows ensures the legitimacy of the data. While there are generally fewer cases which require a third review to correct errors than those cases which need rectifying, it is critical that every case follow the same process to maintain the strength of the data. The implementation of coding pairs guarantees that variables which necessitate discussion are discussed before a decision is made. Furthermore, because tPP is composed of circulating membership, a third review by veteran team members on the Steering Committee secures the accuracy of the codes. While not unique to tPP, the multi-person verification process which the project follows enhances the credibility and effectiveness of the data.