This continues our series of student reflections and analysis authored by our research team.
Keeping Up with the Codebook
Izzy Bielamowicz
One of the greatest challenges I have faced as a member of tPP is keeping up with the constant changes being made to the codebook. Actively following the guidelines of the codebook while coding is essential to maintaining the consistency and quality of tPP’s work. To ensure that the project is steadily evolving and progressing to address issues which inevitably arise during the coding process, the codebook is edited correspondingly. Every few weeks, the project’s codebook is republished, and any changes which may have been made to any given variable must be adopted by all of the coders.
As stated in the manual, “when coding cases, you should always have the codebook open as a reference, even when you become more familiar with the variables.” From personal experience, I can attest that it is easy to fall into the comforting pattern of repetition, but there have conversely been many instances in my time with the project where I realize that I have made mistakes due to habitual coding processes – failing to adapt to new variables or values which may be more fitting for any given case.
Adapting to the changes made to the codebook was arguably the most difficult following a semester of analysis, independent of coding. Without the reiteration of referencing the codebook weekly to complete my assigned cases, I struggled jumping back into coding at the beginning of this semester. Many changes had been made to the codebook in the time since I had last actively coded, including completely new variables such as “Hate crime” and “Group identifier.” It is a constant learning process to be a member of tPP, specifically with the codebook and the structure of the project. My experience with the codebook evidences the difficulty of maintaining consistency throughout the process as tPP is best explained as an evolutionary team, with new members joining, and old members departing the project.
Ensuring that coders adopt any updates to the codebook in their work is next to impossible to monitor. This introduces a very crucial conversation to the tPP narrative – accountability of team members. Within the course associated with the project, multiple tools have been implemented to promote accountability. The primary tool, a simple percentage grade based on accuracy of coding and quality of work, has apparently been quite the powerful incentive for team members to code according to the updated codebook. Additionally, quizzes have been administered to the class to check for understanding of the language of the codebook and individual ability to interpret the codebook in practice.
As mentioned in my previous post, this semester I was given the opportunity to join the Steering Team (ST). Among other roles and responsibilities, it is within the realm of ST privileges to rewrite and rework the codebook – constantly bettering and clarifying the language. Housed within a restricted shared drive is a working version of the codebook available for edit by all ST members. All ST members are welcomed and encouraged to comment and improve the document, accounting for problems which they have encountered themselves, or which have been brought to their attention by coders.
Needless to say, the most recent version of the codebook was updated at the beginning of November. Changes made to Codebook v4.06 include an amendment made to the ‘Veteran status’ variable, a value change within the ‘Tactic’ variable, and an added value within the ‘Ideological target’ variable. While all of the changes made to the codebook (tracked since August of 2019) are relatively narrow in scope, and oftentimes arise as a result an issue within the coding of a singular case, these changes assure the adaptation of tPP to its ever increasing load of potential cases. As the pool of potential cases for inclusion grows, the coding process inevitably becomes more complex as new phenomena occur and more obscure cases are added to the dataset.
For example, a partial list of revisions made in October and November 2019 (time of writing this post) can be seen below:
Changes made November 2019
- 11/12/19: Added value “Identity: other” to variable “ideological target” in order to accommodate case where victim was targeted due to perceived mental illness and homelessness status.
- 11/04/2019: In the variable ‘Tactic’, ‘Other’ was changed to ‘Uncategorized’ in order to avoid confusion with ‘Other weapons’.
- All changes were applied retroactively to the entire dataset.
- 11/01/19: The variable ‘Veteran status’ was amended to officially exclude individuals involved only in non-state military activity from holding veteran statuses.
Changes made October 2019
- 10/31/19: In the variable ‘Tactic’, ‘Blade or blunt weapon’ was changed to ‘Other weapons’ in order to include mace, tasers, and nooses.
- All changes were applied retroactively to the entire dataset.
- 10/29/19: The variable ‘Veteran Status’ was updated by removing the variable ‘Unknown’, unless otherwise stated a defendant is considered a civilian.
- All changes were applied retroactively to the entire dataset.
- 10/28/19: Codebook version 4.0 published including new appendix (revisions and addition history) and introduction.
- 10/18/19: The variable ‘Source Descriptions’ was updated to include a list of possible codes rather than it being an open ended variable.
- 10/15/19: The procedures for how source documents are labeled and field was updated
- 10/14/19: In the variable ‘Group Affiliation’, ‘No affiliation’ was changed to ‘No known affiliation’.
- All changes were applied retroactively to the entire dataset.