Data quality pipeline for the Global Shark Attack File

This project is a data preparation and quality assessment pipeline for the Global Shark Attack File, the historical record of shark attacks, built for the Data and Information Quality course at Politecnico di Milano. It takes the dataset from 6,094 rows × 22 columns to 5,980 × 15, raises completeness from 91.54% to 100%, and removes 91 duplicate rows through record linkage. The image at the top shows the missing values in the 15 columns after standardisation and before imputation.

I designed and implemented the pipeline in a team with other colleagues. The notebook was written jointly, so I describe the choices we made together without claiming individual stages as mine. The part that is specifically mine is the publication: I created the public repository, wrote the README and prepared the published version of the notebook.

Context

For the course we built a data preparation pipeline on the dataset assigned to us, "dataset no. 3": shark attacks. The pipeline covers the whole data quality lifecycle, from profiling through cleaning to a second measurement on the cleaned dataset, and produces an analysis-ready CSV. The work was done in Google Colab in the 2025/26 academic year and received a 4/4 grade.

Data

The Global Shark Attack File is maintained by the Shark Research Institute and published at sharkattackfile.net with all rights reserved, so this page cites and links to it rather than republishing the data. The version used has 6,094 records and 22 columns; the most recent is from 11 June 2017, so it is a historical snapshot, not the current archive.

Profiling, with per-column statistics in pandas and a ydata-profiling report, lined up the problems:

• three near-identical Case Number columns, whose two copies differ from the original in 15 and in 2 rows, plus two link columns (href, href formula) that match in all but 56 rows;
• many gaps: Age is missing in 44.7% of records, Species in 49.2%, Time in 53.3%;
• 124 Year values equal to 0;
• inconsistent values, such as 'M ', N or lli for sex and UNKNOWN or 2017 for the fatal outcome;
• free-text dates, from "Reported 07-Jun-2017" to "Last incident of 1994 in Hong Kong", and characters garbled by encoding errors, such as Maranh�o.

Approach

Measure before cleaning

Before changing anything we measured quality: completeness (non-null cells over all cells), syntactic accuracy with regular expressions and allowed domains, uniqueness, distinctness (the share of distinct values among non-null ones) and constancy per column, and consistency with five rules across columns, such as "a deadly injury implies a fatal outcome". Semantic accuracy and timeliness could not be measured: there is no external ground truth and there are no record timestamps.

Cleaning and standardisation

Columns were renamed to snake_case. As the canonical case number we kept Case Number.2, which differs from the original in only 2 values and is more accurate in exactly those two. We dropped age, time and species (at least 44% missing), the other two case-number columns (Case Number and Case Number.1), original order (correlated with the year), href and href formula (redundant with pdf, after recovering one useful value from them) and location (country and area are enough). That leaves 13 columns; with the two date flags added later, the total is 15. The 4 rows from before the Common Era, whose case number starts with 0., are also removed: from 6,094 to 6,090 rows.

Standardisation went column by column. The most demanding steps:

• Dates. Regex normalisation (removing seasons, "reported" and "circa", fixing misspelt months), ten explicit formats, and year ranges collapsed to their first year. When the month or day is unknown, the date gets a placeholder and two flags, true_month and true_day, record whether they are real, so imputed precision is never mistaken for real precision.
• Places. For dual-country entries ("A / B") the first country is kept and historical names are mapped to current ones (BURMA → MYANMAR); by the end of the pipeline there are 173 distinct countries, against 204 in the raw data. Place names garbled by encoding errors are fixed with targeted replacements (for example Maranh�o → Maranhao).
• From free text to categories. Activities, 1,502 distinct values, fall into five keyword categories plus other; injuries into six categories. Junk values such as #VALUE! and generic descriptors in the name field, such as "male", become null.

Missing values

The missing-value matrix at the top was drawn with missingno on the 6,090 remaining rows, before imputation: every white line is an empty cell. The gaps concentrate in name (1,559, inflated by nulling the generic descriptors), sex (577), activity (541) and area (412).

Imputation:

• a missing date becomes 1 January of the known year, with both flags set to false; rows without a reconstructible date are dropped;
• area and country fill each other in, and so do outcome and injury;
• sex is inferred from the first name with gender_guesser, with u when the name is not enough;
• type, activity, name and source get the explicit value unknown.

At the end no null values remain.

Zero years

The year boxplot shows a cluster of values at 0, which the notebook reads as a probable default value. We replaced them with the year taken from the date. At that point 111 rows needed fixing: of the 124 in the raw data, the other 13 were among the rows already dropped earlier. After the fix, years range from 1555 to 2017.

Deduplication with record linkage

After cleaning there were no exact duplicates, yet the same case number does not always mean the same event: during profiling we had already told apart events in different places or times, cases with several victims, and true repeats. So duplicates are searched for with record linkage (the recordlinkage library), not by ID.

• Candidate pairs. We generated the candidate pairs by combining two indexes: sorted neighbourhood on the date, with a window of 11 (the frequency of the most common date), and blocking on country and area. Result: 53,378 candidate pairs.
• Comparison. 13 comparators: Jaro-Winkler on the Soundex encoding of the name and on the case number; cosine similarity on area, source and PDF file name; a date comparator; exact match on year, country, type, activity, sex, injury and outcome. The maximum score is 13.
• Decision. A pair is considered a duplicate if it scores at least 11: that gives 105 pairs. For each pair not already resolved, the record the notebook scores as less complete (more placeholder values, counting an unknown month or day too) is dropped; 91 rows are removed in total.

The top of the score distribution, where the threshold falls:

Results

Quality measurements and a few profiling statistics, on the raw data and on the final dataset of 5,980 rows:

A methodological note on the 100% completeness: it does not mean every missing piece of information was recovered. Part of it comes from explicit fill values, which the metric counts as present: the final dataset contains, for example, 1,498 unknown names, 528 unknown activities and 521 u sex values, and where the area was missing it holds the country name. The dataset no longer has nulls, but it does contain explicit markers for unknown values.

Limitations

• Consistency does not improve everywhere. The injury–outcome rule drops from 98.11% to 89.32%, but it is phrased differently before and after (on free text, then on categories), and the year rule uses different denominators. The other three rules were not re-measured, because the columns involved had been dropped or cleaned.
• Syntactic accuracy by construction. Type, activity, sex and outcome reach 100% partly because the allowed domains include the pipeline's own placeholders.
• No ground truth. The threshold of 11 out of 13 for duplicates comes with no precision or recall measurement, and inferring sex from a first name remains an approximation.
• Analyses not run. Association rules (apriori with mlxtend) are set up in the notebook but switched off.

The full notebook and README are in the GitHub repository; the original data is at sharkattackfile.net.