Debugging Data Extraction Pipelines
A data extraction pipeline turns formatted text into a useful result.
From Text to Result
A data extraction pipeline turns formatted text into a useful result. Imagine a file containing lines such as New Revision: 39772. The task is not merely to find one number. It is to find every matching revision number, convert the extracted values into numbers, calculate their average, and verify that each stage produced the expected data.
Debugging becomes easier when you treat the process as a sequence of visible stages. Instead of looking only at the final average, inspect the extracted strings, the converted integers, the sum, and the final formatted result. The first stage whose output is wrong identifies where the problem begins.
Matching the Revision Pattern
The input follows a consistent structure: the literal text New Revision: followed by a number. A regular expression can describe that structure. The literal part identifies the correct lines, while a capturing group isolates the digits that should become the extracted value.
In this generated pattern, New Revision: matches the required label. The parentheses identify the capturing group, and the digit-matching part identifies the numerical portion. The capturing group is important because the task is to return the number rather than the entire line.
The findall() method applies the pattern across the available text and collects all matches. Because the pattern contains a capturing group, the collected results are the captured numerical strings. They look numeric, but they are still strings at this stage.
Tracing the Extracted Values
import re
text = "New Revision: 39772\nNew Revision: 39726\n"
pattern = r"New Revision: (\d+)"
extracted = re.findall(pattern, text)
values = [int(item) for item in extracted]
total = sum(values)
average = total / len(values)
result = int(average)
print(extracted)
print(values)
print(total)
print(average)
print(result)- 1The text contains two consistently formatted revision lines.
- 2The pattern describes the literal label and captures the digits that follow it.
- 3
findall()collects the captured values as strings. - 4The list conversion applies
int()to every extracted string. - 5
sum()adds the integer values, and dividing by the count produces the average. - 6The final
int()converts the average to an integer result.
["39772", "39726"]
[39772, 39726]
79498
39749.0
39749The first printed list contains quoted values, showing that the extracted results are strings. The second list contains integer values without quotes. That distinction matters because arithmetic requires the values to be converted before the sum and average are calculated.
Calculating the Average
Once the extracted values are integers, calculate the average in two operations. First, add all the values. Then divide that sum by the number of values. The resulting average is then converted to an integer for the requested formatted result.
A Two-Value Revision Average
Find the integer average of the generated values 39772 and 39726.
- 1Add the values
39772 plus 39726 gives a total of 79498.
- 2Count the values
There are two integer values.
- 3Divide the total
79498 divided by 2 gives an average of 39749.0.
- 4Format as an integer
Converting 39749.0 to an integer gives 39749.
The formatted integer average is 39749.
Finding the First Wrong Stage
A final result can be wrong for several different reasons. The pattern might match too few lines, findall() might return an unexpected collection, conversion might not happen, or the sum and division might be incorrect. Printing intermediate values separates these possibilities instead of treating the whole program as one opaque operation.
| Stage | What to inspect | What an unexpected result suggests |
|---|---|---|
| Extraction | The collection returned by findall() | The pattern or input format may not match the intended lines |
| Conversion | The values after applying int() | The extracted strings may not have been converted before arithmetic |
| Calculation | The sum and the number of values | A value may be missing, duplicated, or included incorrectly |
| Formatting | The final integer result | The average may not have been converted to an integer |
Inspect each stage before trusting the final average.
Debug from left to right. First check whether the right number of lines was matched. Next check whether the extracted values are strings containing the intended digits. Then check the integer list, the sum, the count, and the average. Do not begin by changing the final calculation if the extracted collection is already wrong.
Common Pipeline Mistakes
Common mistake Treating the values returned by findall() as integers
What it looks likeUsing the extracted strings directly in arithmetic
Why it failsfindall()returns strings. Values such as "39772" look numeric but have not yet been converted.Do this insteadApply int() to each extracted value before calculating the sum or average.
Common mistake Checking only the final average
What it looks likeSeeing an unexpected result and changing the division without inspecting the extracted values
Why it failsThe first error may have occurred during matching or conversion rather than during calculation.
Do this insteadPrint the intermediate collection after extraction, the integer collection, the sum, and the average.
Common mistake Extracting the whole formatted line instead of only the number
What it looks likeUsing a pattern that does not isolate the numerical part
Why it failsThe task requires the numerical values, not the label and the number together.
Do this insteadMatch the literal line structure and use a capturing group around the digits.
Common mistake Using the wrong number of values in the average
What it looks likeDividing by a count that does not match the extracted collection
Why it failsThe average requires the sum to be divided by the count of values actually being averaged.
Do this insteadCheck the extracted collection and use its count when calculating the average.
Practice the Trace
A generated input contains these lines: New Revision: 100, New Revision: 140, and New Revision: 120, each on its own line. Describe the expected output at each stage: the list returned by findall(), the list after integer conversion, the sum, the average, and the final integer result.
Hints
- The capturing group should select only the digits.
- The extracted values are strings until
int()is applied. - The sum is 360, and there are three values.
What do you think happens?
What should the final integer result be for the values 100, 140, and 120?
Reveal answer
120
The values sum to 360. Dividing by the three values gives 120, and converting that result to an integer leaves 120.
For a source-based verification target, the material reports an average of 38549 for mbox.txt and 39756 for mbox-short.txt. If your program produces a different result for either file, inspect the intermediate stages rather than assuming the final conversion is the problem.
Pipeline Checklist
- 1Start with the consistent input format and write a regular expression that matches the required label while capturing only the digits. Use
findall()to collect every captured value. Remember that the returned values are strings, so convert them to integers before arithmetic. Calculate the average from the integer values, then convert the average to an integer. Finally, print intermediate values so you can locate the first incorrect stage.
Key Takeaways
- A regular expression can match the fixed text New Revision: and capture the digits that follow it.
findall()collects the captured numerical values as strings.- Convert extracted strings to integers before summing or calculating an average.
- Calculate the average from the total and the number of values, then convert the result to an integer.
- Print each intermediate stage to identify whether an error begins during extraction, conversion, calculation, or formatting.