Automatic Summarization Accuracy Tests: Real Examples Compared

With the rapid adoption of intelligent document processing (IDP) by organizations across industries the world over, there has been a deluge of textual data – invoices, business reports, academic research papers, emails and what not? To sift through each page of volumes of such data to find the right talking points has become a challenge, and here comes the relevance of automatic summarization tools.

Nevertheless, the automatic summarization accuracy of different methods would differ. Some summarization tools paraphrase (abstractive summarization) the entire text to generate sort summaries, while others condense text input by giving weightage to certain phrases or sentences (extractive summarization) and retaining them verbatim. The choice of which type of summarization tool is suitable for your organization is decided by the task at hand.

In this post, we would attempt to understand the automatic summarization accuracy of the two types of summarizations using real-world examples.

Test Setup for AI Summarization Tools

• Objective: Evaluate summarization accuracy across different summarization styles using real-world texts.
• Models Tested: Extractive (rule-based algorithms like LexRank, TextRank), Abstractive (GPT-style large language model)
• Evaluation Metrics: ROUGE Score (Measures word overlap between AI and human reference summary), Factual Accuracy (How correctly the summary reflects the source text), Readability (Human-judged fluency and coherence), Context Preservation (Whether key insights and causal meaning are maintained)

Automatic Summarization Examples from News, Legal, Medical

Example 1 – News Article from BBC

Source: UK won’t relax visa rules for India, Starmer says

Extractive Summary (50 words)

Sir Keir Starmer confirmed the UK will not relax visa rules for India while leading a delegation of over 100 entrepreneurs and cultural leaders to promote trade and investment. The recent UK-India trade deal will reduce export costs for cars, whisky, textiles, and jewellery. Visas were not part of the deal.

Abstractive Summary (65 words)

UK Prime Minister Sir Keir Starmer emphasized that the UK has no plans to ease visa rules for India, focusing instead on strengthening trade, investment, and cultural ties. His delegation aims to boost economic cooperation following the UK-India trade deal, which lowers export costs for goods. While the UK seeks top global talent, new visa routes for Indian workers or students are not planned.

Comparison Table

Metric	Extractive	Abstractive	Notes
ROUGE-1 (Recall/Precision/F1)	0.70	0.67	Extractive retains exact phrases; abstractive rewords content
ROUGE-2 (Bigrams)	0.57	0.55	Extractive preserves factual sequences better
ROUGE-L (Longest Common Subsequence)	0.66	0.64	Extractive slightly better at preserving sentence structure
Factual Accuracy (0–10)	9	9	Both summaries accurately convey visa stance and trade details
Readability (0–10)	7	8	Abstractive flows more naturally; extractive slightly choppy
Context Preservation (0–10)	9	8	Extractive retains all original references; abstractive condenses context

Verdict

• Extractive Summary: Best for accuracy and context retention, especially for policy-focused readers or news reporting.
• Abstractive Summary: Best for readability and narrative flow, suitable for social media or concise briefings.
• Final Choice: Abstractive summary communicates the key points — Starmer’s stance on visas, focus on trade and investment, and UK–India cooperation — in a smooth, readable way that holds attention while remaining accurate.

Example 2 – US Court Verdict on an Environment Protection-related Case

Source: Sierra Club v. U.S. Dept. of the Interior, No. 18-1082 (4th Cir. 2018)

Extractive Summary (50–75 words)

The National Park Service issued a permit for a natural gas pipeline under the Blue Ridge Parkway, creating a 50-foot permanent clearing. Conservation groups challenged the permit, arguing it violated laws protecting the parkway’s scenic and recreational purposes. The Fourth Circuit found the NPS failed to justify the permit’s consistency with park purposes and overlooked key evidence, ultimately vacating the right-of-way permit.

Abstractive Summary (50–75 words)

The Fourth Circuit vacated the NPS permit for a natural gas pipeline under the Blue Ridge Parkway, ruling that the agency failed to justify the project’s alignment with the parkway’s conservation and recreational purposes. The court highlighted that important evidence suggesting the pipeline would harm the park’s scenic and environmental values was overlooked, making the permit inconsistent with statutory protections under the Blue Ridge Parkway Organic Act.

Comparison Table

Metric	Extractive	Abstractive	Notes
ROUGE-1	0.82	0.68	Extractive aligns more closely with original text
ROUGE-2 (Bigrams)	0.74	0.55	Extractive preserves phrases and details better
ROUGE-L (Longest Common Subsequence)	0.80	0.65	Extractive retains sentence structure for legal accuracy
Factual Accuracy (0–10)	9	9	Both accurate
Readability (0–10)	8	9	Abstractive easier to read and digest
Context Preservation (0–10)	9	8	Extractive preserves more specific context

Verdict

• Extractive: Best for legal reference and preserving original phrasing.
• Abstractive: Best for general understanding or summarizing legal outcomes for non-specialists.
• Final Choice: Extractive summary chosen for legal fidelity, contextual completeness, and verifiable phrasing.

Example 3 – Movie Review by an IMDB user

Source: The Fantastic Four: First Steps

Extractive Summary (50 words)

The movie is set in the 1960s, and I really liked the retro vibe. Pedro Pascal did a solid job as Reed Richards, and Vanessa Kirby was great as Sue Storm. The story was okay, nothing too deep. There wasn’t a strong villain presence either—Galactus didn’t feel dangerous, and Silver Surfer barely had any screen time.

Abstractive Summary (55 words)

Set in the 1960s, the movie delivers a charming retro vibe with standout performances from Pedro Pascal and Vanessa Kirby. While the visuals and music are refreshing, the story lacks depth, featuring a weak villain and minimal tension. Despite some fun moments, the pacing feels uneven. It’s a decent start for the new Fantastic Four era.

Comparison Table

Metric	Extractive	Abstractive	Notes
ROUGE-1	0.82	0.76	Abstractive scores slightly lower due to paraphrasing.
ROUGE-2 (Bigrams)	0.74	0.68	Extractive preserves consecutive bigrams; abstractive loses some word-to-word matches.
ROUGE-L (Longest Common Subsequence)	0.79	0.73	Extractive preserves longest common subsequence better; abstractive condenses phrases.
Factual Accuracy (0–10)	10	9	Extractive keeps all facts intact; abstractive slightly condenses details (e.g. “parts felt slow or disconnected” → “pacing feels uneven”).
Readability (0–10)	8	10	Abstractive flows more smoothly and naturally; extractive can feel choppy.
Context Preservation (0–10)	8	9	Both preserve main points; abstractive merges ideas for coherent context.

Verdict

• Extractive Summary: Best for maintaining exact wording and facts; ideal for detailed reference or quotations. Downside: slightly fragmented sentences.
• Abstractive Summary: Best for readable, concise comprehension; conveys reviewer’s sentiment and context smoothly. Slight loss in minor details.
• Final Choice: The abstractive summary is the better overall representation — ideal for reviews, reports, and publication-ready writing.

Example 4 – Medical Record (Fictitious)

Source: Sample Medical Record

Extractive Summary (72 words)

Mr. Tan Ah Kow, 55, has a medical history of hypertension, hyperlipidemia, strokes, and dementia. His cognitive and physical abilities have deteriorated since 2010. Examination on 20 June 2015 showed disorientation to time, place, and person, poor memory, and inability to handle financial or personal matters. Diagnosed with dementia and stroke, he lacks mental capacity for personal welfare and property decisions. Prognosis: unlikely to regain capacity.

Abstractive Summary (70 words)

The report concludes that 55-year-old Mr. Tan Ah Kow, suffering from dementia and post-stroke complications, shows severe cognitive decline. He cannot understand, retain, or process simple information, nor manage daily or financial decisions. Regular assessments since 2010 confirm progressive deterioration. His doctor certifies total loss of mental capacity concerning both personal welfare and property matters, with no expected recovery due to irreversible dementia.

Comparison Table

Metric	Extractive	Abstractive	Notes
ROUGE-1	0.91	0.87	Measures overall word overlap with key content; extractive slightly better due to direct text usage.
ROUGE-2 (Bigrams)	0.83	0.79	Reflects phrase-level consistency; abstractive maintains semantic equivalence with slight rewording.
ROUGE-L (Longest Common Subsequence)	0.89	0.84	Extractive preserves longest common subsequence better; abstractive condenses phrases.
Factual Accuracy (0–10)	10	9	Extractive is fully aligned with source data; abstractive simplifies some phrasing but remains accurate.
Readability (0–10)	7	9	Extractive is clinical and factual; abstractive uses smoother, more narrative language.
Context Preservation (0–10)	10	9	Both maintain the central message (cognitive decline, incapacity, prognosis); extractive edges are ahead in precision.

Verdict

• Extractive Summary: Best for medical, legal, or procedural contexts where factual precision is paramount.
• Abstractive Summary: Ideal for readability, public reports, or inclusion in summary datasets.
• Final Choice: Extractive summary for documentation accuracy; abstractive summary for general audience communication.

Conclusion

As is evident, both kinds of intelligent automation summaries do not miss much on accuracy, but abstractive scores more on readability, as it generates more human-like and comprehensive texts, while extractive remains closer to factual data, making it ideal for medical and legal summaries.

Let’s see a simple chart that explains the pros and cons of extractive vs abstractive summarization:

Attributes	Extractive	Abstractive
Accuracy and Faithfulness	Wins for factual accuracy, using original wording	Better at capturing overall themes and implications
Length and Readability	Can be longer, sometimes with awkward transitions	Generally, more concise and fluid
Processing Speed	Faster, better for real-time summarization	Slower, but improving with advanced AI
Content Suitability	Ideal for fact-heavy presentations, technical talks	Shines with narrative content, panel discussions

Nevertheless, one must be aware of the fact that abstractive summaries are sometimes prone to unintentional text hallucination, where they generate content which is not in sync with the context of the original text document.

Also, abstractive summary, while good for summarizing multiple documents unlike extractive, it is not ideal for summarizing text-heavy (lengthy and wordy) documents. Longer texts will have to be broken down into shorter text inputs, and the summaries will have to be compiled accordingly.

However, advancements in AI automation have given rise to hybrid summarization techniques, where the tool first uses the extractive method to capture the right facts and then uses the abstractive method to generate the summary in a more human tone than just stitching together lines from the original document.

So, are you ready to dive into the world of AI summarization tools? Partnering with an industry expert in the field like DeepKnit AI will go a long way in helping you find the right AI solutions.