Fallacies are arguments that rely on faulty or misleading reasoning (Goffredo et al., 2023).
The goal of this project is to improve fallacy detection and classification in political discourse using Retrieval-Augmented Generation (RAG). By grounding models in external knowledge sources, the system aims to provide more accurate classifications and clearer explanations of fallacious arguments in US political debates.
The main experiments use the ElecDeb60to20 dataset, which contains transcripts from U.S. political debates from 1960 to 2020. The same pipeline also runs on the propaganda dataset, and a separate relation-classification pipeline draws on the LIARArg corpus.
Both an encoder track (BERT-style fine-tuning) and an LLM track (zero-shot, few-shot, and LoRA/QLoRA fine-tuning) are supported for every task, each with and without RAG context.
Headline results. Across 42 retrieval configurations × 14 models, argumentatively guided retrieval raises the best macro-F1 from 0.772 → 0.864 on detection and from 0.653 → 0.725 on classification (ModernBERT in both cases). Retrieval improves every encoder tested except NeoBERT; decoder-only LLMs degrade once retrieved context is added.
Each sentence is turned into a retrieval query — the bare sentence, or the sentence enriched with the argumentative components that support or attack it. The query is sent to the knowledge base through one of four retrievers (three dense embedding models or BM25), optionally restricted to documents predating the debate and optionally reranked by a cross-encoder. The retrieved passages are then concatenated with the sentence and its speech turn into the model input, and an encoder or a decoder produces the final prediction.
Crucially, retrieval happens offline. The create_db.py scripts run the whole retrieval stage once per
configuration and write the passages into a context column, so the training scripts only ever read a pre-built
column. This makes every training run cheap, deterministic, and directly comparable across retrieval settings.
The project covers three tasks, each documented in its own README.
Binary decision of whether a segment of speech contains a fallacious argument (Fallacy / NoFallacy).
See src/detection/README.md.
Given a sentence already identified as fallacious, classify which type of fallacy it contains. Unlike detection,
this task never sees non-fallacious sentences (NoFallacy sentences are dropped during dataset construction).
See src/classification/README.md.
The label inventory is dataset-aware and is selected automatically from the input path. Single- vs. multi-label is auto-detected from the dataset columns.
- ElecDeb60to20 — six categories, exactly one per sentence:
- Ad Hominem: attacking the opponent personally instead of their position or reasoning.
- Appeal to Emotion: emotionally charged language used to manipulate feelings rather than provide logical support.
- Appeal to Authority: relying on an authority figure or perceived consensus (including non-experts or majority opinion) without sufficient evidence.
- Slippery Slope: claiming a small first step will inevitably lead to extreme or unlikely consequences.
- False Cause: assuming a causal link between two events merely because they co-occur.
- Slogan: a short, striking phrase meant to provoke agreement rather than argue.
Classify the argumentative relation between a pair of components as support, attack, or no_relation. This
task serves the RAG pipeline: relations feed the relations query mode of the dataset builders (see below), enriching
the retrieval query with the fallacy's supporting/attacking context.
Where those relations come from depends on the dataset. ElecDeb60to20 ships gold relation annotations, so its
create_db.py reads them straight from relations.csv — no classifier involved. The propaganda dataset has no
relation layer, so relations are predicted there by a classifier trained on LIARArg.
See src/relation-extraction/README.md.
To ground the model, we constructed a knowledge base of approximately 15GB of political and contextual documents,
crawled between May and June 2026. The corpus combines official government sources with curated background
information, organized into six sections (the research papers inside Definitions are indexed as their own papers
category, so the vector store carries seven).
- Bills and Laws: congressional bills and public/private laws approved by the U.S. Congress, collected from their official repository GovInfo, the U.S. Government Manual, and the Universal Declaration of Human Rights.
- Geography: geographic and demographic information about U.S. states, from the U.S. Census and Wikipedia.
- People: biographical information about members of the U.S. Congress, from the Biographical Directory of the United States Congress, plus Wikipedia entries for public figures referenced during the debates.
- History: U.S. historical context, built from per-year Wikipedia "{year} in the United States" pages (from 1900 onward) together with broad "History of the United States" era-overview pages and their linked articles. Each record is tagged with its year to support temporal filtering during retrieval.
- Definitions: conceptual definitions and explanations of argumentative fallacies compiled from peer-reviewed research (Goffredo et al., 2023, Goffredo et al., 2022), 200 fallacy / argumentation / rhetoric papers published from 2015 onwards, retrieved from OpenAlex and filtered by lexical relevance over title and abstract, and the Wikipedia fallacy pages.
- Exemplars: labeled examples of fallacious arguments drawn from public fallacy datasets (LOGIC and MAFALDA), mapped onto the project's fallacy taxonomy to provide concrete reference instances of each fallacy type. Exemplar texts that overlap an ElecDeb60to20 instance are removed, so retrieval cannot leak gold labels into the evaluation.
| Collection | Size | # Documents | # Paragraphs |
|---|---|---|---|
| Bills | 14 GB | 178,383 | 6,668,178 |
| Geography | 516 MB | 125,891 | 125,891 |
| People | 297 MB | 72,950 | 72,950 |
| History | 89 MB | 16,121 | 16,121 |
| Exemplars | 5 MB | 1,273 | 1,284 |
| Definitions | 1.3 MB | 65 | 65 |
| Total | 15 GB | 394,683 | 6,884,489 |
Sections are built independently by src/create_knowledge_base.py into
knowledge_base/<section>/, then indexed into a per-embedder Chroma store by
src/create_persistent_db.py.
create_knowledge_base.pyalso ships anewspapers()builder (Chronicling America historic newspapers). It is not part of the 15GB knowledge base reported above and was not used in the experiments — drop it frommain()to reproduce the paper's corpus exactly.
Indexing splits long documents into chunks of at most 1,000 tokens with 10% overlap (--chunk-size /
--chunk-overlap). Chunking is by token, not character, and the chunk size is clamped to the input window of the
embedding model in use so no passage is silently truncated at encoding time. There is one store per embedding model,
over a fixed document set, so retrievers can be compared in isolation.
Four retrievers are compared: three dense encoders — sentence-transformers/all-mpnet-base-v2 (SBERT),
BAAI/bge-large-en-v1.5 (BGE) and Sentence-RooseBERT (domain-specific, pre-trained on political text) —
against BM25 as a lexical baseline. At inference the top k=5 chunks are kept per query; in hybrid search the
dense and BM25 rankings are fused with Reciprocal Rank Fusion (kRRF=60) over 30 candidates per leg, then
re-ranked with a ms-marco-MiniLM-L-6-v2 cross-encoder before keeping the top 5.
To inspect what a query retrieves without running the full pipeline, use
src/perform_retrieval.py:
python src/perform_retrieval.py \
--query "social security reform" \
--retriever sentence-transformers/all-mpnet-base-v2 \
--search-type similarity --top-k 5Each entry in the knowledge base is stored as a JSON document.
{
"id": "",
"text": "",
"xml": "[when available]",
"source": "",
"url": "",
"year": "[when available]",
"fallacy_label": "[exemplars only]"
}The id, source, url, year and fallacy_label fields become chunk metadata in the vector store, which is what
makes temporal filtering (--temporal-rag) and per-section indexing possible.
| Dataset | Tasks | Role |
|---|---|---|
| ElecDeb60to20 | detection, classification | Main dataset (U.S. political debates, 1960–2020). |
| propaganda | detection, relation-classification | Second detection dataset; relation predictions come from the LIARArg model. |
| LIARArg | relation-classification | Training data for the relation classifier (support / attack / no_relation). |
Each dataset lives under datasets/<name>-dataset/. ElecDeb60to20 and propaganda keep raw train.csv / dev.csv /
test.csv splits at the dataset root (ElecDeb60to20 additionally ships relations.csv, the gold relation layer);
propaganda and LIARArg keep their source files in an original/ folder. RAG-enriched variants are written into
per-task, per-config subfolders by create_db.py.
Split sizes after create_db.py. Detection sentence-splits the speech turns with spaCy, propagates the span-level
annotations down to the sentence level, then downsamples NoFallacy to a 1:1 ratio and re-splits 80/10/10
stratified by label — 2,612 / 327 / 327 train/dev/test sentences. Classification keeps only the fallacious portion
of the corpus — 1,306 / 164 / 163 — and stays sharply imbalanced (AppealtoEmotion alone is 739 of the 1,306
training instances; Slogans is 34), which makes it the harder of the two tasks.
Git LFS. Every CSV under
datasets/is tracked with Git LFS (see.gitattributes). Rungit lfs install && git lfs pullafter cloning, otherwise the splits will be pointer files rather than data.
There is one folder per retrieval configuration, and its name is the configuration. Both the enriched datasets and the results mirror this naming, so a results folder can always be traced back to the data that produced it:
temporal=true-retriever=ddore14-sentence-roosebert-search=hybrid-rerank=cross-encoder-ms-marco-minilm-l-6-v2-query=relations_labeled-summarizer=false
│ │ │ │ │ └── passage summarization model, or false
│ │ │ │ └── query mode: fallacy_relations | relations_labeled (omitted in sentence mode)
│ │ │ └── cross-encoder reranker, or false
│ │ └── search strategy: similarity | mmr | bm25 | hybrid | similarity_score_threshold
│ └── dense embedding model, bm25, or false for no retrieval
└── temporal filter: only retrieve documents predating the debate
The no-RAG baseline is therefore temporal=false-retriever=false-summarizer=false.
Create the conda environment (named rags):
conda env create -f environment.yaml
conda activate ragsA few things are needed on top of the environment file:
| Requirement | Why |
|---|---|
git lfs |
The dataset CSVs are LFS objects — git lfs pull after cloning. |
python -m spacy download en_core_web_sm |
Sentence splitting inside create_db.py. |
python -m spacy download en_core_web_trf |
Named-entity recognition over the debates in notebooks/topic_analysis.ipynb. |
poppler, tesseract |
OCR of PDF sources (OpenAlex papers, UDHR) while building the knowledge base. |
HF_TOKEN |
Gated Hub models (Llama, Mistral). Or pass --token. |
OPENAI_API_KEY |
Only for the LLM scripts run with --backend openai. |
CONTACT_EMAIL |
Sent in the OpenAlex User-Agent when collecting fallacy papers for the knowledge base. |
The .project-root marker file is what rootutils uses to resolve paths — do not delete it, and
run every script from the repository root so relative paths like datasets/… resolve correctly.
The script/ folder is numbered to reflect the pipeline order. Each .sh is an sbatch script, but nothing prevents
running them directly with bash on a single machine.
| Stage | Script | What it does |
|---|---|---|
| 00 | 00-create_knowledge_base.sh |
Builds every knowledge-base section (src/create_knowledge_base.py). |
| 01 | 01-create_persistent_db.sh |
Indexes the knowledge base into a persistent Chroma store, once per embedding model. |
| 02 | 02-create_database.sh, 02-create_database-propaganda.sh |
Runs each dataset's create_db.py over the retrieval grid to produce RAG-enriched splits. |
| 03 | 03-fallacy_no_rag.sh, 03-fallacy_no_rag_llm.sh |
No-RAG training sweep, encoder and LLM tracks. |
| 04 | 04-fallacy_rag.sh, 04-fallacy_rag_llm.sh |
RAG training sweep over the configs listed in the script's DATASETS array. |
| 05 | 05-relation_classification.sh, 05-relation_classification_llm.sh |
Relation-classification sweep on LIARArg. |
| 06 | 06-best_hparams_*.sh |
Generated, not written by hand — see below. |
Stages 00–01 only need to run once. Stage 02 is the expensive one: it performs all retrieval up front, so stages 03–05 never touch the vector store.
The 43 configurations.
02-create_database.shsweeps the dense grid — 2 temporal settings × 3 embedding models × 2 search strategies (similarity,hybrid) × 3 query formulations — plus the no-RAG baseline, i.e. 37 of the 43 configuration folders. The remaining six (retriever=bm25, 2 temporal × 3 query formulations) are built by callingcreate_db.pydirectly with--retriever bm25 --search-type bm25. All 43 are already present underdatasets/elecdeb60to20-dataset/<task>/.
After a single-seed sweep, script/generate_best_hparam_script.py picks the
best hyperparameter cell per model and emits a 06-best_hparams_*.sh that re-runs only those, once per seed:
python script/generate_best_hparam_script.py \
output/one-run/elecdeb60to20-detection/temporal=false-retriever=false-summarizer=false \
--seeds 14 42 55 89 3This writes script/06-best_hparams_<data>_<task>_<config>.sh for the encoders and a matching ..._llm_... script for
the LLMs, whose results land under output/five-runs/.
The 03-*.sh / 04-*.sh scripts default to ElecDeb60to20 detection. Both are set by two variables near the top:
data="elecdeb60to20" # elecdeb60to20 | propaganda
task=detection # detection | classificationThese feed the input paths (datasets/$data-dataset/$task/…), the script actually invoked
(src/$task/run_$task.py), and the output paths (output/one-run/$data-$task/…). The 04-*.sh scripts additionally
take a DATASETS=() array listing which configuration folders to sweep.
| Track | Models | Grid |
|---|---|---|
| Encoders | DeBERTa-v3, Longformer, ModernBERT, RoBERTa, LegalBERT, NeoBERT, and the four RooseBERT variants (cont/scr × cased/uncased) — 10 | batch ∈ {8, 16} × lr ∈ {2e-5, 3e-5, 5e-5} × epochs ∈ {3, 5}, warmup ratio 0.06 — 12 cells per model |
| LLMs | Llama-3.1-8B-Instruct, Qwen3.5-9B-Base, gpt-oss-20b, Mistral-7B-Instruct-v0.3 — 4 | zero-shot, few-shot (one example per class), and LoRA fine-tuning (r=16, α=32, dropout 0.05) at lr ∈ {1e-4, 3e-4, 5e-4} × epochs ∈ {1, 2, 3}, effective batch 8 |
Runs are named after their cell (bs16_lr3e-5_ep5, mode-few_shot, mode-fine_tune_bs8_lr3e-4_ep2), which is how the
aggregation scripts recover the hyperparameters.
Why the grid is not run exhaustively. 42 configurations × 14 models × the hyperparameter grid × 5 seeds is not computationally feasible, so the sweep runs in three stages: (i) every model is trained on the no-RAG baseline over the full hyperparameter grid, and the best cell per model is re-run with five seeds; (ii) only the three strongest baseline models per task are run over the remaining 42 retrieval configurations, which selects the winning configuration; (iii) all 14 models are re-run on that configuration, sweeping hyperparameters and then five seeds. The selected configuration is therefore optimal for those three models and merely reasonable for the rest.
The winning configurations are Sentence-RooseBERT + hybrid search + cross-encoder reranking +
relations_labeled query, with temporal filtering for detection and without it for classification.
| Script | Purpose |
|---|---|
script/check_one_run.py |
Verifies the single-seed grid is complete; reports gaps and exits non-zero. |
script/check_five_runs.py |
Same check for the multi-seed grid (all models × 5 seeds). |
script/extract_results.py |
Aggregates output/one-run/ into a results.csv per config plus a combined all_results.csv. |
script/extract_results_5runs.py |
Aggregates output/five-runs/ into results_mean_std.csv (mean, std, seed count) plus all_results_mean_std.csv. |
script/ablation.py |
Leave-one-out ablation of the RAG components, anchored on the strongest no-retrieval model per task with hyperparameters frozen at the anchor cell, so deltas stay selection-free. Single seed (14). |
script/ttest_best_models.py |
Paired significance tests between the baseline and each config's best model over the five paired per-seed macro-F1 values: two-sided t-test, Shapiro-Wilk, Wilcoxon, an exact sign-flip permutation test, and both Cohen's d and Hedges' g. |
A run is considered complete when it carries its completion marker: predict_results.json for encoders,
predict_results_metrics.json for LLMs. Every aggregation and check script keys off these two filenames.
The notebooks/ folder holds the qualitative analyses: error_analysis.ipynb (confusion matrices comparing the
baseline against the best RAG config, exported to the cm_*.pdf files), knowledge_base_analysis.ipynb (KB
composition and document-length statistics), topic_analysis.ipynb and topic_similarity_search_kb.ipynb (debate
topic extraction and how well the knowledge base covers those topics).
src/
create_knowledge_base.py # build the KB (bills, people, geography, history, papers, exemplars…)
create_persistent_db.py # index the KB into a persistent Chroma vector store
perform_retrieval.py # CLI to query the vector store (dense / BM25)
detection/ # binary fallacy detection (encoder + LLM) — see its README
classification/ # fallacy-type classification (encoder + LLM) — see its README
relation-extraction/ # relation classification + relation prediction — see its README
utils/
fallacy_loader.py # LangChain loader turning KB JSON records into Documents
dataset_prompts.py # dataset-aware system prompts and label spaces for the LLM scripts
util.py # HTTP, Wikipedia and OCR helpers used by the KB builder
datasets/ # source data + per-dataset RAG-enrichment scripts
elecdeb60to20-dataset/ # main dataset
train.csv, dev.csv, test.csv # raw splits
relations.csv # gold argumentative relations (support / attack)
detection/create_db.py # builds RAG-enriched detection datasets
classification/create_db.py # builds RAG-enriched classification datasets
<task>/<config>/ # generated RAG-enriched splits (43 folders per task)
propaganda-dataset/ # detection + relation-classification data
original/ # source files
train.csv, dev.csv, test.csv # raw splits
detection/create_db.py # builds RAG-enriched detection datasets
relation-classification/ # prediction pairs + predicted relations
LIARArg-dataset/ # relation-classification training data
original/ # source files
relation-classification/ # train.csv, dev.csv, test.csv pairs
script/ # sbatch / driver scripts (numbered by pipeline stage)
00-create_knowledge_base.sh
01-create_persistent_db.sh
02-create_database*.sh # runs create_db.py to build RAG-enriched datasets
03-fallacy_no_rag*.sh # no-RAG training sweep (encoder / LLM)
04-fallacy_rag*.sh # RAG training sweep (encoder / LLM)
05-relation_classification*.sh # relation-classification sweep (encoder / LLM)
generate_best_hparam_script.py # generates a 06-best_hparams_*.sh from a finished sweep
extract_results.py # aggregates output/one-run into per-config CSVs
extract_results_5runs.py # aggregates output/five-runs into per-config CSVs
check_one_run.py, check_five_runs.py # verify grid completeness
ablation.py # leave-one-out ablation of the RAG components
ttest_best_models.py # paired significance tests across configs
notebooks/ # topic / knowledge-base / error analysis
misc/ # auxiliary source files (congress roster, UDHR, source URLs, topic coverage)
figures/pipeline.png # the pipeline figure shown above
figures/cm_*.pdf # confusion matrices exported by notebooks/error_analysis.ipynb
output/
one-run/<dataset>-<task>/<config>/<model>/<run>/ # single-seed grid-search results
five-runs/<dataset>-<task>/<config>/<model>/seed_<n>/<run>/ # multi-seed best-hparam results
trained-models/ # checkpoints reused by the relation-prediction scripts
Each create_db.py reads its raw splits from --dataset-dir (default datasets/<name>-dataset) and writes the
RAG-enriched splits back under <dataset-dir>/<task>/<config>/{train,dev,test}.csv, where <config> encodes the
retrieval settings as described in Configuration Folder Names.
Inria holds the ownership rights on the code and data in this repository. Use is granted free of charge to academic
users for research purposes only; any other use requires prior consent from Inria. See LICENSE for the
full terms.