Day 3 — Verify and Evaluate Information
Day 3 is where we verify the sources we gathered on Day 2 and read the most important ones closely — separating what’s genuinely supported by the literature from what an AI tool merely asserted. I’ll keep going with the illustrative health misinformation brief.
Daily Recap
Not every source from Day 2 deserves the same scrutiny. We decide together which ones warrant real verification and close reading, and I prioritise using two questions: how central is this source to our argument? and how much are we taking its claims on trust? A paper I plan to lean on as a main baseline gets verified and read carefully; a source mentioned once in passing I’ll check more lightly.
Rough categorization of papers:
| Priority | Source type | Action |
|---|---|---|
| High | Papers central to the problem / candidate baselines | Verify and read critically today |
| Medium | Supporting evidence for a specific claim | Verify existence and check the specific claim |
| Low | Background / context only | Confirm it exists; park for later |
I also like to set a concrete reading goal before diving in — something like “verify all six shortlisted sources and read the two most central in full.”
Verifying Information
Before any source informs your proposal, I would confirm two things: that it exists as described, and that the claims attributed to it are actually true.
Obtaining BibTex
I search the exact title in Google Scholar and check that the authors, venue, and year line up with what the tool told me. If it doesn’t show up, I’ll try a distinctive phrase from the title in quotes. A source I can’t locate doesn’t make it into my bibliography — no exceptions.
Once I’ve found it, I grab a clean citation via Cite → BibTeX and paste it into my .bib file in Overleaf:
@inproceedings{author2023misinfo,
title = {Explainable Detection of Health Misinformation in Short Text},
author = {Author, A. and Author, B.},
booktitle = {Proceedings of EMNLP},
year = {2023}
}Scholar often lists a published paper’s venue as arXiv preprint even after it appeared at a conference. However, that best practice is to look for the original source/venue (e.g., ICML) that the article is published at!
Validating the claims
There are many patterns of AI hallucination and misattribution! Below are some oft he few I keep an eye out for:
- Incorrect venues — these hallucinations happen most frequently when you specify which venue (e.g., ICML) you want the AI to search from .
- Invented papers — a plausible title paired with real authors who never wrote it.
- Misattribution — a real finding credited to the wrong paper, or two papers’ results merged into one.
- Overstated claims — “this method solves X” when the paper reports a modest improvement under specific conditions.
- Dropped caveats — the dataset, language, or setting a result depends on gets quietly omitted (e.g., an English-only result reported as if it were general).
- Fabricated numbers — precise-sounding statistics that don’t actually appear in the source.
An AI survey claims: “Tan et al. (2022) show that multilingual transformers achieve 92% F1 on code-mixed health misinformation.” Upon checking, Tan et al. evaluate on monolingual English misinformation and never touch code-mixed text; the 92% is on a different task entirely.
For each source on your team’s Day 2 shortlist: (1) find it on Google Scholar and confirm authors/venue/year, (2) export and tidy up its BibTeX entry (or better yet, find the original source), and (3) write 2-3 points about it in your own words and compare them against the AI summary. Drop any source you can’t verify, and flag any claim that doesn’t hold up.
Bringing it together
In the last section, we went through how to verify information. In this one, we’ll work through one or two key papers together to extract relevant information more deeply from more papers. We will be doing this manually.
A structured way in is the three-pass method on the Reading Advice page: a quick bird’s-eye pass, a pass to understand the contribution, and a deep pass only for the papers that earn it. For each key paper, I want to be able to extract three things:
- Contribution — what the paper actually adds. What can we do now that we couldn’t before?
- Limitations — where it falls short: assumptions, narrow evaluation, settings it doesn’t cover.
- Gaps — what it leaves unaddressed. A gap that overlaps my problem is an opening for my own contribution.
Reading a paper on explainable misinformation detection:
- Contribution: a token-attribution method that lets a moderator see which words drove a “misinformation” flag, with a user study showing moderators trust it more than a black-box score. This is often found at the end of the Introduction section.
- Limitations: evaluated only on English Twitter data; the user study had 12 participants; no code-mixed text. This is often found at the Discussion section.
- Gaps (our opening): nobody has tested whether token attributions stay meaningful for code-mixed Singlish, where the model may attend to the “wrong” language’s tokens. That’s exactly where our project can contribute.
As a team, pick the single most central paper from your verified shortlist and read it without AI assistance. Write down its contribution, its limitations, and at least one gap. Then try phrasing that gap as a one-sentence opening for your own research.
Team Review
Each team shares its most important or surprising finding with the room, and then we converge on the research idea with the highest potential for impact. The framing I aim for: “Prior work does X but not Y; Y is our opening.”
A verified shortlist of sources, each annotated with its contribution and limitations, and a short note on the gap the team intends to address. This carries into tomorrow’s (Day 4) proposal design.