Fact Verification with Semi-Structured Knowledge

Fact Verification with Semi-Structured Knowledge

Fact Verification with Semi-Structured Knowledge Yi-Lin Tuan, Wenhu Chen,William Wang Outline Background Related Work FEVER Pipeline TabFact Conclusion 2 Misinformation Our society is struggling with an unprecedented amount of falsehood, hyperbole, and half-truths. 3 Misinformation Our society is struggling with an

unprecedented amount of falsehood, hyperbole, and half-truths. Politicians and organizations repeatedly make false claims that jeopardize the integrity of journalism. 4 Fact Verification To fight false information, the need for fact verification has never been so urgent. 5 Fact Verification To fight false information, the need for fact verification has never been so urgent. The number of fact-checking organization has grown from 44 in 2014 to 115 in 2017 6

Fact Verification The challenge is that the human factcheckers cannot keep up with the amount of misinformation and the speed at which it spreads. 7 Fact Verification The challenge is that the human factcheckers cannot keep up with the amount of misinformation and the speed at which it spreads. We need automatic tools or techniques to help us in fact-checking in various areas 8 Fact Verification The challenge is that the human factcheckers cannot keep up with the amount of misinformation and the speed at which it spreads. We need automatic tools or techniques to help us in fact-checking in various areas ClaimBuster (Naeemul et al. 2019) is

invented as the first end-to-end factchecking system. 9 Related Research Fact Verification can be viewed as a natural language understanding problem. World Evidence Entailed/ Refuted Textual Hypothesis 10 Related Research Fact Verification can be viewed as a natural language understanding problem. Verify textual statement given the world knowledge as evidence.

World Evidence Entailed/ Refuted Textual Hypothesis 11 Related Research Recognizing Textual Entailment (Dagan et al. 2006) Recognizing whether the meaning of the first sentence is contained in the second sentence. 12 Related Research Recognizing Textual Entailment (Dagan et al. 2006) Recognizing whether the meaning of the first

sentence is contained in the second sentence. Natural Language Inference (Bowman et al. 2015) Recognizing whether a premise text can entail another hypothesis text. 13 Related Research Recognizing Textual Entailment (Dagan et al. 2006) Recognizing whether the meaning of the first sentence is contained in the second sentence. Natural Language Inference (Bowman et al. 2015) Recognizing whether a premise text can entail another hypothesis text. Text Fragments

Entailed/ Refuted Textual Hypothesis 14 Related Research Claim Verification (Popat et al. 2017) Verify claims from social media 15 Related Research Claim Verification (Popat et al. 2017) Verify claims from social media Fact Checking (Thorne et al. 2018) Verify claims modified from Wikipedia 16 Related Research Claim Verification (Popat et al. 2017)

Verify claims from social media Fact Checking (Thorne et al. 2018) Verify claims modified from Wikipedia Passage Entailed/ Refuted Textual Hypothesis 17 FEVER Given: 5 million Wikipedia documents Statement Output: Verdict: (SUPPORT, REFUTE, NOT ENOUGH INFO) The dataset simulates the real-world factchecking and extraction scenarios.

18 FEVER Document Retrieval TF-IDF BM25 Keyword Matching ( , ) = 19 FEVER Document Retrieval TF-IDF BM25 Keyword Matching ( , ) = Wikipedia Documents

20 FEVER Document Retrieval TF-IDF BM25 Keyword Matching ( , ) = Claim Wikipedia Documents 21 FEVER Document Retrieval TF-IDF BM25 Keyword Matching ( , ) =

Retrieved Evidence Claim Wikipedia Documents 22 FEVER Sentence Retriever Semantic Matching ( , ) = 23 FEVER Sentence Retriever Semantic Matching

( , ) = Retrieved Evidence 24 FEVER Sentence Retriever Semantic Matching ( , ) = Claim Retrieved Evidence

25 FEVER Sentence Retriever Semantic Matching ( , ) = Sentences Claim Retrieved Evidence 26 FEVER Sentence Retriever Semantic Matching

( , ) = Sentences Claim Rank Network Retrieved Evidence 27 FEVER Sentence Retriever Semantic Matching ( , ) =

Sentences Claim Rank Network Retrieved Evidence 28 FEVER Sentence Retriever Semantic Matching ( , ) = Retrieved Sentence

Claim Retrieved Evidence 29 FEVER Verifier (NLI) Retrieved Sentence Claim Network 30 FEVER Verifier (NLI) Retrieved Sentence Claim Network

{ , , } 31 Free-form Evidence Fever simulates a realistic open-domain verification testbed for fact verification based on publicly available free-form language evidence. 32 Free-form Evidence Fever simulates a realistic open-domain verification testbed for fact verification based on publicly available free-form language evidence. Can we go further to utilize other publicly available evidence for fact verification? 33 What else can we leverage Source:

https://en.wikipedia.org/wiki/Houston_Roc kets Claim: Houston Rocket won 45% of the game in 12-13 season. 34 What else can we leverage Source: https://en.wikipedia.org/wiki/Houston_Roc kets Claim: Houston Rocket won 45% of the game in 12-13 season. 35 What else can we leverage Source: https://en.wikipedia.org/wiki/Houston_Roc kets Claim: Houston Rocket won 45 games in 12-13

season. 36 What else can we leverage Source: https://en.wikipedia.org/wiki/Houston_Roc kets Claim: Houston Rocket won 45 games in 12-13 season. 37 What else can we leverage Free-form evidence is sometimes not sufficient. Under many circumstances, we need to resort to other evidence forms to do fact verification. 38 Structured Evidence

Existing work mainly considers free-form language as evidence representation 39 Structured Evidence Existing work mainly considers free-form language as evidence representation Structured data is also ubiquitous form of world knowledge like 40 Structured Evidence Existing work mainly considers free-form language as evidence representation Structured data is also ubiquitous form of world knowledge like Table ID Name Age 1

xxx 12 2 yyy 13 3 zzz 14 41 Structured Evidence Existing work mainly considers free-form language as evidence representation Structured data is also ubiquitous form of world knowledge like Graph, Table, HTML/ XML, etc. Nam

ID ID Name Age 1 xxx 12 2 yyy 13 3 zzz 14

18 Chen e Stude nt Teach er Age 42 Structured Evidence Existing work mainly considers free-form language as evidence representation Structured data is also ubiquitous form of world knowledge like Graph, Table, HTML/ XML, etc. Nam

ID ID Name Age 1 xxx 12 2 yyy 13 3 zzz 14

18 Chen e Stude nt Teach er Age 43 Fact Verification on Semistructured Tables We explore the fact verification problem under semi-structured Wikipedia tables. Ubiquitous in real-world applications Contain both structured and un-structured forms

44 Fact Verification on Semistructured Tables We explore the fact verification problem under semi-structured Wikipedia tables. Ubiquitous in real-world applications Contain both structured and un-structured forms TabFact Dataset 16K open-domain Wikipedia tables 118K human-annotated statements divided into entailed or refuted categories. 45 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E.

California 3 Moss Phillip California 5 Burton Party Result democratic re-elected democratic re-elected George California 8 democratic Paul Miller California 14

California 15 Jerome R. republican Waldie John J. republican Mcfall Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected

Floyd E. Sims (r) 22.4% re-elected John J. Mcfall (d) unopposed 46 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton Party

Result democratic re-elected democratic re-elected George California 8 democratic Paul Miller California 14 California 15 1. John Jerome R. republican Waldie John J.

republican E.Mcfall Moss and Phillip Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% re-elected John J. Mcfall (d) unopposed Burton are both re-elected in the house

of representative election in 1972. 47 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton Party Result democratic

re-elected democratic re-elected George California 8 democratic Paul Miller California 14 California 15 1. John Jerome R. republican Waldie John J. republican E.Mcfall Moss and Phillip

Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% re-elected John J. Mcfall (d) unopposed Burton are both re-elected in the house of representative election in 1972. 48

TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton Party Result democratic re-elected

democratic re-elected George California 8 democratic Paul Miller California Jerome R. republican 14 Waldie California John J. republican 15 2. In the Mcfall election, four out Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2%

lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% re-elected John J. Mcfall (d) unopposed of five incumbents are re-elected. 49 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe

nt John E. California 3 Moss Phillip California 5 Burton Party Result democratic re-elected democratic re-elected George California 8 democratic Paul Miller

California Jerome R. republican 14 Waldie California John J. republican 15 2. In the Mcfall election, four out Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected

Floyd E. Sims (r) 22.4% re-elected John J. Mcfall (d) unopposed of five incumbents are re-elected. 50 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton

Party Result democratic re-elected democratic re-elected George California 8 democratic Paul Miller California 14 California 15 3. John Candidates

John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% Jerome R. republican Waldie John J. republican re-elected E.Mcfall Moss and George Paul Miller John J. Mcfall (d) unopposed

are both re-elected in the house of representative election. 51 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton Party

Result democratic re-elected democratic re-elected George California 8 democratic Paul Miller California 14 California 15 3. John Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1%

Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% Jerome R. republican Waldie John J. republican re-elected E.Mcfall Moss and George Paul Miller John J. Mcfall (d) unopposed are both re-elected in the

house of representative election. 52 TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton Party Result

democratic re-elected democratic re-elected George California 8 democratic Paul Miller California 14 California 15 4. There Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2%

lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1% democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% Jerome R. republican Waldie John J. republican re-elected John J. Mcfall (d) unopposed Mcfall are five candidates in total, two of them are democrats and three of them are republicans. 53

TabFact Examples District United States House of Representatives Elections, 1972 Incumbe nt John E. California 3 Moss Phillip California 5 Burton Party Result democratic re-elected democratic

re-elected George California 8 democratic Paul Miller California 14 California 15 4. There Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% lost Pete Stark (d) 52.9% Lew M. renomination Warden , Jr. (r) 47.1%

democratic hold Jerome R. Waldie (d) 77.6% re-elected Floyd E. Sims (r) 22.4% Jerome R. republican Waldie John J. republican re-elected John J. Mcfall (d) unopposed Mcfall are five candidates in total, two of them are democrats and three of them are republicans. 54 Challenges Mixed Reasoning in Semi-structured Input 55

Challenges Mixed Reasoning in Semi-structured Input Linguistic Reasoning on semantic-level Result re-elected re-elected lost renomination democratic hold re-elected re-elected Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% Pete Stark (d) 52.9% Lew M. Warden , Jr. (r) 47.1% Jerome R. Waldie (d) 77.6% Floyd E. Sims (r) 22.4% John J. Mcfall (d) unopposed 56

Challenges Mixed Reasoning in Semi-structured Input Linguistic Reasoning on semantic-level Result re-elected re-elected lost renomination democratic hold re-elected Candidates John E. Moss (d) 69.9% John Rakus (r) 30.1% Phillip Burton (d) 81.8% Edlo E. Powell (r) 18.2% Pete Stark (d) 52.9% Lew M. Warden , Jr. (r) 47.1% Jerome R. Waldie (d) 77.6% Floyd E. Sims (r) 22.4% JohnParty J. Mcfall (d) unopposed District Incumbent

re-elected Symbolic Reasoning on structure-level California 3 California 5 California 8 California 14 John E. Moss democratic Phillip Burton democratic George Paul Miller democratic

Jerome R. Waldie republican 57 Models Latent Program Analysis Semantic-parsing baseline Synthesize the latent logic forms for the statement and execute the logic form against table to verify. Table BERT NLI baseline Linearize the table as paragraph of sentences, then use large-scale pre-trained language model to verify. 58 Latent Program Analysis Latent Program Analysis There are more democrats than republicans in the election. Table

59 Latent Program Analysis Latent Program Analysis Feature-based Entity Linking Se a rc h There are more democrats than republicans in the election. Table 60 Latent Program Analysis Latent Program Analysis Feature-based Entity Linking

Se a rc h There are more democrats than republicans in the election. String Table incumbe nt incumbe nt democrat ic republica n 61

Latent Program Analysis Latent Program Analysis Feature-based Entity Linking Se a rc h There are more democrats than republicans in the election. String Table Party Party democrat ic republica

n V1=Filter(T, incumbent==democratic)) View Sub V1 62 Latent Program Analysis Latent Program Analysis Feature-based Entity Linking Se a rc h There are more democrats than republicans in the election. String

Table Party republica n V2=Filter(T, incumbent==republican)) View Sub Sub V1 V2 63 Latent Program Analysis Latent Program Analysis Feature-based Entity Linking Se

a rc h There are more democrats than republicans in the election. Table Sub Sub 3=Count(V1) V1 V2 Num Count Party democratic democratic

democratic republican republican 3 64 Latent Program Analysis Latent Program Analysis Feature-based Entity Linking Se a rc h There are more democrats than republicans in the election. Table Sub

2=Count(V2) V2 Num Count Count Party democratic democratic democratic republican republican 3 2 65 Latent Program Analysis Latent Program Analysis

Feature-based Entity Linking Se a rc h There are more democrats than republicans in the election. Table 3 2 Count Count Greater(3, 2) Bool True

Party democratic democratic democratic republican republican Entail ed 66 Latent Program Analysis BFS to search for potential programs Root Func Func Func Func Func

Func Func Func Func 67 Latent Program Analysis Program Re-Ranking 68 Latent Program Analysis Program Re-Ranking 69 Latent Program Analysis Program Re-Ranking 70

Table-BERT Scanning the Table Gam e Date 51 February 3 , 2009 Florida 3-4 52 February 4 , 2009 Buffalo

0-5 53 February 7 , 2010 Montreal Opponent Score Horizontal Scan 5-2 71 Table-BERT Scanning the Table Gam e Date 51

February 3 , 2009 Florida 3-4 52 February 4 , 2009 Buffalo 0-5 53 February 7 , 2010 Montreal

Opponent Score Horizontal Scan 5-2 Row one game is 51, date is February 3, 2009 Row two Game is 52, date is February 4, 2009, 72 Table-BERT Template the table with natural language Word Position [CLS] row one game

is 51 ; 0 1 2 3 4 5 6 dat e 7

is 8 February 3 9 10 73 Table-BERT Concatenate the statement Word Position [CLS] row one game

[SEP] in the game 0 1 2 3 10 11 12 13

Table Statement 74 Table-BERT BERT NLI Framework Label 12-Layer BERT-Base Model Word Position [CLS] row one game [SEP]

in the game 0 1 2 3 10 11 12 13 Table

Statement 75 Table-BERT BERT NLI Framework Suppo rt Refut e Label 12-Layer BERT-Base Model Word Position [CLS] row one

game [SEP] in the game 0 1 2 3 10 11 12

13 Table Statement 76 Experimental Results 77 Experimental Results 78 Experimental Results Latent Program Analysis Pros: robust and explainable Cons: Heavily rely on entity linking, the execution pipeline requires hand-written rules

79 Experimental Results Latent Program Analysis Pros: robust and explainable Cons: Heavily rely on entity linking, the execution pipeline requires hand-written rules Table BERT Pros: simple and general Cons: training unstable, black-box without explainability 80 Conclusion This paper investigates an underexplored fact verification problem under structured data and release a dataset to promote research in this area. 81 Conclusion

This paper investigates an underexplored fact verification problem under structured data and release a dataset to promote research in this area. Both Proposed Symbolic and Neural models have pros and cons. 82 Conclusion This paper investigates an underexplored fact verification problem under structured data and release a dataset to promote research in this area. Both Proposed Symbolic and Neural models have pros and cons. How to design a more powerful model to combine linguistic and symbolic reasoning remains an open problem. 83

Recently Viewed Presentations

  • Embryonic Growth - Sam Houston State University

    Embryonic Growth - Sam Houston State University

    Define key vocabulary. Chart different phases in somitogenesis. ... Embryonic Growth. Blastulation. Rapid increase in blastomere (embryonic cell) number. ... Understand different biological process' in all phases of prenatal development.
  • Filosofía y Sistema Kaizen - ilustrados

    Filosofía y Sistema Kaizen - ilustrados

    Disciplina en el Gemba TPM Layout Desarrollo de nuevos productos. Kanban Just in Time Función de Pérdida de Taguchi Poka Yoke Andon Cero Defectos Actividades de Grupos Pequeños QFD AMFE El mensaje de la estrategia Kaizen es que no debe...
  • Travaux en préparation

    Travaux en préparation

    La conférence d'Alma-Ata (1978) est une révolution en santé publique. Mission des pays non alignés : redéfinir la santé, s'extirper du modèle colonial (hospitalo-centrisme, santé urbaine, programmes contre les grandes maladies endémiques)
  • Principal Certification Network (PCN)

    Principal Certification Network (PCN)

    Cohort model. Individual leadership development plan and action project. 6 hour of pivotal executive leadership coaching including 2 on campus visits. Program Coordinator weekly support and two on-campus 360 assessment feedback visits. Orientation and Leadership Kick-Off. 7 on-site weekend seminars,...
  • ODM Work Stream Summary

    ODM Work Stream Summary

    Work Stream Template: ODM Basel, September 2, 2008 CDISC User Group Work Streams Goals Sharing of expertise and knowledge Sharing of useful tools (e.g. Stylesheets, ODM vendor extensions, SAS macros) Joint mock eSubmission Test new CDISC models and provide joint...
  • The PageRank Citation Ranking: Bring Order to the

    The PageRank Citation Ranking: Bring Order to the

    Convergence Property PR (322 Million Links): 52 iterations PR (161 Million Links): 45 iterations Scaling factor is roughly linear in logn Convergence Property The Web is an expander-like graph Theory of random walk: a random walk on a graph is...
  • The Ten obligatory acts (Shia Islam) - Hodge Hill College

    The Ten obligatory acts (Shia Islam) - Hodge Hill College

    2.2 Sexual Relationships. Islam teaches that sex is an expression of love and means to having children. A sexual relationship fulfils physical, emotional and spiritual but sex is only acceptable within marriage.
  • Integer programming solutions for graph problems In this

    Integer programming solutions for graph problems In this

    It can be shown that A is totally unimodular, that is, every square nonsingular submatrix has determinant 1 or -1. As a result of that property, the integer program for Maximum Assignment Problem has the following feature: All CPF solutions...