Cohesion and Learning in a Tutorial Spoken Dialog System


Results: Aptitude/Treatment



Yüklə 317 Kb.
səhifə9/9
tarix15.06.2023
ölçüsü317 Kb.
#130757
1   2   3   4   5   6   7   8   9
cohesion3

Results: Aptitude/Treatment

  •  
  • Tests
  •  
  • Train: 2003 Data
  • Test: 2005 Data
  • R
  • P-Value
  • R
  • P-Value
  • Grouped by Token (with stop words)
  • All Students
  • 0.380
  • 0.098
  • 0.207
  • 0.239
  • Low Pretest
  • 0.614
  • 0.026
  • 0.448
  • 0.062
  • High Pretest
  • 0.509
  • 0.244
  • 0.014
  • 0.958
  • Grouped by Token (Stop words removed)
  •  
  • All Students
  • 0.431
  • 0.058
  • 0.269
  • 0.124
  • Low Pretest
  • 0.676
  • 0.011
  • 0.481
  • 0.043
  • High Pretest
  • 0.606
  • 0.149
  • 0.132
  • 0.627
  • Grouped by Stem (Stop words removed)
  •  
  • All Students
  • 0.423
  • 0.063
  • 0.261
  • 0.135
  • Low Pretest
  • 0.685
  • 0.010
  • 0.474
  • 0.047
  • High Pretest
  • 0.633
  • 0.127
  • 0.121
  • 0.655

Results: Aptitude/Treatment

  • Test: partial correlation of post-test & cohesion count, controlling for pre-test
  • Cohesion correlated with learning for low pre-test students
  • Not for high pre-test students
  • Little difference between types of measurement
  • Less significant on testing data, “token with stops” level reduced to a trend
  •  
  • Tests
  •  
  • Train: 2003 Data
  • Test: 2005 Data
  • Students
  • R
  • P-Value
  • R
  • P-Value
  • Grouped by Token (with stop words)
  • All Students
  • 0.380
  • 0.098
  • 0.207
  • 0.239
  • Low Pretest
  • 0.614
  • 0.026
  • 0.448
  • 0.062
  • High Pretest
  • 0.509
  • 0.244
  • 0.014
  • 0.958
  • Grouped by Token (Stop words removed)
  •  
  • All Students
  • 0.431
  • 0.058
  • 0.269
  • 0.124
  • Low Pretest
  • 0.676
  • 0.011
  • 0.481
  • 0.043
  • High Pretest
  • 0.606
  • 0.149
  • 0.132
  • 0.627
  • Grouped by Stem (Stop words removed)
  •  
  • All Students
  • 0.423
  • 0.063
  • 0.261
  • 0.135
  • Low Pretest
  • 0.685
  • 0.010
  • 0.474
  • 0.047
  • High Pretest
  • 0.633
  • 0.127
  • 0.121
  • 0.655

Results: Aptitude/Treatment (2003 data)

  • No significant difference
  • between amounts of
  • (turn normalized) cohesion
  • in high and low pre-test
  • groups.
  • Difference in correlation between high and low pre-testers not due to different amounts of cohesion.

Results: Interactivity (2003)

Results: Interactivity (2003)

  • No evidence that cohesion between student productions is correlated with learning (but student utterances are very short with computer tutor)

Discussion

  • Both high and low pre-testers successfully learned from these dialogs
  • Our measure of lexical cohesion seems to reflect only what the low pre-testers do to learn, not correlated with what high pre-testers do.
  • McNamara & Kintsch also found a positive correlation for low pre-testers, but a negative correlation for high pre-testers.

Discussion

  • Our measures are slightly different:
    • McNamara & Kintsch: Manipulated coherence in text
      • Reader does not contribute to coherence
      • Coherence is the extent to which semantic relations are spelled out in the text, rather than inferred by the reader.
      • Low pre-testers probably learned because high coherence text allowed them to make inferences they couldn’t from the low cohesion text.
        • Low pre-testers & low coherence: didn’t know the terms
      • High coherence may allow a greater number of successful inferences for their low pre-testers
    • Our work: Dialog
      • Student does contribute to cohesion
      • Higher cohesion means using more of same terms
      • Speculation: High cohesion may indicate the number of successful inferences our low pre-testers already made.
      • High pre-testers already know the terms, so new inferences are not involved in using them.

Summary

  • We have taken automatically computable measures of cohesion from computational linguistics
    • Applied them to tutorial dialog
    • Found correlations with student learning

Conclusions

  • Simple, automatically computable measures of lexical cohesion correlate with learning
    • But only for students with low pre-test scores, even though low and high pre-testers showed similar amounts of cohesion.
  • Correlation is robust to differences in type of measurement
  • It’s the cohesion between student and tutor that’s important

Future Work

  • Short term:
    • Cohesion may also be related with learning in high pre-testers, but we’re measuring the wrong kind of cohesion
      • Work underway to try “sense” level measures
        • Halliday & Hassan’s “synonym” levels of reiteration
        • “Acceleration” & “speeding up”
      • New issues:
        • Word sense disambiguation (one sense per discourse?)
    • Or measuring it in the wrong places
      • Try finding cohesion at impasses (VanLehn 03)
      • Try finding change in cohesion over time (Pickering & Garrod 04)
    • Is it the dialog, or the essay?
  • Long term:
    • Test by manipulating cohesion in ITSPOKE

Thanks

  • Diane Litman
  • ITSPOKE group

Questions?

Cohesion vs Coherence

  • Cohesive Devices
    • Things that “tie” different parts of a discourse together:
      • Anaphora, repetition, etc…
    • But still may not make sense:
      • John hid Bill’s car keys. He likes spinach. (Jurafsky & Martin 00)
  • Coherence relations
    • Semantic relations between utterances.
      • Result, Explanation, elaboration, etc. (Hobbs 79)

Britton & Gulgoz 91

  • Original text:
  • Air war in the North, 1965
  • By the fall of 1964, Americans in both Saigon and Washington had begun to focus on Hanoi as the source of the continuing problem in the south.
  • Modified text:
  • Air war in North Vietnam, 1965
  • By the beginning of 1965, Americans in both Saigon and Washington had begun to focus on Hanoi, capital of North Vietnam, as the source of the continuing problems in the south.

Yüklə 317 Kb.

Dostları ilə paylaş:
1   2   3   4   5   6   7   8   9




Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur ©azkurs.org 2024
rəhbərliyinə müraciət

gir | qeydiyyatdan keç
    Ana səhifə


yükləyin