CEP 930 Notes

_ Topic/project report 9/17 start of class, including research questions and methodology
Inquiry is the search for a mechanistic explanation, with a strong interest in how things work, as well as outcomes. In other industries, outcomes are the primary interest.
Rand says you can ask the same research questions with different methods
Qualitative results may be more immediately applicable, since educators can change environment/context. More quantitative results, like demographic stastics, may be harder to change.
Different methods applied to the same problem have the same effect as multiple light sources giving shape to an actor on stage
What is a proper or improper method? The absolute quality of a method is probably impossible to assess, although you can examine popularity or apparent effectiveness
Defensibility. Can you defend your methodology in the context of your research? [G]
Defensibility depends on a social (semiotic) domain, relies on an objective Other to read our research. These domains are paradigms and paradigms conflict.
Science of philosophy. Keep in mind the telepathy example: if I'm telepathic and you're not, I can't depend on my telepathy as a methodology, so we have to find a common methodology within our common abilities.
Several possible paradigms of studying history (or educational psych):
mechanistic (a determinant world with cause and effect relationships) (you might look for common patterns in history, describe chains of events)
organistic (like an organism, separate organs working in concert) /teleological (trends or memes inform and compel human action) (you might look for ideas and movements, like an Age of Revolution)
contextualistic/tapestry (a network of fibers or nodes with complex codependencies)
formist/ideographic/essentialism (some essential characteristics) (like Plato's forms, or fractals)
The goal is seeking truth, not necessarily resolving the differences between paradigms
Ed psych is different from chemistry because we assume all atoms of the same element are effectively the same, while we prize the individuality of human beings. Nevertheless, we need humans to have some essential characteristics in common in order to generalize results.
Disciplined inquiry allegedly doesn't depend on the eloquence of the researcher. However, a report must be constructed with assumptions about the audience's prior knowledge and their ability to follow your logic (some err in too much explication of logic, some in too little). These are social norms, making this a more subjective activity than alleged. Further, the inculcation of the norms is disciplining. (I admire Carl Sagan for his eloquence.) Clarity is also a factor/norm.
Poor generalization alone can/should disqualify a study.
Programmed instruction offers advantages for generalizability, as does desegregation and gender balance (including gender balance in teachers).
Beware of confounding experimental conditions, like teaching method 1 to all boys and method 2 to all girls. Two variables, two homogenous groups, ouch!
Make your research ready and amenable to research synthesis.
Be perceptive and open to unforeseen confounds, like one group sharing an unforeseen commonality (like all having the same teacher last year)
Random selection is picking subjects. Random assignment is picking treatments for already selected students. Random selection is important for supporting generalizability.
In Yoshida et al. can't build Cornfield-Tukey inferential bridge, i.e. poor generalizability.

9_17_03

CEP930_17_Sep_03
Negative correlation: smoking and education level, up/down, down/up
Cook & Cambell: Quasi Experimental Methods or Q. Experimentation, keeping the good of experimental design while working in real schools and avoiding the problems/imposibilities of random assignment
* At MSU, human subjects research review is managed by UCRIHS
At some institutions, this is managed by an Institutional Review Board (IRB)
Trade offs are part of designing methods
Can use case studies for generalization, by arguing that the specific is a case of a general (e.g. a representative school)
Can use case study to monitor an experimental intervention, like a new program
"Anecdotal" research is a perjorative term
Consider Viagra vs. male contraceptive
Consider SBR vs. case study, marketing, school stories
Consider the research of Rand and others, in which an intervention may increase cognitive flexibility but impair retention. How do you evaluate the intervention?
Consider the dilemmas in biology as parallel situations in which value judgements must be applied rather than strict objective methodology. For example, what if a drug alleviates pain but slows tissue regeneration? How do you evaluate the drug?
Mosteller's Evidence Matters is more popular with the current administration than Scientific Research in Education (which we're reading for this class)
Consider also the erratic nature of intelligence across a population or even an individual (over different sampling moments) (e.g. test exhaustion)
? nomothetic studies
* The Structure of Scientific Revolutions, by Thomas Coone (Rand likes it)
Popper said scientific work should be evaluated on its reproducibility. Coone rejects this
Think about inquiry as creating a mosaic, possibly with multiple patters and alternative explanations
"Life is a research program." -Rand
_ Check out the Santa Fe Institute and their studies of complexity. Is anyone studying games?
Always looking to isolate the unique contribution of a specific influence
Every study has demand characteristics, which can confound inquiry. For example, people want to be helpful, look smart, and other "norms of self presentation."
To describe complex behaviors you have to deconstruct them. So expressing tacit knowledge can distort the nature of the knowledge in the process of expression.
Repeated measures can be valuable, but their impossible with irreversible behaviors (e.g. students can't unlearn a strategy). Long-term change in the population can also confound (e.g. student development over the course of a year).

CEP930_Notes_24_Sep_03

Consider the importance of research literacy for policy makers and other non-researchers, to foreclose on erroneous generalization

Rand's “conspiracy of convenience” leads to over-simplification and erroneous generalization. “Unfortunately the world fails to cooperate in that conspiracy.”

Consider the development of communities of practice, including the recent emergence of Special Interest Groups in DiGRA

Compare successive research projects to jazz improvisation (compare also to jumping from rock to rock on the seashore): where you are determines where you can go

Try triangulation to deal with simplification (e.g. “well, that group of students was just inferior to last year's”)

Consider the confound of subject interest or subject proficiency, e.g. Good in math so like math
e.g. Violent math game vs. nonviolent social studies game. Two dependent variables, so confounded
A confound is like an ambiguity, a different interpretation of the results of your research.
(for instance, do the boys like the math better then the social studies, or the violence more then the non-violence?)

The group interview is a minor intervention (as an artificial event), making JD's case study method a little bit of a hybrid

Avoid confounding content and medium (e.g. Software uses examples from youth culture, non-software uses examples from adult culture)

Consider that various modifications of the overall intervention can allow for some controls. So all students are getting the same intervention, but at any given time a specific group isn't getting one component
This connects with the possibility that
A + B = improvement
but
A = no improvement
B = no improvement

Last group presented on evaluating a Head Start program that helps current instructors earn Associates or Bachelors
One perhaps-overlooked confound is that they went to different colleges
Only about 1/4 part of formal program, although seeking education on their own
Could evaluate improvement of Head Start program (e.g. student performance), or evaluate improvement of teachers, or parent satisfaction

If multiple measures, put the most important first to avoid possibility of contamination by other measures

Test statistic = Variability across group means / Variability across individuals within groups
(= V1/V2, where both V1 and V2 are affected by individual differences, while V1 is also influenced by differences in treatment effectiveness)

Multiple methods are good
There is no single method
Selection of appropriate methods is an act of judgement
Disasters have come about through good execution of methodology, but wrong methods (e.g. Nazi experiments)
Just because we can't be perfect researchers, doesn't mean we shouldn't try (e.g. surgery in the sewer)

10_15_03

Be Careful What You Wish For - You May Get It: Educational Research (James W. Pellegrino and Susan R. Goldman)
* bad to legislate scientific method
* peer review is needed but hard
* reconsider researcher training (we said the same thing)
* connect with practitioners (ditto)

Educational Research: The Hardest Science of All (David C. Berliner)
! * don't confuse the methods of science with the goals of science
* hard and soft science: a flawed dichotomy

Culture, Rigor, and Science in Educational Research (Frederick Erickson and Kris Gutierrez)

Consider: the NRC report, as a book, is a piece of culture (esp. b/c written by committee)
the article is explicitly a piece of culture

Consider that a critical ingredient for peer review is journal editing and review, and conferences. These things cost money, but don't directly contribute to test scores

Post-modernism isn't a rejection of rationality. Rather, it situates rationality in context of assumptions

I think it's a matter of emphasis. People are well-aware of the various elements and issues, but it's the combination of them in which we find arguments.

Consider whether the nature of a subject should determine the nature of the language? Should education use different semiotics than grave digging or prescription drugs? Consider jargon

Consider how the different structures of K-12 and higher ed make it difficult to build connections

Consider deliberate community arising from opportunities or just community for the sake of community

The federal leadership should be more connected to local research, by more ways than money (but local only assimilates the expectations of federeal if there's money involved, and then only as far the money influences)

!! Consider that the majority of politicians have legal or business backgrounds (and get into office through competition), so dialectic and competition make sense to them. It's easier to have dialectic with quant. research. Grant model is competitive
Policy makers want to know "how do should I vote?"
Higher ed enjoys and embraces plurality
Perhaps we're too lax in criticizing others because we fear criticism
Part of how we got here is wacky or wasteful research

! There's a culture clash going on. One sign of this clash is the battle over language (e.g. scientific as a synonym for random, and vice versa). We're ill-equiped to deal with that, and we can invest a lot of energy in the wrong areas. Many of the people with power would rather we debate endlessly among ourselves

As ivory tower intellectuals, discussion of money seems like a necessary evil, something dirty. Sounds petty to plead we need more money, but funding is exactly how our government displays its priorities

10_22_03

beware of creeping zealotry
research method comes first, and determines the appropriate method

not everything can be done in a randomized experiment
but there should be more randomized experiments
such experiments yield effect sizes, which are useful

beware of dismissing some disciplines or perspectives as un-scientific
is philosophy science? is history science?

Rand says that the idea of cooperative movement toward consensus is hypothical; historically, different factions have competed for power/influence

Authors say the research community should focus on what unites
Rand says that we should focus on what unites and what divides
Metaphor of a face: you can talk about the abstract whole without constraining discussion of the elemental components

Consider how science uses a specific persuasive rhetoric (e.g. introduction, methods, results, discussion)(e.g. homogeneity, reproducibility, objectivity)
Consider how education relies on several rhetorics. For example, the autobiographical insights of practitioners wouldn't fit in the traditional scientific rhetoric (e.g. not objective)

In order to use a probe, you have to think through the probe. Example: using a pen to poke around blindly. You feel in your palm, but you think about what's happening to the pen. Another example: as an expert driver, you can't think about every step.

Kym says education research is young. Newton benefited from a heritage of physics research. Behaviorism has some laws, which are simple. Newton's laws are simple. Newton doesn't directly help you build a better anti-lock braking system, but it's a good start. We're just getting started with education research.

Rand says education is an ill-structured domain (in the perspective of cog. flex.)

Consider Newton's Laws with a hamster ball. Compare to beh. Law of Effect with human beings. Laws require specific antecedents and knowing those antecedents. Vygotsky said we can only understand knowledge through its development. Laws describe propensities or inertias. In beh., stronger causalities (e.g. perversions) can supersede propensities.

Rand says everybody is right (in part) and everybody is wrong (in thinking they have all the answers)

Rand says implicit is better than codified, because codifying inhibits
Education needs to grow

Idealogy always enters in theorizing
Rand says if you wear a lens long enough you forget you're wearing a lens

10_29_03

In qualitative research, in the transition from inquiry to publication, much is excluded.

A qualitative study usually entails an iteration of inductive and deductive reasoning. Hypotheses are brought in, and hypotheses are inspired in context.

Ethnographers try to tell a good story: a revealing story. The editing process can be similar to editing fiction or film.

Technology may enable researchers to share their raw data. This could change the nature of peer review.

Contrast "writing like crazy" to create a rich description with interpretative research to reveal meaning

See Qualitative Methods of Research on Teaching (Ch 5, Frederick Erickson, Handbook of Research on Teaching, 3rd. ed, ed. by M.C. Whittrock, 1986)

Trying to define ethnography might be like trying to define "game"
Meaning is in use, and resemblance to a family of concepts

When do corroborators become collaborators? How important is collaborating with a native?

There is some hubris in asserting that you know what it's like to walk in someone else's shoes. Beware your assumptions

Awareness of cultural differences can help mitigate the skew of perceptions across cultural boundaries

11_5_03

Rand's point of departure is Qualitative Methods of Research on Teaching (Ch 5, Frederick Erickson, Handbook of Research on Teaching, 3rd. ed, ed. by M.C. Whittrock, 1986)

General and abstract ideas are the sources of the greatest errors of mankind. - Rousseau

education and classrooms are ill-structured
ontologies are limited
some resemblances, but a lot of variation

summarizing, generalization, and abstraction are dangerous
look more carefully at individual cases, look more deeply

What is general nature? Is there such a thing? What is general knowledge? Is there such a thing? Strictly speaking, all knowledge is particular? - William Blake

Erickson distinguishes between kinds of qual. research, in particular interpretive
you have to draw inferences. you can't just describe

Erickson says all qual. researchers are interested in human meaning and social life
Intepretivists are interested in local meanings (specific person in specific situation)

people take meaning from events. need to be inferential, not merely descriptive, about events

intepretive researchers can be seen as purely inductive, entering without assumptions, to absorb
actually, Erickson says, there's a mix of inductive and deductive. Researchers enter with some assumptions and beliefs

When do you want to do interpretive research?
1. want to know more about the specific structure of something, rather than general character
2. want to know more about the meaning perspectives of the particular actors in particular contexts
3. when an experimental design isn't possible (e.g., can't experimentally determine why teachers leave teaching). Instead, you might look for "natural experiments" in situations with parallel characteristics. Find what's different
4. identification of specific causal linkages that weren't revealed by experiments

Why do you need interpretive research? The invisibility of everday life. We don't see what's going on around us.

"The fish would be the last creature to discover water." - (not Erickson or Rand)

the inter. researcher is always saying look at context
different contexts, different meanings

positivist research believes that history repeats itself
interpretive researchers are more cautious

a world of lived experience (a "life world")
a classroom, or any social situation, is a collective balancing act

consider if you're looking at time on task, and a student is looking out the window. How can you tell if the student is ruminating or daydreaming?

perhaps classrooms only seem ill-structured because we bring the interpretive perspective and expect to find them ill-structured
perhaps classrooms can be structured with intent

Erickson and the interpretive perspective seize on a larger context, within which quant. methods or strongly-structured teaching can create pockets of well-structured phenomena. but these pockets aren't necessarily representative of the larger context, and making inferences from those pockets is risky.

quant. research is reductive, but this isn't a critical flaw, b/c of what we gain in exchange

qual. research is also reductive, b/c you tell a specific story from a mass of observations and other evidence

stereoscopic - your lenses, the lenses of the people you're observing
to see the meaning of events for the people

interpret. - data collection is progressive problem solving
partially regulated by intuition

intuition plays an important role: enough sources, intersubjectivity, seeing what I see

"bring research questions and data collection into a consistent relationship, albeit an evolving one"

Things can wrong
1. insufficient evidence
2. inadequate variety in kinds of evidence
3. faulty interpretation of evidence (e.g., researcher was not participating long enough or intensively enough, researcher was mislead)
4. inadequate disconfirming evidence (you must search for disconfirming evidence to be sure it's not there)

think Margaret Mead and the natives finding it amusing to surprise her with accounts of sexual promiscuity

progressive problem solving requires successive sampling
starting with a wide lens, then looking more narrowly
think Open Options, open-ended questions, then modifying multiple-choice questions

beware of premature typification (i.e., jumping to conclusions)
e.g., doctors making premature diagnoses, then ignoring disconfirming evidence (b/c of blinders)

look for key linkages (i.e., things that unite the case)
(e.g., a teacher treating poor readers who demonstrate effort just like good readers)
find the string to pull, and pull it

CEP930_11_19_03

Random assignment is different than random selection
Any systematic bias can skew results

Causal inference depends on random assignment of subjects to experimental conditions

Random selection of subjects improves generalizability (i.e., external validity)
e.g., experimental psychology built on experiments with college sophomores

Controlling for all but one variable helps eliminate alternative interpretations

? Yoshida, Fernandez, and Stigler, hoping for differences b/n American and Japanese pre-to-post, settled for differences in mem. recognition

Number cases vs. story cases. Different people will be impressed by different kinds of cases
Another merit of pluarlist methods

CEP930_12_3_03

xv: Rand doesn't see a significant break from past general questions. Rather, the same old questions are now being applied in more vivid contexts (i.e., domain specific)

The different question is whether we should be trying to describe general phenomena with the low resolution for generalizability, or to describe specific phenomena with the high resolution for deep understanding.

At different points in the book, many of the authors lament the complexity of educational settings. They portray a dilemma between simplifying for the sake of rigor, or sacrificing rigor to capture complexity. But, "You can look at complex things systematically." -Rand

408: the need to work toward consensus on the purpose of education, against which the value of all research should be measured

51: you need to do rigorous research that doesn't reduce complexity so that results have practical utility

169: education research not in a period of normal science. Paradigms are shifting

You can't abstract away from a specific case in an ill-structured domain. All areas of education research are ill-structured domains.

need a balance point with order and chaos
(e.g., polymer chemistry, social networks, economics)

Test of a good research question: From this question, I have confidence that I will be able to recognize an answer as an answer (to this question).

time on task vs. socializing or otherwise
choosing it over another activity
self-reported enjoyment
pre- to post-test gains

We will study a population of third grade students, videotaping a 45 minute computer lab visit. We will present them with software X, Y, and Z. When multiple researchers review the tape, does one gender exhibit greater time on task when using a specific program?

We will study a population of third grade students, videotaping a 45 minute computer lab visit. We will present them with software A, which is allegedly designed to appeal to one gender. When multiple researchers review the tape, does one gender exhibit greater time on task when using this program?

CEP 930 Survey

Question: Are some kinds of educational software more appealing to elementary-age boys than girls?

Survey: Kym, Kyong Hee
Historic: Leigh, Derek
Case study: JD, Bridge, Adam
Experimental: Alex, Aroutis

Give a survey to a population or sample population of elementary school students. If sample then random, except roughly gender balance.

[B] Ask about the kinds of educational software they are using or have used, and have them rate the software.

[C] Describe several popular educational programs, and ask if they've used them, or if they'd want to use them. If they have used them, ask how much they liked them.

[D] Describe several hypothetical educational programs, with each illustrating some of the elements that might appeal more to boys or to girls. Ask if they'd want to use them.

Administer the survey using a Web interface without the option to go back, to be sure that [C] doesn't influence [B].

Group_10_1_03

CEP930_Notes_1_Oct_03

Note the numerous citations to other Stigler publications, indicating a larger agenda of research

Potential confounding: Japanese students may be more familar with the Japanese style of instruction

Remember that generalizing is difficult because the study populations aren't representative of general populations

It's not clear that classroom instruction will be primarily responsible for learning mathematics. e.g. Japanese parents may be more helpful to their children with their homework.

Rand suggests that they might have been smarter to start with the American lesson study they describe in the conclusion p616

As Shavelson & Towne emphasize, education is very heterogenous, making generalization more challenging than in other fields

Rand wonders why they don't try to correlate the ability to distinguish relevant from irrelevant with academic ability

This seems like a circuitous approach to analyzing learning, and schemas for learning

Accurate recognition performance may not be desirable, or as desirable as students assimilating the meaning of the lesson

FYI: regression to the mean. All tests are subject to some measurement error. If you retest, the high and low extreme students will rescore closer to the mean.

Consider that the slope of the Japanese line p615 with the performance scores p614. If the ability to distinguish relevance is critical, why doesn't the poor distinguishing skills of the Americans not hurt their post-test performance relative to the Japanese?

Ceiling effect is like a floor effect. Consider a test repeatedly given to college students and 4th graders. If it's advanced physics, the college students may get progressively better scores, while the 4th graders have consistently low scores. The 4th graders are at the floor. If it's basic multiplication, the 4th graders may get progressively better scores, while the college students have consistently high scores. The college students are at the ceiling. You can't talk about the effects of practice if the population is at a ceiling or floor.

A statistical interaction requires us to talk about two variables together. For example, if two populations vary in parallel across experimental conditions, you can compare the populations with each other without regard to experimental conditions, and you can compare the experimental conditions without regard to population. But without that parallel variance, you have to talk about both together. [see hand-drawn graphs]

The postest could interfere with the recognition test, since the postest was administered between the instruction and the recognition test

Group_10_8_03

book is for a more general audience
tacit knowledge is made explicit, broken down
book goes back to basics
book has lots of examples
describes ed research as a general activity

article is for the education researcher audience, mainly higher ed researchers
article starts with NCLB
more focused on community
article says unifying theme of book is community and culture. we did not get that from the report, feels more like process than community

CEP 930 Implications for Survey

Be Careful What You Wish For - You May Get It: Educational Research (James W. Pellegrino and Susan R. Goldman)
(1) Book: education research needs to improved in all these ways. P and G: that costs money, and you haven't given enough money.
* The survey is low-cost and reusable.
(2) Book: focus on researchers. P and G: use practitioners. Change how grad students are trained, make it multidisciplinary.
* We should collaborate with one or more experts outside education on best practices in survey design and administration. We should view games and other software in the larger context of a socially-integrated medium (e.g. entertainment, art).

Educational Research: The Hardest Science of All
David C. Berliner
(1) Book: look to other sciences for how to research education. Berliner: education is different and hard.
* We should recognize that our survey data has limited prescriptive value for creating games and other software.
(2) Book: education research should converge on "best" methods. Berliner: value and maintain diversity of methods
* Again, outside collaboration is desirable, including one or more classroom teachers who use games and other software.

Culture, Rigor, and Science in Educational Research
Frederick Erickson and Kris Gutierrez
(1) Book: doesn't discuss long-term implications. Erickson and Gutierrez: that's bad, see thalidomide.
* This isn't very applicable. However, we should consider the use of a survey for longitudinal study. For example, we might track how students' attitudes about games and other software changes by grade, or due to exposure.
(2) Book: have faith in SBR, which will be stepwise. E and G: it's a swamp, like Rand's jumping from stone to stone.
* We should administer a relatively short, directed survey, to probe our immediate next steps. We should be careful about what assumptions we build into the survey.

"Science" Rejects Postmodernism
Elizabeth Adams St. Pierre
(1) Book: science tracks truth. St. Pierre: be open to diverse methodologies and epistemologies.
* And again, outside collaboration is desirable. We should be wary of how far we generalize or otherwise extend our conclusions from the survey.
(2) Book: qualitative and quantitative can be synthesized into one. St. Pierre: don't do that. Question your assumptions. Avoid scientism.
* While some of the survey data can be quantitatively analyzed, we should be careful of losing the qualitative details. For example, only looking at means can obscure data clumping at the extremes.

CEP930_Group_Qualitative_Study

Are some kinds of educational software more appealing to elementary-age boys than girls? If so, why?

1. The specific structure of occurrences rather than their general character and overall distribution.
2. The meaning-perspectives of the particular actors in the particular events.

- A public elementary school in an urban area with a significant minority population
- 4th grade class of 25 students
- Using a computer lab for at least 45 minutes at least every two weeks
- Investigating the full academic year, from September to June
- Studying each student, with the ultimate goal of writing revealing student profiles and a class profile

- Gather demographics, test scores, transcripts for each student
- We need to become residents of the school

- Two researchers working as teacher aides in the classroom (one male, one female)

- Upfront interview with teacher
- His attitudes toward software
- Looking at influence on teacher
- Looking at influence on students (esp. gender bias)

- Ask them about access and exposure to technology outside of school
- Ask them about attitudes toward technology
- Ask them to predict their engagement and success with technology

- Ask about the kinds of educational software they are using or have used, and have them rate the software.
- Describe several popular educational programs, and ask if they've used them, or if they'd want to use them. If they have used them, ask how much they liked them.
- Describe several hypothetical educational programs, with each illustrating some of the elements that might appeal more to boys or to girls. Ask if they'd want to use them.

- Whole class
- Small group girls only
- Small group boys only
- Small group mixed

- Aids and researchers should explore community
- Attend extracurricular activities

---+ 3. The location of naturally occurring points of contrast that can be observed as natural experiments.

- Contrast interviews with different-gendered researchers
- Contrast responses in individual and group interviews
- Some kids will have more access and exposure to technology outside of school
- Contrast teacher declarations and predictions to researchers with their interactions with students
- Students performance in non-technology activities
- Monitor how students make use of flexible time (e.g., spending extra time on computer when could be doing something else)

---+ 4. The identification of specific causal linkages that were not identified by experimental methods, and the development of new theories.

- Can't change gender of teacher
- Can't control teacher bias
- Wide variability of computers and software across schools

- Video tape interviews and classes, watch much later
- Using video tape to elicit multiple perspectives from uninvested observers without telling them our hypothesis
- Using video tape to elicit perspective from the teacher
- Analyze a few video taped sessions, map teacher movement, tally interactions

- Looking at culture (creation & modification of culture)
- Collecting examples of classroom work

- What is the garbage can model of decision making? (p.59, Lagemann & Shulman)
- How do we get graduate students involved in research as soon as possible? (Pelegrino & Goldman, Lagemann & Shulman Readings)
- The issues and problems in education require multiple perspectives. How do we welcome and unite these perspectives?
- Insight and reform needs colloborative and cummulative research. How do prevent research from fragmenting into studies that can't be integrated?
- How do we apply more rigor to counter poor perceptions of education research?

Created by Kym Buchanan | http://KymBuchanan.org | This work is licensed under a Creative Commons License.