Topic 4 - Measurement

D) 📋 Course Agenda

1. Concepts, measures, and indicators
2. Clarification of concepts
3. Development of indicators
4. Evaluation of indicators

E) 🔍 Main Steps in Quantitative Research

• Theory and concepts
• Hypotheses
• Research design
• Devising measures of concepts
• Research site and sampling
• Data collection
• Data processing/coding
• Data analysis
• Findings/conclusions/write-up
• Operationalisation

F) 💡 What is a Concept?

• Definition: Building blocks of theory; labels for elements of the social world.
• Nature: Unobserved, abstract, and theoretical.
• Examples:
  • Socio-economic status
  • Quality of life
  • Health
  • Personality
  • Political orientation
  • Religiosity
  • Social, cultural, and human capitals
  • Factual demographic concepts (e.g., education, union formation, dissolution)

G) 📏 Measurement

• Definition: Linking theory with quantitative data.
• Process: Concept (theory) → Data in the form of statistical variable(s) (concrete observations coded by numbers).

H) ❓ Why Measuring?

• To delineate fine differences between units of analysis (variance).
• To provide a consistent device for gauging distinctions.
• To produce precise estimates of the degree of relationship between concepts.

A) 📊 Measurement of Concepts

1) Key Points

• To collect quantitative data, concepts must be translated into a measurable form.
• Example concepts: Education and Racial Prejudice.

B) 📈 Statistical Correlations

1) Key Points

• Focus on statistical correlations between variables.
• Understanding how variables relate helps in analyzing data effectively.

C) 🛠️ Indicators Development

1) Key Points

• Indicators are created to tap into concepts that are less directly quantifiable.
• The process of devising indicators is crucial for accurate measurement.

A) 📊 Asking Questions

1) Key Points

• Asking questions is fundamental in social research.
• It helps in gathering qualitative and quantitative data.

B) 👀 Observing Behaviours

1) Key Points

• Observing behaviours provides insights into real-life actions.
• It complements survey data by capturing context.

C) 📺 Examining Media Content

1) Key Points

• Media content analysis reveals societal trends and perceptions.
• It can highlight biases and representation in media.

D) 📈 Using Official Statistics

1) Key Points

• Official statistics offer reliable data for analysis.
• They can be used to validate findings from primary research.

E) 📚 Indicator Development

1) Key Points

• Develop indicators by translating abstract concepts into survey items.
• Specify sub-dimensions of a concept for clarity.

F) 🔍 Descending the Ladder of Abstraction

1) Key Points

• Move from abstract concepts to concrete items.
• Consider how many indicators to use and how to develop them.

G) ❓ Why Use More Than One Indicator?

1) Key Points

• Better classification: Multiple indicators reduce misclassification.
• Better coverage: They capture different dimensions of a concept.
• More variance: Multiple questions allow for finer distinctions.

A) 🛠️ Developing Indicators

1) Key Points

• Avoid ‘re-inventing the wheel’: Utilize established indicators when possible.
• Indicators should be relevant: Ensure they align with research objectives.
• Consider feasibility: Indicators must be measurable and practical to implement.
• Engage stakeholders: Involve relevant parties in the development process for better insights.
• Test indicators: Validate their effectiveness through pilot studies or feedback.

A) 🧠 Definition of a Concept

1. A concept is a mental representation of a category or idea.
2. It helps in organizing and interpreting information.
3. Concepts can vary in complexity and specificity.

B) 📊 Importance of Concepts in Social Research

• Concepts are essential for:
  • Formulating research questions.
  • Developing hypotheses.
  • Analyzing data.
• They provide a framework for understanding social phenomena.

C) 🔄 Evolution of Concepts

1. Concepts can evolve over time based on:

* New **research findings**.

* Changes in societal **norms**.
* Contextual **adaptations**.

1. Older measures may be updated or adjusted to fit different contexts.

A) 📏 Measurement Development

1. New measures can be created when existing ones are unavailable.
2. A less-structured approach, like semi-structured interviews, can be useful in pilot studies.
3. Engaging with informants or experts helps refine question wording.

B) 📊 Measuring Relationship Satisfaction

1. Example: The Relationship Assessment Scale (Hendrick, 1988) measures:

* Meeting **needs** (+)

* General **satisfaction** (+)
* Relative **satisfaction** (+)
* Regret (-)
* Meeting **expectations** (+)
* **Love** (+)
* Number of **problems** (-)

C) 📈 Results of Measurement

1. Measurements yield numerical data (variables) for statistical analysis.
2. Data matrix example:
   | ID | Item1 | Item2 | Item3 | Item4 | Item5 | Item6 |
   |----|-------|-------|-------|-------|-------|-------|
   | 1 | 2 | 3 | 2 | 2 | 1 | 1 |
   | 2 | 4 | 5 | 5 | 4 | 2 | 3 |
   | 3 | 2 | 3 | 4 | 2 | 5 | 4 |

D) 🔗 Correlation and Reliability

1. Good indicators show correlation between variables:

* Correlation ranges from **-1** (perfect negative) to **1** (perfect positive).

1. Average correlation helps assess reliability (e.g., Cronbach’s alpha).

E) 🛠️ Scale Construction

1. Multiple-indicator measurements can be combined into a single scale to enhance validity and reliability.
2. Common procedures for scale construction include:

* **Additive scaling**.

A) 📊 Measurement Models

1) Simplest Methods

• Sum or average scores across multiple items.
• Factor-based scaling assesses if multiple items are related.

2) Latent Variables

• Underlying latent variables cause item responses.
• Example: Relationship satisfaction measured through multiple items.

B) 🧮 Factor Analysis

1) Example Items

2) Assumptions

• The latent variable generates responses to multiple items.
• Measurement error represented as ( \epsilon ).

C) ✅ Evaluating Indicators

1) Key Criteria

• Measurement Validity: Are we measuring what we intend to measure?
• Measurement Reliability: Is the measurement consistent and stable?
• Discrimination: Does the measure capture real and meaningful variance?

A) 📊 Evaluating Indicators

1) Validity & Reliability

• Good measures are valid and reliable.
• They should capture true variance in data.
• Example: A marksman’s target picture illustrates this concept.

2) Early Testing

• Check validity and reliability early by testing questions in a pilot study before the main research.
• Avoid ex-post fixes as they can mislead results.

3) Common Issues

• Concept-indicator mismatch: Indicators may not capture the intended concept.
• Capitalising on chance: Random patterns may appear valid by coincidence.

4) Separate Stages

• Keep stages separate:
  • First, develop and test indicators.
  • Then, collect real data.

A) 📊 Reliability

1) Definition

• Reliability refers to the consistency of a measure.
• A reliable measure yields the same results under consistent conditions.

2) Example

• Researchers want to measure depression.
• They include an item in a questionnaire: “How do you feel today?”
• Question: Is this a reliable measure of depression?

A) 📏 Definition

1) Reliability Overview

• Reliability refers to the consistency of measures in research.

B) 📅 Stability

1) Measure Stability

• A reliable measure should remain stable over time.

C) 🔄 Internal Reliability

1) Indicator Consistency

• Indicators within a measure must be consistent with each other.

D) 👥 Inter-Observer Consistency

1) Observer Agreement

• Measures should show consistency between different observers.

A) 📋 Poorly Worded Questions

1. Interpretation Variability: Respondents may interpret the same question differently at different times, leading to inconsistent answers.

B) ⏳ Memory Effect

1. Response Consistency: Individuals may repeat earlier answers to appear consistent, which can inflate reliability scores.

C) 🔄 Real Change

1. Attitude Shifts: Changes in attitudes or circumstances over time can lead to underestimated reliability.

D) 📊 Internal Reliability

1. Internal Consistency: Reliable measures show high internal consistency, indicated by high inter-item correlation.
2. Cronbach’s Alpha: A summary measure (0–1) for internal consistency; values close to .9 indicate very good internal reliability.

A) 📊 Understanding Variability

1. Question Interpretation: Responses can vary based on personal or contextual meanings (e.g., "Do you support welfare?").
2. Interviewer Effects: Responses may depend on the interviewer's characteristics (e.g., gender, age, ethnicity, tone).

B) 📝 Reducing Errors

1. Coding Errors: Different coders may classify the same answer inconsistently (e.g., job titles in occupational data).
2. Rough Answers: Quick responses on unfamiliar topics can lead to inaccuracies (e.g., estimating neighborhood size).

C) 🧠 Enhancing Recall

1. Recall Problems: Respondents may forget or misremember past events, affecting data accuracy.

A) 📏 Definition of Validity

1) What is Validity?

• Validity assesses whether a measure truly reflects the concept it aims to measure.

B) 🧩 Types of Validity

1) Face Validity

• Reflects the concept.
• Example: A math test appears to measure math ability.

2) Convergent Validity

• Supported by results from other methods.
• Example: Two anxiety scales yield similar results.

3) Concurrent Validity

• Correlates with an established measure taken at the same time.
• Example: A new IQ test aligns with standard IQ tests.

4) Predictive Validity

• Accurately predicts future outcomes or behavior.
• Example: Leaving certificate scores predict college performance.

5) Construct Validity

• Reflects theoretical expectations.
• Example: A new English test measures English language proficiency.

C) 🔍 Assessing Construct Validity

1) Example: English Language Proficiency

• Measure: A new English test (grammar and vocabulary).
• Hypothesis: Non-native speakers' proficiency levels improve after a language class.

2) Differential Group Study

• Groups:
  • Group 1: Native speakers
  • Group 2: Non-native speakers

D) 📊 Measurement of Concepts

E) 🔮 Outlook

• Upcoming topics:
  • Structured interviewing
  • Self-completion questionnaires
  • Designing effective survey questions
• Preparation: Review key readings for Week 5 as provided.

A) 📏 Why Measure?

• Construct validity is essential for ensuring that a test or measurement accurately reflects the concept it intends to measure.
• It helps in determining whether the operational definition of a construct aligns with the theoretical concept.
• Assessing construct validity involves evaluating both convergent and discriminant validity.

1) Convergent Validity

• Measures whether constructs that are expected to be related are, in fact, related.
• High correlations with similar constructs indicate strong convergent validity.

2) Discriminant Validity

• Assesses whether constructs that should not be related are indeed unrelated.
• Low correlations with dissimilar constructs suggest strong discriminant validity.

B) 🧪 Practical Steps for Assessment

1. Define the Construct: Clearly articulate what you intend to measure.
2. Select Measurement Tools: Choose appropriate instruments that align with the construct.
3. Collect Data: Gather data using the selected tools.
4. Analyze Relationships: Use statistical methods to evaluate convergent and discriminant validity.
5. Interpret Results: Determine if the evidence supports the construct validity of your measurement.

A) 📊 Indicators

• Indicators are essential for measuring concepts.
• They provide quantifiable data to assess abstract ideas.

B) 📏 Using Multiple-Indicator Measures

• Multiple indicators enhance measurement accuracy.
• They capture different dimensions of a concept.

C) 📐 Dimensions of Concepts

• Concepts can have various dimensions.
• Understanding these dimensions is crucial for effective measurement.

D) 🔍 Reliability and Validity

1. Reliability

* Refers to the **consistency** of a **measurement**.

* A reliable **measure** yields the same results under consistent conditions.

1. Validity

* Indicates whether a **measurement** accurately reflects the **concept** it intends to measure.

* **Validity** ensures that the findings are meaningful.

E) 💭 Reflections on Reliability and Validity

• Both reliability and validity are critical for credible research.
• Researchers must continually assess these aspects throughout their studies.

F) 🎯 The Main Preoccupations of Quantitative Researchers

• Measurement is a primary concern.
• Researchers focus on establishing causality, generalization, and replication.

G) 🔗 Measurement

• Measurement is the process of quantifying concepts.
• It involves defining and operationalizing variables.

H) 🔄 Causality

• Understanding causality is vital for interpreting research findings.
• Researchers must establish clear cause-and-effect relationships.

I) 🌍 Generalization

• Generalization allows findings to be applied beyond the sample studied.
• It requires careful consideration of the study's context.

J) 🔁 Replication

• Replication tests the reliability of research findings.
• It is essential for validating results across different studies.

K) ❌ The Critique of Quantitative Research

• Quantitative research faces various criticisms.
• Critics argue it may oversimplify complex social phenomena.

L) 🧐 Criticisms of Quantitative Research

• Concerns include lack of depth and context.
• Critics question the applicability of findings to real-world situations.

M) 🤔 Is It Always Like This?

• Not all quantitative research faces the same criticisms.
• Context and methodology play significant roles in the validity of findings.

A) 🔍 Reverse Operationism

• Definition: Reverse operationism refers to the process of evaluating the implications of research findings.
• Importance: It helps in understanding the broader impact of research results on social theories.

B) 📊 Reliability and Validity Testing

1. Reliability: Consistency of a measure across time and different contexts.
2. Validity: The degree to which a tool measures what it claims to measure.
3. Testing Methods:

* **Test-Retest**: Same **test** administered at different times.

* **Inter-Rater**: **Consistency** between different **observers**.

C) 📋 Sampling

• Definition: The process of selecting a subset of individuals from a population to estimate characteristics of the whole population.
• Types of Sampling:
  • Random Sampling: Each member has an equal chance of being selected.
  • Stratified Sampling: Population divided into subgroups, and samples are drawn from each.

D) 📌 Key Points

• Understanding reverse operationism is crucial for interpreting research findings.
• Reliability and validity are fundamental for ensuring quality in quantitative research.
• Effective sampling techniques enhance the representativeness of research results.

E) ❓ Questions for Review

1. What is reverse operationism and why is it significant?
2. How do reliability and validity differ in research?
3. What are the advantages of different sampling methods?

A) 📊 Overview

1. Definition: Quantitative research involves the collection of numerical data.
2. Approach: It adopts a deductive view of the relationship between theory and research.
3. Philosophy: Emphasizes a natural science approach, particularly positivism.
4. Reality: Holds an objectivist conception of social reality.

B) 🔍 Distinction from Qualitative Research

1. Not Just Numbers: Quantitative research is not solely defined by the presence of numbers.
2. Epistemological Position: It has a unique epistemological and ontological stance.

C) 📝 Main Steps in Quantitative Research

• The chapter outlines the main steps involved in conducting quantitative research.

D) ⚖️ Key Concerns

1. Measurement Validity: Addresses concerns regarding the validity of measurements in quantitative studies.
2. Practitioner Issues: Discusses various issues of concern among practitioners in the field.

E) 📚 Chapter Guide

• Focuses on the characteristics of quantitative research as the dominant strategy in social research.

A) 📊 Overview of Quantitative Research

1) Influence and Context

• Quantitative research's influence has decreased since the mid-1970s.
• Qualitative research has gained prominence but quantitative research remains significant.

2) Main Steps of Quantitative Research

• Presented as a linear succession of stages.

3) Importance of Concepts

• Concepts are crucial in quantitative research.
• Measures can be devised for concepts.
• Indicator: A measure created for concepts lacking direct measures.

4) Reliability and Validity

• Procedures exist to check the reliability and validity of measurements.

5) Key Features of Quantitative Research

• Four main preoccupations:
  • Measurement
  • Causality
  • Generalization
  • Replication

6) Criticisms of Quantitative Research

• Common criticisms are discussed throughout the chapter.

A) 📊 Overview

Quantitative research follows a series of steps, often depicted in a linear fashion. While real research may not be as straightforward, these steps provide a useful framework.

B) 🔍 Steps in the Research Process

C) 📝 Key Considerations

• Hypothesis Specification: Common in experimental research but not always required.
• Operationalization: Essential for measuring concepts accurately.
• Data Processing: Involves coding information for quantitative analysis.
• Research Design Impact: Influences validity and the ability to infer causality.

A) 📍 Selecting Research Sites and Sampling Respondents

1) Social Change and Economic Life Initiative (SCELI)

• SCELI involved research in six labour markets: Aberdeen, Coventry, Kirkaldy, Northampton, Rochdale, and Swindon.
• These sites were selected to reflect contrasting economic changes in the 1980s.

2) Main Surveys Conducted

• Work Attitudes/Histories Survey:
  • Random sample of 6,111 individuals interviewed.
  • Focused on work history and attitudes.
• Household and Community Survey:
  • Conducted with about one-third of Work Attitudes respondents.
  • Topics included domestic labour division, leisure activities, and welfare state attitudes.
• Baseline Employers’ Survey:
  • Employers of individuals from the Work Attitudes Survey were interviewed.
  • Covered job gender distribution, new technologies, and trade union relations.

B) 📊 Data Analysis and Interpretation

1) Analyzing Results

• Researchers interpret findings based on data analysis.
• Connections between findings and research motivations are explored.
• Hypotheses are tested for support.

2) Writing Up Research

• Findings must be communicated effectively to gain significance.
• Research should be presented in papers, reports, or articles to enter the public domain.

C) 🔄 Feedback Loop in Research

• Findings contribute to the body of knowledge, creating a feedback loop to earlier research stages.
• This reflects both deductivism and inductivism in quantitative research.

D) 📏 Importance of Measurement

1) Why Measure?

• Measurement delineates fine differences among individuals.
• Provides consistent devices for gauging differences.
• Enables precise estimates of relationships between concepts.

2) Indicators and Operational Definitions

• Indicators are necessary for measuring concepts.
• They can be derived from various sources, including surveys and observations.

E) 📈 Multiple-Indicator Measures

1) Advantages

• Reduces misclassification risks.
• Captures a broader range of the underlying concept.

2) Likert Scale Example

• Used to measure attitudes through a series of statements rated on a scale.
• Example: Commitment to work measured with ten statements.

F) 🔍 Reliability and Validity

1) Reliability

• Consistency of measures is crucial.
• Stability, internal reliability, and inter-observer consistency are key factors.

2) Validity

• Validity assesses whether a measure truly reflects the concept.
• Types include face validity, concurrent validity, predictive validity, and construct validity.

G) 📚 Research Examples

1) Research in Focus 7.2

• Multiple-indicator measure of commitment to work among steelworkers.

2) Research in Focus 7.3

• Measurement of religious beliefs using multiple indicators.

H) 🧪 Developing and Testing Measures

• Importance of testing measures for reliability and validity.
• Examples of scales developed to measure various concepts, including attitudes towards vegetarianism and organizational climate.

A) 📏 Measurement

1) Key Concerns

• Measurement is central to quantitative research.
• Issues of reliability and validity are crucial but not always evident in practice.

B) 🔍 Causality

1) Explanation Focus

• Quantitative researchers aim to explain phenomena, not just describe them.
• They investigate causal relationships between variables (e.g., racial prejudice as a dependent variable influenced by authoritarianism as an independent variable).

2) Experimental vs. Cross-Sectional Designs

• Experimental designs provide clear causal direction.
• Cross-sectional designs create ambiguity about causality since data is collected simultaneously.

C) 🌍 Generalization

1) Extending Findings

• Researchers seek to generalize findings beyond the specific context of their study.
• Creating a representative sample is essential for generalization.

2) Probability Sampling

• Probability sampling aims to eliminate bias through random selection.
• Generalization is limited to the population from which the sample is drawn.

D) 🔄 Replication

1) Importance of Replication

• Replication helps ensure findings are not influenced by researcher biases.
• Quantitative researchers value the ability to replicate studies to confirm results.

2) Challenges of Replication

• Replication is often undervalued and less frequently published.
• Differences in findings can arise from variations in study conditions, complicating interpretation.

A) 📉 General Criticisms

• Quantitative research faces criticism from qualitative research proponents.
• Key areas of critique include:
  • General criticisms of quantitative research as a strategy.
  • Critiques of its epistemological and ontological foundations.
  • Specific method and design criticisms.

B) 🔍 Key Criticisms

C) 📚 Implications of Critiques

• These criticisms reflect concerns from qualitative research emphasizing:
  • Interpretivist epistemology: Focus on individual meaning.
  • Constructionist ontology: View social reality as created by individuals.
• Despite these critiques, quantitative researchers also have their criticisms of qualitative methods.

A) 📊 Ideal Types vs. Actual Practice

1) Gap Between Ideal and Reality

• Research strategies often outline ideal-typical approaches.
• Actual practice may not fully reflect these ideals.
• This gap arises due to:
  • Inability to cover every eventuality in social research.
  • Focus on common features rather than exhaustive accounts.

B) 🔄 Reverse Operationism

1) Conceptualization and Measurement

• Reverse operationism suggests that measurement can be inductive.
• Sometimes, measures lead to conceptualization rather than the other way around.

C) 📏 Reliability and Validity Testing

1) Importance of Good Practices

• Researchers often do not follow recommended practices for reliability and validity.
• Many published studies lack tests for stability and validity of measures.
• Reasons for neglecting these practices include:
  • Time constraints.
  • Cost considerations.

D) 📊 Sampling Issues

1) Non-Probability Sampling

• Good practice favors random sampling.
• Many studies use non-probability samples due to:
  • Difficulty in obtaining probability samples.
  • Resource limitations.
  • Opportunities to study specific groups.

Key Points

• Quantitative research is a linear process but often deviates from the ideal.
• Measurement in quantitative research seeks indicators.
• Reliability and validity are crucial for assessing measure quality.
• Central preoccupations in quantitative research include measurement, causality, generalization, and replication.
• Qualitative researchers criticize quantitative methods for their natural science model.

Questions for Review

• What are the main steps in quantitative research?
• How do measurement and indicators differ?
• Why is reliability important in the measurement process?
• What are the main preoccupations of quantitative researchers?