Chap. 1
Introduction to Data Analysis
Data analysis is the process of inspecting, cleansing, transforming, and modeling data to discover useful information, support decision-making, and draw conclusions. It is essential in various fields such as business, science, and social sciences.
- Purpose: Extract insights, identify patterns, test hypotheses, and support decisions.
- Types of data: Quantitative (numerical), qualitative (categorical), structured, unstructured.
Chap. 2
Types of Data Analysis
| Type | Description | Examples |
|---|---|---|
| Descriptive | Summarizes past data to describe what happened | Mean, median, mode, frequency |
| Exploratory | Finds patterns and relationships without prior hypotheses | Scatter plots, correlation |
| Inferential | Makes predictions or generalizations about a population based on a sample | Hypothesis testing, confidence intervals |
| Predictive | Uses historical data to predict future outcomes | Regression, machine learning |
| Prescriptive | Suggests actions based on data analysis | Optimization, decision trees |
Chap. 3
Data Collection and Preparation
1. Data Collection
- Sources: Surveys, experiments, databases, sensors, web scraping.
- Sampling: Random, stratified, cluster sampling to ensure representativeness.
- Data types: Numerical (discrete, continuous), categorical (nominal, ordinal).
2. Data Cleaning
- Handling missing data: deletion, imputation (mean, median, mode).
- Removing duplicates and correcting errors.
- Detecting and treating outliers.
- Ensuring consistency and correct data formats.
3. Data Transformation
- Normalization and standardization to scale data.
- Encoding categorical variables (one-hot, label encoding).
- Aggregation and feature engineering to create new variables.
Chap. 4
Exploratory Data Analysis (EDA)
- Purpose: Understand data distribution, detect anomalies, test assumptions.
- Techniques:
- Summary statistics: mean, median, variance, skewness, kurtosis.
- Visualization: histograms, boxplots, scatter plots, bar charts.
- Correlation analysis: Pearson, Spearman coefficients.
- Tools: Excel, Python (Pandas, Matplotlib, Seaborn), R.
Chap. 5
Statistical Methods in Data Analysis
1. Descriptive Statistics
- Measures of central tendency: mean, median, mode.
- Measures of dispersion: variance, standard deviation, range, interquartile range.
2. Inferential Statistics
- Hypothesis testing: Null hypothesis (H0), alternative hypothesis (H1).
- Tests: t-test, chi-square test, ANOVA, regression analysis.
- Confidence intervals: range where population parameter likely lies.
- p-value: probability of observing data if H0 is true; significance level (α) usually 0.05.
Chap. 6
Data Modeling and Machine Learning
1. Types of Models
| Model Type | Description | Use Cases |
|---|---|---|
| Regression | Predict continuous outcomes | Linear regression, logistic regression |
| Classification | Predict categorical outcomes | Decision trees, SVM, random forests |
| Clustering | Group similar data points | K-means, hierarchical clustering |
| Dimensionality Reduction | Reduce number of variables | PCA, t-SNE |
2. Model Evaluation Metrics
- Regression: R-squared, Mean Squared Error (MSE), Root MSE.
- Classification: Accuracy, precision, recall, F1-score, ROC-AUC.
- Cross-validation: Technique to assess model generalization.
Chap. 7
Data Visualization
- Purpose: Communicate findings clearly and effectively.
- Common charts:
- Bar charts for categorical comparisons.
- Line charts for trends over time.
- Scatter plots for relationships.
- Heatmaps for correlation matrices.
- Best practices: Clear labels, appropriate scales, avoid clutter, use color meaningfully.
Chap. 8
Tools and Software for Data Analysis
| Tool/Software | Description | Strengths |
|---|---|---|
| Excel | Spreadsheet software | Easy to use, good for small data |
| Python | Programming language with libraries (Pandas, NumPy, Scikit-learn) | Flexible, powerful for large data and ML |
| R | Statistical programming language | Advanced statistical analysis |
| SQL | Query language for databases | Efficient data retrieval |
| Tableau / Power BI | Data visualization platforms | Interactive dashboards |
Chap. 9
Ethical Considerations in Data Analysis
- Data privacy: Respect user consent and protect sensitive information.
- Bias and fairness: Avoid biased data and models that discriminate.
- Transparency: Document methods and assumptions.
- Reproducibility: Ensure analysis can be replicated by others.
Key takeaway: Data analysis transforms raw data into actionable insights through systematic collection, cleaning, exploration, modeling, and visualization, enabling informed decision-making across disciplines.
