```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r, echo=FALSE, message=FALSE}


# import libraries
library(knitr)
library(tidyverse)
library(GGally)
library(gridExtra)
library(RColorBrewer)
library(gplots)
library(corrplot)
library(ggthemes)
library(caret)

set.seed(3456)

# import helper functions
source('../helper/data_munging.R')
source('../helper/visual.R')

# read in data
train <- read_csv('../data/CaseStudy2-data_train.csv')
train$JobLevel <-  as.factor(train$JobLevel)
```

## Data Partitioning

The training dataset will be split into a training set and a test set. The training set `dfTrain` will be used for model cross validation. The test set will be used for final model selection.

```{r}
# split off a test set
trainIndex <- createDataPartition(train$MonthlyIncome , p = .75, 
                                  list = FALSE, 
                                  times = 1)
dfTrain <- train[trainIndex,]
dfTest <-  train[-trainIndex,]
```

## Helper Code

CV Runner - cross validate model on traning parition

```{r}
train.cv <- function(model, method, folds){
  # Set up repeated k-fold cross-validation
  train.control <-trainControl(method = "cv", number = folds)
  # Train the model
  model.cv <-dfTrain %>% train(model,
                   data = .,
                   method = method,
                   trControl = train.control)
  # print model summary
  model.cv
}
```

## Model Construction

From EDA, it appears that monthly income is correlated to `TotalworkingYears`, `Age`, `YearsAtCompany`, `YearsInCurrentRole`, and `YearsWithCurrentManager`.

From the factors:

* `JobLevel` appears to partion `TotalWorkingYears` and `MonthlyIncome` very well.
* `JobRole` appears to partion `MonthlyIncome` very well.

The following models will be used to for regression: `Linear Regression` and `k-Nearest Neighbors`.

### Linear Regression

Based on EDA, the following model will be used for linear regression:

$$ \mu \lbrace MonthlyIncome \rbrace = \beta_0 + \beta_1 (JobLevel) + \beta_2(JobRole) + \beta_3(JobLevel)(TotalWorkingYears) $$

Estimate model on whole dataset and create fit assessement plots.

```{r}
model.linear <- dfTrain %>% lm(MonthlyIncome ~ JobLevel + JobRole + TotalWorkingYears:JobLevel, data = .)

summary(model.linear)

dfTrain  %>% basic.fit.plots(., model.linear)
```

Cross-validate model with linear regression

```{r, cv_pipe}

model.formula <- MonthlyIncome ~ TotalWorkingYears:JobLevel + JobLevel + TotalWorkingYears:JobRole

# Eval model on 5 folds
lin.model.cv <- train.cv(model.formula, 'lm', 5)

# print model summary
lin.model.cv

# print md table with model performance
kable(data.frame('RMSE' = c(lin.model.cv$results$RMSE),
                 'Adj R Square' = c(lin.model.cv$results$Rsquared)))
```

### K Nearest Neighbors Regression

Since KNN is a nonparametric model, a few correlated parameters will be added `Age` and an interation between `JobRol` and `TotalWorkingYears`. This will be compared to a benchmark model that was used for linear regression.

#### KNN with Simpler Model

```{r}
model.formula <- MonthlyIncome ~ TotalWorkingYears:JobLevel + JobLevel + TotalWorkingYears:JobRole

# Eval model on 5 folds
knn.model.cv <- train.cv(model.formula, 'knn', 5)

# print the model
knn.model.cv
```

#### KNN with More Complex Model

```{r}
model.formula <- MonthlyIncome ~ TotalWorkingYears:JobLevel + JobLevel + JobLevel:Age + TotalWorkingYears:JobRole + JobRole

# Eval model on 5 folds
knn.model.cv <- train.cv(model.formula, 'knn', 5)

# print the model
knn.model.cv
```

#### Model Decisions

The more complex model appears to give a better R-Squared after CV. Then model will be used to go farward with KNN.

With the model chosen for KNN, the parameter k will now be tuned.

##### Tune KNN regressor

5-fold CV is used to tune k over possible values of 1-15. Based on the scatter plot of RMSE and R-Squared vs k, the performance appears to flatten bewteen k = 4 and k = 9. k = 7 will be chosen.

```{r}
model.formula <- MonthlyIncome ~ TotalWorkingYears:JobLevel + JobLevel + JobLevel:Age + TotalWorkingYears:JobRole + JobRole

#model.formula <- MonthlyIncome ~ TotalWorkingYears:JobLevel + JobLevel + TotalWorkingYears:JobRole
#c(2,3,5,7,9,11,13,15)
knn.tuningGrid <- expand.grid(k = seq(1:15))

# Set up repeated k-fold cross-validation
train.control <-trainControl(method = "repeatedcv", number = 5)
# Train the model
model.cv <-dfTrain %>% train(model.formula,
                 data = .,
                 method ='knn',
                 trControl = train.control,
                 tuneGrid = knn.tuningGrid)
# print model summary
model.cv
```

Plot k VS RMSE to visualize model tuning

```{r}
k.values <- model.cv$results$k
RMSE <- model.cv$results$RMSE
Rsquared <- model.cv$results$Rsquared

knn.tune <- data.frame(k.values, RMSE, Rsquared)
p1 <- knn.tune %>% 
  ggplot(aes(x = k.values, y = RMSE)) + geom_point() + 
  scale_fill_few(palette = 'Dark') + 
  ggtitle('RMSE vs k for KNN Tuning') +
  xlab('k Parameter')

p2 <- knn.tune %>% 
  ggplot(aes(x = k.values, y = Rsquared)) + geom_point() + 
  scale_fill_few(palette = 'Dark') + 
  ggtitle('R Squared vs k for KNN Tuning') +
  xlab('k Parameter')

grid.arrange(p1,p2, ncol = 2)
```

### Model Selection From Test

With the models chosen and tuned, they will be tested against a validation set to see how well the models generalize to a new set of data `dfTest`. This validation will be used for final model selection.

The table below shows the RMSE for each model for the training set and the test set. Both models appear to generalize reasonably well to the new dataset. Linear regression shows a low absolute difference in RMSE between the training set and the test set.


```{r}
# helper function for calculating model performance
RMSE <- function(model, df){
  predictions <- predict(model, df)
  sqrt(mean((df$MonthlyIncome - predictions)^2))
}

knn.test.RMSE <- RMSE(knn.model.cv, dfTest)
lin.test.RMSE <- RMSE(lin.model.cv, dfTest)

knn.train.RMSE <- RMSE(knn.model.cv, dfTrain)
lin.train.RMSE <- RMSE(lin.model.cv, dfTrain)

kable(data.frame(
  Model = c('Linear Regression', 'KNN'),
  RMSE.Test = c(lin.test.RMSE, knn.test.RMSE),
  RMSE.Train = c(lin.train.RMSE, knn.train.RMSE),
  Abs.Difference = c(abs(lin.test.RMSE - lin.train.RMSE),
                     abs(knn.test.RMSE - knn.train.RMSE))
))

```

## Output Data for Audit

The following code is used to generate the predictions that will be used for model auditing. The output file will be named `Cas2PredictionsMiller Salary.csv` and will be output at the top of the repo.

```{r, message=FALSE}
# load the audit data
AuditData <- read_csv('../data/CaseStudy2CompSetNoSalary_test.csv')
# factor JobLevel
AuditData$JobLevel <- factor(AuditData$JobLevel)

predictions <- as.double(predict(lin.model.cv, AuditData))

write_csv(data.frame(
  ID = AuditData$ID,
  MonthlyIncome = predictions
  ), 
  path = '../../Cas2PredictionsMiller Salary.csv')

```








