Introduction

RealAgents is a real estate company that focuses on selling houses and sells a variety of types of house in one metropolitan area.

Some houses sell slowly and sometimes require lowering the price in order to find a buyer. In order to stay competitive, RealAgents would like to optimize the listing prices of the houses it is trying to sell. They want to do this by predicting the sale price of a house given its characteristics. If they can predict the sale price in advance, they can decrease the time to sale.

Data Description

The dataset contains records of previous houses sold in the area.

Column Name	Criteria
house_id	Nominal. Unique identifier for houses. Missing values not possible.
city	Nominal. The city in which the house is located. One of ‘Silvertown’, ‘Riverford’, ‘Teasdale’ and ‘Poppleton’. Replace missing values with “Unknown”.
sale_price	Discrete. The sale price of the house in whole dollars. Values can be any positive number greater than or equal to zero.Remove missing entries.
sale_date	Discrete. The date of the last sale of the house. Replace missing values with 2023-01-01.
months_listed	Continuous. The number of months the house was listed on the market prior to its last sale, rounded to one decimal place. Replace missing values with mean number of months listed, to one decimal place.
bedrooms	Discrete. The number of bedrooms in the house. Any positive values greater than or equal to zero. Replace missing values with the mean number of bedrooms, rounded to the nearest integer.
house_type	Ordinal. One of “Terraced” (two shared walls), “Semi-detached” (one shared wall), or “Detached” (no shared walls). Replace missing values with the most common house type.
area	Continuous. The area of the house in square meters, rounded to one decimal place. Replace missing values with the mean, to one decimal place.

Task 1

The team at RealAgents knows that the city, that a property is located in, makes a difference to the sale price.

Unfortunately they believe that this isn’t always recorded in the data.

Calculate the number of missing values of the city.

You should use the data in the file “house_sales.csv”.
Your output should be an object missing_city, that contains the number of missing values in this column.

# Reading Data
data <- readr::read_csv("house_sales.csv")

Rows: 1500 Columns: 8
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (3): city, house_type, area
dbl  (4): house_id, sale_price, months_listed, bedrooms
date (1): sale_date

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

# Checking missing values
unique(data$city)

[1] "Silvertown" "Riverford"  "Teasdale"   "Poppleton"  "--"

missing_city <- data |>
  dplyr::filter(city == "--") |>
  nrow()

missing_city

[1] 73

Task 2

Before you fit any models, you will need to make sure the data is clean.

The table below shows what the data should look like.

Create a cleaned version of the dataframe.

You should start with the data in the file “house_sales.csv”.
Your output should be a dataframe named clean_data.
All column names and values should match the table below.

Column Name	Criteria
house_id	Nominal. Unique identifier for houses. Missing values not possible.
city	Nominal. The city in which the house is located. One of ‘Silvertown’, ‘Riverford’, ‘Teasdale’ and ‘Poppleton’ Replace missing values with “Unknown”.
sale_price	Discrete. The sale price of the house in whole dollars. Values can be any positive number greater than or equal to zero.Remove missing entries.
sale_date	Discrete. The date of the last sale of the house. Replace missing values with 2023-01-01.
months_listed	Continuous. The number of months the house was listed on the market prior to its last sale, rounded to one decimal place. Replace missing values with mean number of months listed, to one decimal place.
bedrooms	Discrete. The number of bedrooms in the house. Any positive values greater than or equal to zero. Replace missing values with the mean number of bedrooms, rounded to the nearest integer.
house_type	Ordinal. One of “Terraced”, “Semi-detached”, or “Detached”. Replace missing values with the most common house type.
area	Continuous. The area of the house in square meters, rounded to one decimal place. Replace missing values with the mean, to one decimal place.

clean_data <- data |>
  dplyr::mutate(city = dplyr::case_when(city == "--" ~ "Unknown",
                                        TRUE ~ city),
                months_listed = ifelse(is.na(months_listed), round(mean(months_listed, na.rm = T), 1), months_listed),
                house_type = factor(dplyr::case_when(house_type == "Terr." ~ "Terraced",
                                                     house_type == "Semi" ~ "Semi-detached",
                                                     house_type == "Det." ~ "Detached",
                                                     TRUE ~ house_type)),
                area = as.numeric(stringr::str_remove_all(area, "sq.m."))
  )

head(clean_data)

house_id	city	sale_price	sale_date	months_listed	bedrooms	house_type	area
1217792	Silvertown	55943	2021-09-12	5.4	2	Semi-detached	107.8
1900913	Silvertown	384677	2021-01-17	6.3	5	Detached	498.8
1174927	Riverford	281707	2021-11-10	6.9	6	Detached	542.5
1773666	Silvertown	373251	2020-04-13	6.1	6	Detached	528.4
1258487	Silvertown	328885	2020-09-24	8.7	5	Detached	477.1
1539789	Riverford	47143	2020-09-25	5.1	2	Semi-detached	123.2

The rest of the columns already match the conditions.

Task 3

The team at RealAgents have told you that they have always believed that the number of bedrooms is the biggest driver of house price.

Producing a table showing the difference in the average sale price by number of bedrooms along with the variance to investigate this question for the team.

You should start with the data in the file ‘house_sales.csv’.
Your output should be a data frame named price_by_rooms.
It should include the three columns bedrooms, avg_price, var_price.
Your answers should be rounded to 1 decimal place.

One of the common mistakes is using the clean_data instead of the original one.

price_by_rooms <- data |>
    dplyr::group_by(bedrooms) |>
    dplyr::summarise(avg_price = round(mean(sale_price), 1),
                    var_price = round(var(sale_price), 1))

price_by_rooms

bedrooms	avg_price	var_price
2	67076.4	565289570
3	154665.1	2378289077
4	234704.6	1725211192
5	301515.9	2484327529
6	375741.3	3924432280

Task 4

You need the caret and kernlab packages installed for the following models in this section.

Fit a baseline model to predict the sale price of a house.

Fit your model using the data contained in “train.csv”
Use “validation.csv” to predict new values based on your model. You must return a dataframe named base_result, that includes house_id and price. The price column must be your predicted values.

library(caret)
set.seed(8)

# Reading data
train <- readr::read_csv("train.csv")
test <- readr::read_csv("validation.csv")

# Extracting the sale price column from the train data
price_train <- train$sale_price
train$sale_price = NULL

# One-hot encoding categorical columns
dum1 <- dummyVars(" ~ .", data = train)
train <- predict(dum1, train)

# Extracting the house id column before one-hot encoding the columns
id <- test$house_id

dum2 <- dummyVars(" ~ .", data = test)
test <- predict(dum2, test)

# Training the first model with tuning
glmnet <- train(data.frame(train),
                price_train,
                method = "glmnet",
                tuneLength = 20)

# Results dataframe
base_result <- data.frame(house_id = id,
                          price = predict(glmnet, newdata = data.frame(test)))

The final values used for the model were alpha = 1 and lambda = 305.2986.

Task 5

Fit a comparison model to predict the sale price of a house.

Fit your model using the data contained in “train.csv”
Use “validation.csv” to predict new values based on your model. You must return a dataframe named compare_result, that includes house_id and price. The price column must be your predicted values.

set.seed(8)
rf <- train(data.frame(train),
            price_train,
            method = "ranger",
            tuneLength = 10)

compare_result <- data.frame(house_id = id,
                             price =  predict(rf, newdata = data.frame(test)))

Assessing the models performance

summary(resamples(list(glmnet = glmnet, rf = rf)))


Call:
summary.resamples(object = resamples(list(glmnet = glmnet, rf = rf)))

Models: glmnet, rf 
Number of resamples: 25 

MAE 
            Min.   1st Qu.   Median     Mean  3rd Qu.     Max. NA's
glmnet 15588.925 16392.379 16626.17 16663.06 17033.11 17553.02    0
rf      9297.067  9846.596 10206.44 10123.70 10483.63 10631.30    0

RMSE 
           Min.  1st Qu.   Median     Mean  3rd Qu.     Max. NA's
glmnet 20669.24 21447.26 21736.71 21806.38 22226.55 22922.65    0
rf     13054.65 13891.95 14223.36 14290.14 14734.58 15277.03    0

Rsquared 
            Min.   1st Qu.    Median      Mean   3rd Qu.      Max. NA's
glmnet 0.9604558 0.9648323 0.9657298 0.9656403 0.9663279 0.9696366    0
rf     0.9831788 0.9837440 0.9850914 0.9852629 0.9864090 0.9879923    0

You can always tune your models using the tuneLength argument or manually insert parameters with tuneGrid.

Both models match the requirements for the RMSE with the random forest one being the best model.