Import the csv data into a table in a database on your machine. You can use SQL, workbench or python to achieve this. Show the process you did to achieve

INSTRUCTIONS TO CANDIDATES

ANSWER ALL QUESTIONS

Description

You have been supplied with the datafile all_weather_data.csv which contains data from multiple weather stations. This data has been collected from Mesonet.

The locations that have been collected can be seen below.

The dataset needs to be analyzed using PCA, but it contains missing values and anomalies that must be removed before the analysis can be performed. It is your task to clean the dataset into a form that can be fed to PCA for analysis. After that, you will be doing a PCA analysis on the dataset.

Part 1: Data processing

1. Import the csv data into a table in a database on your machine. You can use SQL, workbench or python to achieve this. Show the process you did to achieve

2. Plot a timeseries plot of selected variables for station FGBT and FMMI. For each station plot heat_index_set_1d together with air_temp_set_1 in one plot and wind_speed_set_1 together with wind_gust_set_1 on another. All in all, provide 4 plot images (2 for each station). Below is an example for the station

3. Create a query that gives the name of all stations that have at least one row of values for heat_index_set_1d

4. Find columns that have more that have at least 90% values (not having more than 5% NULL values). The columns that have less then 10% missing values are to be used for analysis. You can choose to use SQL or Python. Show how you proceed and list the columns that have too many missing

5. Create a view that shows only the columns that have more than 90% non-null values. This view should also remove all rows that have any null values. Provide the SQL code to create this

6. The data we have in its current for is not normalized and therefore contains a lot of redundant data. Create a normalized model. Provide an image of the model and an explanation for which variables broke what normalization

Part 2: PCA analysis

Once you have your cleaned data file, carry out a PCA on the data matrix. All the weather measurements will be your observations, all the weather properties (temperature, dew point etc.) will be your  variables. The station IDs will be your labels.

Carry out the PCA using an existing PCA algorithm in Python, or write your own. With the hoggorm algorithm you can easily plot the scores and loadings https://hoggorm.readthedocs.io/en/latest/

With the scikit-learn you can plot the observations of the different station IDs with different colours.

https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

You can see an example of how to apply the PCA algorithms in the python script in the lecture notes.

Make a figure showing a score plot and a loadings plot with PC1 and PC2.

Based on these plots, discuss shortly (2-5 lines) about the weather at different geographical locations.

Attachments:

Related Questions

. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C

DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma

. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an

Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t

. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual

Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th

. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 1 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of