you will perform a data linkage on two real-world datasets and explore different classification algorithms

INSTRUCTIONS TO CANDIDATES

ANSWER ALL QUESTIONS

Introduction

For this project, you will perform a data linkage on two real-world datasets (Part 1) and explore different classification algorithms (Part 2).

The project is to be coded in Python 3. Eight (8) relevant data sets can be downloaded from LMS and are available on Juptyer Hub:

ˆ datasets for Part 1: Data Linkage

• csv

• csv

• amazon google csv

• amazon csv

• google csv

• amazon google truth csv

ˆ datasets for Part 2: Classification:

• csv

• csv

Please note that your solution program should behave the same way as intended if a similar but different input dataset in an identical format is used. Hard-coding any part of solution will attract a heavy mark-deduction. 

Part 1 - Data Linkage (12 marks)

Amazon and Google both have product databases. They would like to link the same products in order to perform joint market research.

Marking guide for Task 1-A:

Na¨ıve data linkage without blocking (4 marks)

• mark Not reproducible - code does not pass the test script given:  this could    be due to the source file is not named py and the output file task1a.csv is not produced.

The output is hardcoded or the output is completely incorrect.

• mark The output is in the format as At least one of Precision or Recall are below 0.6.

• marks The output is in the format as specified. Both Precision and Recall are above 6.

• marks The output is in the format as specified. Both Precision and Recall are above 8.

• marks The output is in the format as specified. Both Precision and Recall are above 9.

Marking guide for Task 1-B

Blocking for efficient data linkage (4 marks)

• mark Not reproducible - code does not pass the test script given:  this could    be due to the source file is not named py and the output files amazon blocks.csv and google blocks.csv are not produced; or

The output is hardcoded or the output is completely incorrect, or The blocking method is not linear in time complexity.

• mark The output is in the format as specified. PC is below 0.4 and/or RR is below 6.

• marks The output is in the format as specified. PC is above 0.4 and RR is above 0.6.

• marks The output is in the format as specified. PC is above 0.6 and RR is above 0.8.

• marks The output is in the format as specified. PC is above 0.8 and RR is above 0.9.

Marking guide for Task 1-C:

Report on the Data Linkage Project (4 marks)

Scoring method: How the product comparison works

 1 mark  the comparison functions, final scoring function and threshold are appropriate and are clearly and succinctly described, including any preprocessing applied.

0.5 mark  Description of the product comparison is understandable but somewhat unclear. It contains insufficient or excessive unnecessary or incorrect information. Missing some key information on the methods applied.

• mark The description misses key information about the product com- parison method and/or is

Evaluation of the overall performance of the product comparison

 mark Evaluation of the performance is clear and succinct; the potential improvements are sensible and not overly

0.5 marks            Evaluation of the performance lacks sufficient details or contains excessive information to give clarity and/or the potential improvements are not convincing or are overly simplistic.

• mark Evaluation of the performance is substantially incorrect and/or the potential improvements are incorrect or

Blocking method: How the blocking method works

 mark The blocking method is appropriate and is clearly and succinctly described; includes any preprocessing

0.5 mark            Description of the blocking method is understandable but some- what unclear. It contains insufficient or excessive unnecessary information or some incorrect information.

• mark The description misses key information about the blocking method and/or is

Evaluation of the blocking method

• mark Evaluation of the blocking performance is clear and succinct and the overall performance relates well to the measures

The potential improvements are sensible and convincing.

The time-complexity of the blocking method is correct and clearly indicated or can be clearly inferred.

0.5 mark            Evaluation of the blocking performance is somewhat unclear. It lacks sufficient details or contains excessive information to give clarity; or

Overall performance is not well related to the measures used; or The potential improvements are not convincing; or

The time-complexity of the blocking method and its correctness are unclear.

0 mark               The evaluation of the blocking performance is substantially incor- rect, or the potential improvements are incorrect or missing.

Marking guide for Task 2-A:

Comparing Classification Algorithms (3 marks)

Please note that your solution program should behave the same way as intended if a similar but different input dataset in an identical format is used. Hard-coding any part of the solution will attract a heavy mark-deduction.

Failure to pass the test script will result in 0 mark for the task.

 mark Not reproducible - code does not pass the test script given:  this could    be due to the source file is not named py and the output file task2a.csv is not produced.

The classification accuracy of the three classifiers are not printed to stan- dard output.

The output is hardcoded or the output is completely incorrect.

• mark The output file csv is correct but the classification accuracies are substantially incorrect.

• marks The classification accuracies for some of the classifiers are slightly

• marks The output file csv is produced and has correct number of rows/columns and has correct values.

The classification accuracies are correct for all three classifiers.

Marking guide for Task 2-B:

Feature Engineering and Selection(6 marks)

Please note that your solution program should behave the same way as intended if a similar but different input dataset in an identical format is used. Hard-coding any part of the solution will attract a heavy mark-deduction.

Failure to pass the test script will result in 0 mark for the task.

 Overall

 mark Not reproducible - code does not pass the test script given: this could be due to the source file not being named py.

Feature engineering – interaction terms

 mark Evidence of correct implementation of interaction term pairs. Output that support the correct implementation produced and the outputs are

0.5 mark      Evidence of partially correct implementation of interaction term pairs. Output that support the implementation produced and the outputs are partially correct.

0 mark        No output is produced as supporting evidence of implementation of this sub- task (even if the sub-task is implemented in task2b.py)

 Feature engineering – clustering

 2 marks Evidence of correct implementation of feature engineered from clustering. Outputs that support the correct implementation produced and the outputs are correct.

Evidence of implementing a good choice of k for k-means clustering. Outputs that support a good selection of k produced.

1 mark Implementation of feature engineering from clustering and choice of k for k- means clustering are largely correct but contain some minor issues.

Appropriate outputs as evidence of the implementation are produced but choice of k is hard-coded or inappropriate.

• mark No output is produced as supporting evidence of implementation of this sub-  task (even if the sub-task is implemented in py)

Attachments:

Related Questions

. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C

DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma

. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an

Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t

. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual

Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th

. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 1 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of