write a MapReduce program which undertakes an interesting new analysis of the data. Compare the performance of your program using different size clusters.

INSTRUCTIONS TO CANDIDATES

ANSWER ALL QUESTIONS

Big Data Processing Assignment

Aim: design, implement, critically analyze and report on a substantial big data project.

Requirements:

Using a big public AWS dataset, you are asked to write a MapReduce program which undertakes an interesting new analysis of the data. Compare the performance of your program using different size clusters. Note this is an individual assignment.

Deliverables:

• Code, output and logs (15 marks)

• You must use AWS EMR (such as provided by RMIT through this course) for running your MapReduce program (which may be developed on any machine) and accessing one (or more) of the public data sets in S3, such as:

s3://datasets.elasticmapreduce/ngrams/books/ s3://aws-publicdatasets/common-crawl/

You do NOT have to use these two datasets. The full list of datasets can be found here:

https://registry.opendata.aws/

• In choosing a dataset(s) and analysis to undertake, a combination of the two factors may contribute to novelty: (i) difficulty of analysis task, (ii) for a similar difficulty of analysis task, more novelty will be given for using the common-crawl instead of ngrams, and even more novelty will be given for another similar size dataset available on s3.

• Paths should not be hard-coded.

• As you are required to time your tasks, you should carefully log your experiments on different sized

• Your MapReduce program(s) must be well written, using good coding style and including appropriate use of comments. The assignment markers will look at your source code. Coding style will form a small part of the assessment of this

• You should provide instructions with your code on how to compile and execute it. If the assignment marker cannot compile your programs, you risk yielding zero marks for the coding component of your

 

• Depending on the size of output your MapReduce program generates, you may choose or need to submit only samples of the output (in which case you should also provide a complete summary of all the output).

• Depending on your implementation, you may wish to provide additional information about your If so, put this information into a plain text file called readme.txt.

• You can use either Java, Python, Hive or Pig to develop your MapReduce program over AWS EMR.

·    Results of experiments and written report (10 marks)

• You are required to write a short report in the style of a conference paper on your experiments.

• The text of the report should be only TWO A4 pages (not including references and appendices), using the IEEE conference template for Word or LaTex

http://www.ieee.org/conferences_events/conferences/publishing/templates.html

However you may provide additional material including references, and appendices containing output, tables, graphs in additional pages.

• Your report must be a PDF file, called report.pdf, where sNNNNNNN is your student number. Files that do not meet this requirement may not be marked. Your report must be well- written.

• Your report should include sections which

  1. explain how you implemented your map reduce program;

  2. report the experiments undertaken using your programs, report the output and timings;

  3. discuss your results and critically analyze the scalability of your approach as more nodes are added to the cluster (are the results as you expected?);

  4. compare what you have done with the published studies using the same

• Important: Your report will be marked on the quality of your written explanations and analysis. After writing your report you should carefully revise it checking for clarity of expression and quality of

Attachments:

Related Questions

. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C

DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma

. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an

Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t

. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual

Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th

. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 1 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of