CSCI312: Analysis and Processing of Unknown HDFS Applications, Implementation of MapReduce Application: Big Data Management Assignment, UOW, Singapore

Scope
The objectives of Assignment 1 include an analysis and processing of unknown HDFS applications, implementation of MapReduce application without the Reduce phase, describing a MapReduce application, and implementation of MapReduce applications.

Task 1
Discovering the functionality and processing of unknown HDFS applications.

Consider the available source code of the Java application in a file Unknown.java.

Perform the following steps. Each step listed below is worth 1 mark

(1) Read an analyze the contents of a file Unknown.java and discover what functionality is implemented by the unknown Java application. Insert yourexplanations as a comment located in the first few lines of the applications. “Few lines” means, that we expect comprehensive explanations.

Next, insert the comments into a file Unknown.java, that explain step by step functionality of each line(s) such that while reading the comments it would it possible to easily understand how the application is implemented. Please note, that a comment like “An expression on the right-hand side of an assignment statement is computed and the results become a value of a variable on the left-hand side of assignment statement” is an absolutely meaningless comment. The comments must explain the semantics of the Java statements in the context of the functionality of the application.

Assume, that solution will be evaluated by reading the comments one-by one and trying to understand how the application is implemented. Assume, that the comments will be read by someone who does not know how to write the computer programs in Java.

(2) At this point you should know what the functionality of the Java application is. Change a name of file Unknown.java and a name of application to a name solution1.java.

Few text files are zipped in a file FewFiles.zip. Compile a Java application Unknown.java and create a jar file, and use to it to process the files in FewFiles.zip.

Report all your Terminal commands and the output to demonstrate, that you successfully run the application. A simple way to create a report is to use Copy from Terminal Window and then Paste it into a text file and later on print it into a file solution1.pdf.

Deliverables

A file solution1.java with the explanations of the functionality of the unknown application and with the comments explaining the implementation of the application. A file solution1.pdf with the command used to compile and process the application.

Task 2 
Implementation of MapReduce application without the Reduce phase

The application described in a document Filter.java has the functionality equivalent to the functionality of the following SQL statement.

The SELECT key, value
FROM Sequence-of-key-value-pairs
WHERE value > given-value;

The application is a MapReduce application without the Reduce phase.

An objective of this task is to use the Java code included in a file Filter.java to implement a MapReduce application, that has the functionality the following SELECT statement.
The SELECT key, value
FROM Sequence-of-key-value-pairs
WHERE value IN (value-1, value-2, value-3);

Save your solution in a file solution2.java.

When ready, compile, create jar file, and process your application. Display the results created by the application. When finished, Copy and Paste the messages from a Terminal screen into a file solution2.pdf.

Deliverables

A file solution2.java with a source code of the application that implements the functionality of the SELECT statement given above. A file solution2.pdf with a report from compilation, creating jar file, processing, and displaying the results of processing solution2.java

Task 3
Describing MapReduce implementation

Assume, that a file customers.txt has the following contents.
00001 James
00002 Harry
00003 Peter
00004 Jane
… …
The numbers in the first column represent a customer number and the names in the second column represent customer name.

Assume, that a file orders.txt has the following contents.
0000001 00001 34.5
0000002 00001 23.0
0000003 00002 123.0
0000004 00003 12.3
… … …
The numbers in the first column represent order numbers, the numbers in the second column represent customer numbers, and the number in the third column represents a total order value.

The objective of this task is to describe the implementation of an application that finds all customers who have not to submit any order yet.

Assume that both files have been loaded to HDFS. Explain would you implement the Map phase and Reduce phase of the MapReduce application, which lists all customers who have not submitted any orders yet.

Save your explanations in a file solution3.pdf. This task does not require you to write any code in Java. However, comprehensive explanations on how to join the rows are expected. You are allowed to support your explanations with the fragments of pseudocode.

Deliverables

A file solution3.pdf with comprehensive explanations on how to implement an application that finds all customers who have not to submit any order yet

Task 4 
Implementation of MapReduce application

Assume, that a bank records in a text file the withdrawals and deposits of certain amounts of money from the bank accounts. A single row in a file with the withdrawal/deposit records consists of an account number, a date when a withdrawal/deposit occurred, and an amount of money involved. Assume, that the withdrawals are represented by the negative numbers and the deposits are represented by the positive numbers and that each withdrawal/deposit modulo 50 = 0. All values in a single record are always separated with a single blank.
An objective of this task is to implement a MapReduce application that finds the total amount of money deposited by each customer per year. For example, if a sample file with the speed measurements contains the following lines

1234567 12-DEC-2019 200
1234567 15-DEC-2019 50
9876543 25-JUL-2018 150
9876543 12-FEB-2018 -50
9876543 01-JAN-2019 150
1234567 21-OCT-2020 -250
9876543 22-OCT-2019 300

then your application supposed to produce the following outputs.

1234567 2019 250
9876543 2018 150
9876543 2019 450

The order of the lines listed above is up to you.

Perform the following steps.
Implement the application and save its source code in a file solution4.java file. The name of the file with the source code in a local file system is up to you.

Compile the Java source code and create a jar file.

Upload to a local file system a small file for the purpose of future testing. The file must contain the speed measurements and it must have an internal structure the same as it is explained and visualized above. The name of the file and the location of the file in a local file system is up to you.

Use Hadoop to process your application that finds the total amount of money deposited by each customer per year

Use Hadoop to list an input file with the speed measurements and the results produced by your application.

Deliverables

A file solution4.java with a source code of the application, that implements an application described above. A file solution4.pdf with a report from compilation, creating jar file, processing, and displaying the results of processing solution4.java.