Hadoop Distributed File System Architecture

What is HDFS  ? 

The Hadoop distributed file system (HDFS) is a distributed, scalable, and portable file-system written in Java for the Hadoop framework. Each node in a Hadoop instance typically has a single namenode; a cluster of datanodes form the HDFS cluster. The situation is typical because each node does not require a datanode to be present. Each datanode serves up blocks of data over the network using a block protocol specific to HDFS. The file system uses the TCP/IP layer for communication. Clients use Remote procedure call (RPC) to communicate between each other.

HDFS stores large files (typically in the range of gigabytes to terabytes) across multiple machines. It achieves reliability by replicating the data across multiple hosts, and hence does not require RAID storage on hosts. With the default replication value, 3, data is stored on three nodes: two on the same rack, and one on a different rack. Data nodes can talk to each other to rebalance data, to move copies around, and to keep the replication of data high. HDFS is not fully POSIX-compliant, because the requirements for a POSIX file-system differ from the target goals for a Hadoop application. The trade-off of not having a fully POSIX-compliant file-system is increased performance for data throughput and support for non-POSIX operations such as Append.

The HDFS file system includes a so-called secondary namenode, which misleads some people into thinking that when the primary namenode goes offline, the secondary namenode takes over. In fact, the secondary namenode regularly connects with the primary namenode and builds snapshots of the primary namenode’ s directory information, which the system then saves to local or remote directories. These check pointed images can be used to restart a failed primary namenode without having to replay the entire journal of file-system actions, then to edit the log to create an up-to-date directory structure. Because the namenode is the single point for storage and management of metadata, it can become a bottleneck for supporting a huge number of files, especially a large number of small files. HDFS Federation, a new addition, aims to tackle this problem to a certain extent by allowing multiple name-spaces served by separate namenodes.

An advantage of using HDFS is data awareness between the job tracker and task tracker. The job tracker schedules map or reduce jobs to task trackers with an awareness of the data location.

LIMITATIONS:-

HDFS was designed for mostly immutable files and may not be suitable for systems requiring concurrent write-operations.

Another limitation of HDFS is that it cannot be mounted directly by an existing operating system. Getting data into and out of the HDFS file system, an action that often needs to be performed before and after executing a job, can be inconvenient.

MapReduce Architecture !

Hello,

I feel better than reading books, Images speaks louder !!
Just have a complete look and then study from Hadoop Definitive Guide.

Hadoop Basic Commands

Hadoop Commands
No	Command	Command Usage	Description
1	cat	hadoop dfs -cat <path>	prints the file contents
2	chgrp	hadoop dfs -chgrp [-R] GROUP URI [URI …]	Change group association of files. With -R, make the change recursively through the directory structure. The user must be the owner of files, or else a super-user.
3	chmod	hadoop dfs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI
4	chown	hadoop dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]
5	copyFromLocal	hadoop dfs -copyFromLocal <localsrc> URI
6	copyToLocal	hadoop dfs -copyToLocal [-ignorecrc] [-crc] URI
7	count	hadoop dfs -count [-q] <paths>
8	cp	hadoop dfs -cp URI [URI …] <dest>
9	du	hadoop dfs -du URI [URI …]	Get size of each file in dir
10	dus	hadoop dfs -dus URI [URI …]	Gets total file size
11	expunge	hadoop dfs -expunge	Empty the Trash
12	get	hadoop dfs -get [-ignorecrc] [-crc] <src> <localdst>
13	getmerge	hadoop dfs -getmerge <src> <localdst> [addnl]	Takes a source directory and a destination file as input and concatenates files in src into the destination local file. Optionally addnl can be set to enable adding a newline character at the end of each file
14	ls	hadoop dfs -ls path
15	lsr	hadoop dfs -lsr <args>	Recursive version of ls. Similar to Unix ls -R
16	mkdir
17	moveFromLocal
18	moveToLocal
19	mv
20	put
21	rm
22	rmr	hadoop dfs -rmr [-skipTrash] URI [URI …]
23	setrep	hadoop dfs -setrep [-R] <path>	Changes the replication factor of a file. -R option is for recursively increasing the replication factor of files within a directory.
24	stat	hadoop dfs -stat URI [URI …]
25	tail	hadoop dfs -tail [-f] URI	Displays last kilobyte of the file to stdout. -f option can be used as in Unix
26	test	hadoop dfs -test -[ezd] URI	e check to see if the file exists. Return 0 if true. z check to see if the file is zero length. Return 0 if true. -d check to see if the path is directory. Return 0 if true
27	text	hadoop dfs -text <src>	Takes a source file and outputs the file in text format
28	touchz	hadoop dfs -touchz URI [URI …]	Create a file of zero length.
29	jar	hadoop jar	The hadoop jar command runs a JAR file.
30	fsck	hadoop fsck	HDFS supports the fsck command to check for various inconsistencies. It it is designed for reporting problems with various files, for example, missing blocks for a file or under-replicated blocks.
31	job	hadoop job -list

Brief on Hadoop

BRIEF ON HADOOP

Ø Apache Hadoop is an open-source software framework for storage and large scale processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level project being built and used by a global community of contributors and users. It is licensed under the Apache License 2.0.

Ø The Apache Hadoop framework is composed of the following modules:

ü Hadoop Common - contains libraries and utilities needed by other Hadoop modules

ü Hadoop Distributed File System (HDFS) - a distributed file-system that stores data on the commodity machines, providing very high aggregate bandwidth across the cluster.

ü Hadoop YARN - a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users' applications.

ü Hadoop MapReduce - a programming model for large scale data processing.

Ø All the modules in Hadoop are designed with a fundamental assumption that hardware failures (of individual machines or racks of machines) are common and thus should be automatically handled in software by the framework. Apache Hadoop's MapReduce and HDFS components originally derived respectively from Google's MapReduce and Google File System (GFS) papers.

Ø Beyond HDFS, YARN and MapReduce, the entire Apache Hadoop “platform” is now commonly considered to consist of a number of related projects as well – Apache Pig, Apache Hive, Apache HBase, and others.

Ø Hadoop consists of the Hadoop Common package, which provides filesystem and OS level abstractions, a MapReduce engine (either MapReduce/MR1 or YARN/MR2) and the Hadoop Distributed File System (HDFS). The Hadoop Common package contains the necessary Java Archive (JAR) files and scripts needed to start Hadoop. The package also provides source code, documentation and a contribution section that includes projects from the Hadoop Community.

Ø For effective scheduling of work, every Hadoop-compatible file system should provide location awareness: the name of the rack (more precisely, of the network switch) where a worker node is. Hadoop applications can use this information to run work on the node where the data is, and, failing that, on the same rack/switch, reducing backbone traffic. HDFS uses this method when replicating data to try to keep different copies of the data on different racks. The goal is to reduce the impact of a rack power outage or switch failure, so that even if these events occur, the data may still be readable.

  

Ø A small Hadoop cluster includes a single master and multiple worker nodes. The master node consists of a JobTracker, TaskTracker, NameNode and DataNode. A slave or worker node acts as both a DataNode and TaskTracker, though it is possible to have data-only worker nodes and compute-only worker nodes. These are normally used only in nonstandard applications. Hadoop requires Java Runtime Environment (JRE) 1.6 or higher. The standard start-up and shutdown scripts require Secure Shell (ssh) to be set up between nodes in the cluster.

  

Ø In a larger cluster, the HDFS is managed through a dedicated NameNode server to host the file system index, and a secondary NameNode that can generate snapshots of the namenode's memory structures, thus preventing file-system corruption and reducing loss of data. Similarly, a standalone JobTracker server can manage job scheduling. In clusters where the Hadoop MapReduce engine is deployed against an alternate file system, the NameNode, secondary NameNode and DataNode architecture of HDFS is replaced by the file-system-specific equivalent