Hadoop architecture notes

Below are some reference architectures for Hadoop:

https://www.cloudera.com/documentation/other/reference-architecture.html

http://en.community.dell.com/techcenter/ready_solutions/data_analytics/

Cluster Hosts and Role Assignments

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_host_allocations.html

CDH 5 and Cloudera Manager 5 Install/Upgrade/Requirements and Supported Versions

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#concept_xdm_rgj_j1b

https://www.cloudera.com/documentation/enterprise/latest/PDF/cloudera-installation.pdf

 

High Availability support

https://www.cloudera.com/documentation/enterprise/latest/topics/admin_ha.html

HDFS Directory Structure recommendation (Eric Sammer):

/data : Contains canonical, raw data sets ingested from other systems. Read only to users.
/user/<username> : Home directories / scratch pads for users.
/etl : Contains ETL process queue directories
/tmp : Sticky-bit set scratch for tools and users (no guarantee on longevity).

/data/<dataset name>/<optional partitions>

/etl/<group>/<application>/<process>/{incoming,working,complete,failed}

Reference:

https://www.quora.com/What-is-the-best-directory-structure-to-store-different-types-of-logs-in-HDFS-over-time

Why Hadoop cannot replace traditional RDBMS at present:

https://www.quora.com/How-many-companies-actually-use-both-Oracle-MySQL-RDBMS-cluster-and-Hadoop-cluster-deployments-simultaneously

Active Directory concepts:

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2000/bb742424(v=technet.10)

Miscellaneous topics:

https://www.youtube.com/channel/UCYbBB8FLtf-N0ZW1EYmiw-Q/videos

https://github.com/Esri/gis-tools-for-hadoop

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s