Install Subversion On Solaris 10 Ntp
How to Set Up a Hadoop Cluster Using Oracle Solaris Zonesby Orgad Kimchi. How to combine an Apache Hadoop cluster with Oracle Solaris Zones and the new network virtualization capabilities of Oracle Solaris 1. Hadoop cluster on a single system. Published January 2. February 2. 01. 3)About Hadoop and Oracle Solaris Zones.
Install the Chef Server. A connection to Network Time Protocol. Solaris 10 "system" Apache rather than install yet-another. NTP server and client on Solaris 10 and. Wicked Angel Julia London Download Itunes.
Solaris 10 Install
Visit our News Page to read the latest goings on at UNIX Packages and. Ntp 4.2.8p10 Added Ntp packages version 4.2.8p10 for Solaris 10. Compiling Rancid on an x86 Solaris 10 platform. Solaris (2) Spam (5). CentOS / Red Hat Configure an NTP Client And Serve. Solaris has long included.
Download and Install Hadoop. Configure the Network Time Protocol. Create the Virtual Network Interfaces. Create the Name. Node Zones. Set Up the Data. Node Zones. Set Up SSHVerify the SSH Setup.
Verify Name Resolution. Format the HDFS File System from the Name. Node. Start the Hadoop Cluster. Run a Map. Reduce Job. Conclusion. See Also.
About the Author. This article starts with a brief overview of Hadoop and follows with an example of setting up a Hadoop cluster with a Name. Node, a secondary Name. Node, and three Data.
Nodes. For more information about Hadoop and HDFS see http: //hadoop. The Hadoop cluster building blocks are as follows: Name. Node: The centerpiece of HDFS, which stores file system metadata, directs the slave Data. Node daemons to perform the low- level I/O tasks, and also runs the Job. Tracker process. Secondary Name. Node: Performs internal checks of the Name.
Node transaction log. Data. Nodes: Nodes that store the data in the HDFS file system, which are also known as slaves and run the Task.
Tracker process. In the example presented in this article, all the Hadoop cluster building blocks will be installed using the Oracle Solaris Zones, ZFS, and network virtualization technologies. Figure 1 shows the architecture: Figure 1. Architecture. Download and Install Hadoop. To get a Hadoop distribution, download a recent stable release from one of the Apache download mirrors.
Handy scripts. Check domains. To setup a Solaris 10 server as an ntp client. Directory setup wizard to install a DNS server.
For this article, I used the . Hadoop Configuration Files. Filename. Descriptionhadoop- env. Specifies environment variable settings used by Hadoop. Specifies parameters relevant to all Hadoop daemons and clients. Specifies parameters used by the HDFS daemons and clients.
Specifies parameters used by the Map. Reduce daemons and clients. Contains a list of machines that run the Secondary Name. Node. slaves. Contains a list of machine names that run the Data. Node and Task. Tracker pair of daemons. To learn more about how the Hadoop framework is controlled by these configuration files, see http: //hadoop. Configuration. html.
Run the following command to change to the conf directory. In this example, the global zone is configured as an NTP server. Note: It is best to select an NTP server that can be a dedicated time synchronization source so that other services are not negatively affected if the machine is brought down for planned maintenance. The following example shows how to configure an NTP server. Edit the NTP server configuration file, as shown in Listing 2. NTP Server Configuration File. Enable the NTP server service.
For a Hadoop cluster, this means that the Data. Nodes can replicate the HDFS blocks much faster. For more information about network virtualization benchmarks, see .
Creating the name- node Zone. Create the sec- name- node zone, as shown in Listing 4. Creating the sec- name- node Zone. Set Up the Data. Node Zones. In this step, we can leverage the integration between Oracle Solaris Zones virtualization technology and the ZFS file system that is built into Oracle Solaris. Hadoop best practice is to use a separate hard disk for each Data. Node. Therefore, every Data.
Node zone will have its own hard disk in order to provide better I/O distribution, as shown in Figure 2. Figure 2. Separate Disk for Each Data. Barclays Capital Software Developer Interview. Node. Table 2 shows a summary of the Hadoop zones configuration we will create: Table 2.
Zone Summary. Function. Zone Name. ZFS Mount Point. VNIC Name. IP Address.
Name. Nodename- node/zones/name- nodename. Setting the zonecfg zonepath Property.
Now, install the name- node zone; later we will clone it in order to accelerate zone creation time. Booting the name- node Zone. Provide the zone host information by using the following configuration for the name- node zone: For the host name, use name- node. Ensure the network interface name. In this article, we will use /etc/hosts for name resolution, so we won't set up DNS for host name resolution.
After finishing the zone setup, log in to the zone. Developing for Hadoop requires a Java programming environment.
You can install Java Development Kit (JDK) 6 using the following command. Verify the Java installation. When you try to log in with SSH the second time, you shouldn't get any prompt to add the host to the known keys list. Verify Name Resolution. Verify that all the Hadoop zones have the following host entries in /etc/hosts. Note: If you are using the global zone as an NTP server, you must also add its host name and IP address to /etc/hosts. Format the HDFS File System from the Name.
Node. Run the commands shown in Listing 7. R hadoop: hadoop /hdfs/. INFO namenode. Name. Node: STARTUP. Formatting the HDFS File System. On every Data. Node (data- node. Hadoop data directory to store the HDFS blocks. R hadoop: hadoop /hdfs/.
R hadoop: hadoop /hdfs/. R hadoop: hadoop /hdfs/. Start the Hadoop Cluster.
Table 3 describes the startup scripts. Table 3. Startup Scripts.
Filename. Descriptionstart- dfs. Starts the Hadoop DFS daemons, the Name. Node, and the Data. Nodes. Use this before start- mapred.
Stops the Hadoop DFS daemons. Starts the Hadoop Map. Reduce daemons, the Job. Tracker, and Task. Trackers. stop- mapred. Stops the Hadoop Map.
Reduce daemons. From the name- node zone, start the Hadoop DFS daemons, the Name. Node, and the Data. Nodes using the following command. Start the Hadoop Map/Reduce daemons, the Job.
Tracker, and Task. Trackers using the following command. To view a comprehensive status report, execute the command shown in Listing 8 to check the cluster status.
The command will output basic statistics about the cluster health, such as Name. Node details, the status of each Data. Node, and disk capacity amounts.
Checking the Cluster Status. You can find the same information on the Name. Node Web status page at http: //< namenode IP address> :5. Figure 3. Cluster Summary. Run a Map. Reduce Job.
Map. Reduce is a framework for processing parallelizable problems across huge data sets using a cluster of computers. The input and output consist of text files, each line of which contains a word and the number of times the word occurred, separated by a tab. For more information about Word. Count, see http: //wiki. Word. Count. For the input file, download the following e. Book from Project Gutenberg as a plain- text file with UTF- 8 encoding, and store the file in a temporary directory of choice, for example /tmp/data: The Outline of Science, Vol.
J. Arthur Thomson. Copy the file to HDFS using the following command.
From. Local /tmp/data/ /hdfs/data. Verify that the file is located on HDFS. Start the Map. Reduce job using the command shown in Listing 9.
INFO input. File. Input. Format: Total input paths to process : 1. WARN util. Native. Code. Loader: Unable to load native- hadoop library. Starting the Map.
Reduce Job. Verify the output data. Each line contains a word and the number of times the word occurred, separated by a tab. For more information about ZFS encryption, see . For 5 years he has specialized in virtualization and cloud computing technologies.