This project is read-only.
1
Vote

Copy from local folder to Hadoop

description

Hi,
When i try to create file using CopyFromLocal on hadoop its through an error:

Step:1 Create a Directory name TestDir
WebHDFS hdfs = new WebHDFS("Azure VM DNS Name like abc.cloudapp.net/public IP address of VM", "user Name", "50070");
// Create Directory
hdfs.MkDir("user/hive/TestDir");

Step:2 Try to copy txt file from local machine to hadoop.

// Copy file from local to hdfs (Upload)
hdfs.CopyFromLocal(@"C:\Users\Public\TestData.txt", "user/hive/TestDir", 1);

Step:3:
When i go inside the CopyFromLocal function first i call rest api like this
http://abc.cloudapp.net:50070/webhdfs/v1/user/hive/TestDir/TestData.txt?op=CREATE&overwrite=true&user.name=hadoop

i got an url from var url = new Uri(resp.Headers["Location"]);
like
http://InternalIP of the AZURE VM:50075/webhdfs/v1/user/hive/Shretest/TestData.txt?op=CREATE&overwrite=true
Explain in image 1.

http://0.0.0.0:50075/webhdfs/v1/user/hive/Shretest/TestData.txt?op=CREATE&overwrite=true
when i call this url by HttpWebRequest got an error A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 0.0.0.0:50075

When i change this internal ip to public IP or DNS name of the VM it works but but throws an error in the last HttpWebResponse response = (HttpWebResponse)request.GetResponse();
The remote server returned an error: (403) Forbidden. Explain in image 2.

Please let me know the issue.

Thanks
Shreyansh

file attachments

comments

shreyansh92 wrote Jul 2, 2015 at 1:50 PM

Done :)