Passing configuration options to Hadoop

Posted March 7th, 2013 in Razno by Metod

If you want to override Hadoop job configuration options, you can do so via command line -D key=value.

When I wanted to pass multiple configuration options including some special characters:

hadoop jar MyJob.jar MyJob -D key=something,else -D key2=jdbc:mysql://host?user=u&password=p

I got some errors like:

-bash: -D: command not found

The solution is to put values in quotes like this:

hadoop jar MyJob.jar MyJob -D key="something,else" -D key2="jdbc:mysql://host?user=u&password=p"

Enjoy! 🙂