Cloudera Certified Developer for Apache Hadoop

Phew!!!

Today I cleared CCD-410: Cloudera Certified Developer for Apache Hadoop exam. Really tough one, I must say…

It all started when I took the Cloudera training in November 2013. Joris Bontje (Twitter: @mids106) was the trainer. It was very wonderful and knowledgeable training.

Some tips in case you are one of the Hadoop aspirants and want to get certified. If you have been developing Map Reduce application for some significant period then it is fine otherwise it is going to be tough to clear the exam. Cloudera training will definitely help you in that case as we did some hands-on on various map reduce programs. It started from simple map reduce to various interfaces like Reducer, Mapper, Partitioner etc.

You should read “Hadoop: The Definitive Guide” thoroughly before attempting the exam. If you have not worked on Mapreduce and didn’t take the training as well, then please read “Hadoop in Action” too apart from doing good amount of hands-on exercises on either Hortonworks or Cloudera VM.

Since I attended the training I got unlimited practice exams as complimentary. After many trials I was able to achieve 100% marks. It consisted of 60 questions (actual exam had 55). The difficulty level was higher in actual exam though 1 or 2 questions were taken from practice exam.

The sections are listed on this page. One should have good understanding of Job execution, Data Management and Queries. Questions from features like Counters, Secondary Sorts, HiveQL were also asked in the form of Map reduce programs. HDFS commands and parameters to run the program must be understood properly.

No handkerchief, pen/pencil, mobile/ bag was allowed in the room. The whole process was under CCTV surveillance. Exam started as I clicked on “Agree” button of terms and conditions. Clock started ticking and I started answering the multiple choice. MCQs may contain some of the questions where multiple answers are possible.

Many answers may look similar so it is important to read the question and the options available again and again till you are absolutely sure (It is the matter of 295 USD  :-) ).

All the best friends!!

Resources that I got knowledge from:

• Hadoop-The Definitive Guide
• 24 Interview Questions
• Cloudera Practice Exam (Complimentary with training)
• Cloudera Certification FAQs
• Cloudera and Hortonworks VM
• Cloudera Study Guide

Gartner Magic Quadrant For BI & Analytics 2013

Download PDF from here

Enums and Singleton - A Friendly Relation

Those coming from C++ background must be knowing what enumerated datatypes are. But Java enums may give you some surprises. Enums are used to define the predefined set of constants that are logically related to each other. For an example, a set of months names or a set of colours or currency notes that are fixed in numbers.

Enums in Java were introduced in jdk 1.5. Prior to that developers had to create a class containing public static final integers to define constants. Something like this:

public static final int CONST1 = 0;
public static final int CONST2 = 1;

They could be accessible using CLASS_NAME.CONST1, and so on.
Though introduced late in Java, enums are much more powerful than any other programming languages.

Here are some of its highlights:

• Enums can be defined as simply as following:

public enum Color
{
  RED, BLUE, BLACK, GREEN
}

Keep in mind that enums can be compared only with enums and not with integers or any other data type. Enums can be compared in switch statement as well.

• Enums are like classes and interfaces. They can have member variables as well as member functions defined in it. Unlike C++ where enum can contain only constants. 

public enum Color
{
  RED, BLUE, BLACK, GREEN;
  
  private int x;
  public void show()
  {
    System.out.println("inside show" + x);
  }
}

• Enums can have abstract methods which mean that the methods can behave differently for different values of enum variable.

public enum Color
{
  RED {
    @Override
    public void show()
    {
      System.out.println("inside show" + RED);
      
    }
  }, BLUE {
    @Override
    public void show()
    {
      System.out.println("inside show" + BLUE);
      
    }
  }, BLACK {
    @Override
    public void show()
    {
      System.out.println("inside show" + BLACK);
      
    }
  }, GREEN {
    @Override
    public void show()
    {
      System.out.println("inside show" + GREEN);
      
    }
  };
  
  private int x;
  public abstract void show();
}

• Another special behavior different from C++ is that it can implement interfaces as well. By default it has implicitly implemented Serializable and Cloneable interfaces.

public enum Color implements Runnable
{
  RED, BLUE, BLACK, GREEN;
  
  @Override
  public void run()
  {
    // TODO Auto-generated method stub
    
  }
}

• Enums can have private constructor. In the following example enum variable with RED as value would have x as 1 and enum variable with BLACK value would have x as 7. Defining RED(1) instead of RED calls the contructor.

public enum Color
{
  RED(1), BLUE(3), BLACK(7), GREEN(9);
  
  private int x;
  private Color(int in_x)
  {
    this.x = in_x;
  }
}


Though enums can implement interfaces but they cannot extend the classes.

For more information on enums.
http://javarevisited.blogspot.in/2011/08/enum-in-java-example-tutorial.html

Now how enums can be used in Singleton pattern. We know how to implement a Singleton pattern. A static private object, a private constructor, and a public static getter method of the instance  in a class form a Singleton class.

Then there comes popular multi-threading issue which can be easily solved using double checking solution. 

public class Singleton{
     private Singleton instance;
  
     private Singleton(){}
  
     public Singleton getInstance(){
         if(instance== null){
            synchronized(Singleton.class){
                // Double check here
                if(instance== null){
                    instance= new Singleton();
                }
            }
         }
         return instance;
     }
}

The other possible and better way of implementing Singleton is by having early initialization of the instance variable. The instance would be initialized when the class is first loaded and thus ensures the that only single instance is created.

public class Singleton{

    private static final Singleton instance = new Singleton();
  
    private Singleton(){}

    public static Singleton getInstance(){
        return instance ;
    }
}

So these are some of the possible variants of Singleton pattern. But these all are using classes. The most simple way of implementing the Singleton pattern came into the picture when enums were introduced in the world of java. 

public enum Logger
{
  instance;
  
  public void log(String str)
  {
    
  }
}

That's it!! Java ensures that Enums are type safe as well as thread safe. The instance can be accessed in following way:

Logger.instance.log("test");

Since enums can implement any of the interface (they are Serializable and Cloneable) and can also have member methods, they can be pretty useful in achieving Singleton behavior.

This is the power of enums and thus the power of java.

Read more: http://javarevisited.blogspot.com/2012/07/why-enum-singleton-are-better-in-java.html#ixzz2RZA7XTvL

Thanks for reading!

Dependency Injection in Spring - Short Article

Any software's life depends upon it's architecture and design. Stronger is the design better would be the sustainability against the series of changes that come along with the journey. Weaker design makes the software vulnerable and thus fragile. There are certain design principles that act as the pillars of good design. You may not be knowing the design patterns but following these principles could lead you to strong design. 

The beauty of java is that it's frameworks force you to follow certain principles. One of these is Dependency Inversion in Spring.

Let us understand this principle through a hypothetical example. Suppose there is a car manufacturing unit which manufactures different models of the car. Now, initial code may look like this:

class CarManufacturingUnit
{
 public void setup(int index)
{
    if (index == 0)
    {
      return new Omni();
    }
    else if (index == 1)
    {
      return new Alto();
    }
    else ...
  }
}

CarManufacturingUnit --> Omni, Alto

Here the class CarManufacturingUnit is directly depending on the low level classes  of different car models viz. Omni, Alto, Esteem, Swift etc.. That means any modifications done to these low level classes would eventually may modify the high level class as well. We may not want to get into the situation where code suffers from the ripple effect of any small change which would then result in increasing testing effort as QA would have to test other scenarios as well.

To get out of this problem we could create an abstraction layer between these two levels. The high level class would depend only on the abstraction of these low level classes. Some what like this:

class CarManufacturingUnit
{
  private CarFactory carFactory;
  public void setup(int index)
  {
    CarModel carModel = carFactory.getCar(index);
  }
  
}

class CarFactory
{
  CarModel getCar(int index)
  {
    if (index == 0)
    {
      return new Omni();
    }
    else if (index ==1)
    {
      return new Alto();
    }
    else ...
  }
}

Here, CarManufacturingUnit doesn't depend directly upon the low level classes, rather an abstraction layer CarModel has been created.

CarManufacturingUnit --> CarModel <-- Omni, Alto

Thus by having the abstraction we have inverted the dependency. Now car manufacturing unit doesn't depends directly on models of the car. Changing any of these models wouldn't effect the high level class. 

The principle is also called as Inversion of Control(IoC). This dependency Inversion can very easily achieved using factory class or factory methods which segregate the object creation logic from the main business method.

Now when we study spring in Java we would found that this is the core principle of spring technology. Spring achieves this principle and forces in developers to use this by Dependency Injection technique.

class CarManufacturingUnit
{
  private CarFactory carFactory;
  
  public void setCarFactory(CarFactory carFactory)
  {
    this.carFactory = carFactory;
  }
  public void setup(int index)
  {
    CarModel carModel = carFactory.getCar(index);
  }
  
}

The low level object creation role has been taken care by spring container's initialization logic itself. Whenever the object is instantiated it's dependencies are also get initiated automatically. 

In the above example, we have provided setter method for carFactory member variable. The entry in bean registry XML creates the bean of CarFactory and injects it's object in dependent class, CarManufacturingUnit via setter method. We can have constructor as well in place of setter.

Entry in spring bean:

<bean id="CarManufacturingUnit" class="CarManufacturingUnit">
     <property name="carFactory" ref="carFactory"/>
</bean>


<bean id="carFactory" class="CarFactory">
</bean>


Thus spring has completely segregated the business logic and object creation logic. Developer needs not to take care of objects life cycle as well. Spring controls the their life cycle pretty well.

I tried to cover Dependency Inversion / IoC principle implemented inside Spring container.