{"id":396,"date":"2019-02-24T07:18:01","date_gmt":"2019-02-24T07:18:01","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=396"},"modified":"2020-07-10T05:19:46","modified_gmt":"2020-07-10T05:19:46","slug":"hadoop-flume-tutorial","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/hadoop\/hadoop-flume-tutorial\/","title":{"rendered":"Hadoop flume tutorial"},"content":{"rendered":"

 <\/p>\n

Hadoop flume tutorial<\/h1>\n

 <\/p>\n

Hadoop flume tutorial<\/strong>, Welcome to the world of Hadoop<\/a> flume Tutorials. In these Tutorials, one can explore\u00a0how to fetch Flume Data from Twitter. Learn More advanced Tutorials on flume configuration in Hadoop from India\u2019s Leading Hadoop Training institute<\/a> which Provides Advanced Hadoop Course<\/a> for those tech enthusiasts who wanted to explore the technology from scratch to advanced level like a Pro.<\/p>\n

We Prwatech<\/a>, the Pioneers of Hadoop Training<\/a> Offering advanced Certification course and Hadoop flume setup to those who are keen to explore the technology under the World-class Training Environment.<\/p>\n

 <\/p>\n

Fetching Flume Data from Twitter<\/h2>\n

 <\/p>\n

    \n
  1. Ubuntu v12 (or above)<\/li>\n
  2. Apache flume 1.3.1 bin.tar<\/li>\n
  3. Flume source 1.0. SNAPSHOT<\/li>\n<\/ol>\n

     <\/p>\n

    Twitter data analysis using flume<\/h3>\n

    Make a new directory in \/usr\/lib for flume<\/p>\n

    $cd \/usr\/lib\/<\/p>\n

    \"Hadoop<\/strong><\/p>\n

    $mkdir myflume<\/p>\n

    \"Hadoop<\/strong><\/p>\n

    moving the apache-flume 1.3.1 bin.tar to \/usr\/lib\/myflume<\/p>\n

    $sudo mv \/home\/cloudera\/Desktop\/apache flume 1.3.1 bin.tar \/usr\/lib\/myflume<\/p>\n

    Untar the file.<\/p>\n

    $sudo tar -zxvf apache flume 1.3.1 bin.tar<\/p>\n

    Now we will have two files in \/usr\/lib\/myflume<\/p>\n

    apache flume 1.3.1 bin.tar.gz<\/p>\n

    apache flume 1.3.1 bin<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    This apache “flume 1.3.1 bin” will have many directories one among them will be lib . move the “flume-source -1.0.SNAPSHOT.jar to this lib. $ sudo mv \/home\/cloudera\/Desktop\/flume source 1.0. SNAPSHOT<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    \/usr\/lib\/myflume\/apache-flume 1.3.1 bin \/lib\/<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    Go to the conf directory<\/p>\n

    $ cd ..\/conf\/<\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    Create a copy of flume-env.sh.template as flume-env.sh in the same \/conf\/ dir. as :<\/strong><\/p>\n

    $ cp \/usr\/lib\/myflume\/apache flume 1.3.1 bin \/conf\/flume-env.sh.template\/usr\/lib\/myflume\/apache flume 1.3.1 bin \/conf\/flume-env.sh<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    Hence it will contain :<\/p>\n

     <\/p>\n

    flume-env.sh.template
    \n2 flume-env.sh<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    configuring the flume-env.sh as<\/p>\n

    $ sudo gedit flume-env.sh<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    JAVA_HOME=\/usr\/lib\/jvm\/java-6-sun<\/p>\n

    FLUME_CLASSPATH=”\/usr\/lib\/myflume\/apache flume 1.3.1 bin\/lib\/flume-source -1.0.SNAPSHOT.jar”<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    CREATING API CREDENTIALS:<\/strong><\/p>\n

    app twitter –> twitter application management :
    \nsign in : username:
    \npassword :<\/p>\n

     <\/p>\n

    CREATE NEW API :<\/strong><\/p>\n

     <\/p>\n

    Application Details:<\/p>\n

    * Name:<\/p>\n

    * Description:<\/p>\n

    * Website :<\/p>\n

    * Callback URL: not require<\/p>\n

    *Finally click on \u201cyes I agree\u201d<\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    KEY AND ACCESS TOKENS :<\/strong><\/p>\n

     <\/p>\n

    \"Hadoop<\/strong><\/p>\n

     <\/p>\n

    #NOTE: Get the following information and fill it in the flume.txt file.<\/p>\n

      \n
    • ConsemerKey<\/li>\n
    • ConsumerSecret:<\/li>\n
    • AccessTokens:<\/li>\n
    • AccessTokenSecret:<\/li>\n<\/ul>\n

       <\/p>\n

      Now move the flume.txt file to the Cloudera<\/p>\n

      create a new file in the \/conf\/ directory.<\/p>\n

      $ cd \/usr\/lib\/myflume\/apache flume 1.3.1 bin\/conf\/<\/p>\n

      $sudo gedit flume.conf<\/p>\n

       <\/p>\n

      \"Hadoop<\/p>\n

       <\/p>\n

      copy the content of the flume.txt in this file and save it<\/p>\n

       <\/p>\n

      \"Hadoop<\/strong><\/p>\n

       <\/p>\n

      Go to the bin directory and fire the final command:<\/strong><\/p>\n

        \n
      • $cd \/usr\/lib\/myflume\/apache flume 1.3.1 bin\/bin\/<\/li>\n
      • $.\/flume-ng agent -n TwitterAgent -c conf -f \/usr\/lib\/myflume\/apache \/conf\/flume.conf<\/li>\n<\/ul>\n

         <\/p>\n

        \"Hadoop<\/strong><\/p>\n

         <\/p>\n

        Now we use our virtual machine web browser to see our records collected by user from Twitter.<\/p>\n

         <\/p>\n

        \"Hadoop<\/strong><\/p>\n

         <\/p>\n

        \"Hadoop<\/strong><\/p>\n

         <\/p>\n

        Goto NameNode Status > user > flume > tweets<\/p>\n

         <\/p>\n

        \"Hadoop<\/strong><\/p>\n

         <\/p>\n

        As you can see all the data collected from twitter is in json format which is needed to be converted in csv so that user can understand the data collected.<\/p>\n

         <\/p>\n

        For this we can use online json to csv converter to convert the following data.<\/p>\n

         <\/p>\n