Flume Collector Sink Decorator Plugin Finished

Before Christmas, I posted a blog about my trouble to write a Flume Collector Sink Decorator Plugin.

After doing some research and continue digging the underlining issue, I finally get a solution to this, which makes me super happy.

The issue I had was caused by error:

Exception in thread "main" java.lang.UnsupportedClassVersionError: garbagefilter/GarbageFilterDecorator : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at com.cloudera.flume.conf.SourceFactoryImpl.loadPluginBuilders(SourceFactoryImpl.java:160)
at com.cloudera.flume.conf.SourceFactoryImpl.(SourceFactoryImpl.java:126)
at com.cloudera.flume.conf.FlumeBuilder.(FlumeBuilder.java:89)
at com.cloudera.flume.agent.LogicalNodeManager.spawn(LogicalNodeManager.java:75)
at com.cloudera.flume.agent.FlumeNode.setup(FlumeNode.java:529)
at com.cloudera.flume.agent.FlumeNode.main(FlumeNode.java:665)

This is due to the fact that I compiled my plugin under Java 1.7 and tried to run the .jar file under Java 1.6. This error did not appear in the flume log when I ran it as a service using command, which was why I had no clue on what was going on:

$ service flume-node start

It only appeared when I ran it directly on the command line:

$ flume node -n collector1

Not exactly sure why though. Anyway, it is now working, simply follow the steps outlined here.

This is my first Java code in the last 5 years, wow. I think I will spend more time to write more Flume plugins as we need to push more jobs to Flume to do some post processing for us.

Flume Collector Sink Decorator Plugin

In the last few days I have been trying to create a Flume sink decorator plugin to filter out any data that we consider as garbage (It is very disappointing that Flume doesn’t even support a very basic filtering capability). I have followed the steps on R&D Blog with slight modifications to the HelloWorldSinkDecorator Java class. The code compiled but when I plugged it into Flume and enabled it, the collector simply died with no error message what so ever.

I had to stop working on this on Thursday afternoon as I had to join our company’s Xmas party lunch in the city. But with this outstanding task not finished before the break makes me very unhappy. Currently I still can’t find a solution and have to continue digging it when I go back to office next week.

Flume has so many problems since the first day I used it and it is not flexible enough to do lots of simply tasks. I am wondering whether we should be looking for alternatives.

Very interested to see how Flume NG will turn out.