Provide search term and filter other categories using - operator:
search_term in:inbox -category:{social promotions updates forums}
--NO_CROSS_POST
Tuesday, November 10, 2015
Monday, November 9, 2015
External jars not getting picked up in zeppelin or cli in spark, issue with mysql and spark dependency
If you cant get External jars picked up in zeppelin or cli in spark , this will help you. I first tried pointing to local jars but that did not work, you can resolve using maven dependencies as you will see below
In this example I am trying to use a mysql jdbc jar in zeppelin and cli and getting errors
But getting an exception :
java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/hive
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
Apparently this is not very documented feature of Spark (and not an issue with Zeppelin itself) Here is the code that works for me and solves the similar issue:
Dependency loading
When your code requires external library, instead of doing download/copy/restart Zeppelin, you can easily do following jobs using %dep interpreter.
Load libraries recursively from Maven repository
Load libraries from local filesystem
Add additional maven repository
Automatically add libraries to SparkCluster (You can turn off)
Dep interpreter leverages scala environment. So you can write any Scala code here.
Here's usages.
%dep
z.reset() // clean up previously added artifact and repository
// add maven repository
z.addRepo("RepoName").url("RepoURL")
// add maven snapshot repository
z.addRepo("RepoName").url("RepoURL").snapshot()
// add artifact from filesystem
z.load("/path/to.jar")
// add artifact from maven repository, with no dependency
z.load("groupId:artifactId:version").excludeAll()
// add artifact recursively
z.load("groupId:artifactId:version")
// add artifact recursively except comma separated GroupID:ArtifactId list
z.load("groupId:artifactId:version").exclude("groupId:artifactId,groupId:artifactId, ...")
// exclude with pattern
z.load("groupId:artifactId:version").exclude(*)
z.load("groupId:artifactId:version").exclude("groupId:artifactId:*")
z.load("groupId:artifactId:version").exclude("groupId:*")
// local() skips adding artifact to spark clusters (skipping sc.addJar())
z.load("groupId:artifactId:version").local()
Note that %dep interpreter should be used before %spark, %pyspark, %sql.
Thanks to Ali and Neeraj from HWX for help in solving this issue.
Thursday, February 26, 2009
Setting Up A New Web Application Using Eclipse, Maven, Ant and Tomcat
Create a new application structure using maven
Create Eclipse project
In Eclipse import as a maven project
Modify the pom.xml file:
Next Run an ant command to generate the jar dependencies property file
To download mvn dependencies with sources:
Maven Ant Integration Tutorial by Peter Thomas great way to do dependency management and use ant at the same time
Article to create a java app and then a webapp as 2 seperate projects using maven
mvn archetype:create -DgroupId=com.zeltov.appstore -DartifactId=appstore2 -DarchetypeArtifactId=maven-archetype-webapp
Create Eclipse project
mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
In Eclipse import as a maven project
Modify the pom.xml file:
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${spring-version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<version>${spring-version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>taglibs</groupId>
<artifactId>standard</artifactId>
<version>${taglibs-standard-version}</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>${jstl-version}</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>${servlet-api-version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
<properties>
<spring-version>2.5.6</spring-version>
<junit-version>4.1</junit-version>
<jstl-version>1.2</jstl-version>
<taglibs-standard-version>1.1.2</taglibs-standard-version>
<servlet-api-version>2.5</servlet-api-version>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.0.2</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>info.jtrac</groupId>
<artifactId>maven-antprops-plugin</artifactId>
</plugin>
</plugins>
</build>
<pluginRepositories>
<pluginRepository>
<id>jtrac.info</id>
<url>http://j-trac.sourceforge.net/repository</url>
</pluginRepository>
</pluginRepositories>
Next Run an ant command to generate the jar dependencies property file
mvn antprops:generate
To download mvn dependencies with sources:
mvn dependency:sources
Maven Ant Integration Tutorial by Peter Thomas great way to do dependency management and use ant at the same time
Article to create a java app and then a webapp as 2 seperate projects using maven
Subscribe to:
Posts (Atom)