Deploy Atlas Hive Hook — Part I

Fix the Hive version issue in Atlas 2.1.0

create table foo;
[HiveServer2-Background-Pool: Thread-52]: HiveHook.run(): failed to process operation CREATETABLE
java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.Database.getCatalogName()Ljava/lang/String;
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.getDatabaseName(HiveMetaStoreBridge.java:577) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.getQualifiedName(AtlasHiveHookContext.java:201) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.init(AtlasHiveHookContext.java:293) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.<init>(AtlasHiveHookContext.java:83) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.<init>(AtlasHiveHookContext.java:64) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:175) [hive-bridge-2.1.0.jar:2.1.0]
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java#getDatabaseName()public static String getDatabaseName(Database hiveDB) {
String dbName = hiveDB.getName().toLowerCase();
/*
String catalogName = hiveDB.getCatalogName() != null ? hiveDB.getCatalogName().toLowerCase() : null;

if (StringUtils.isNotEmpty(catalogName) && !StringUtils.equals(catalogName, DEFAULT_METASTORE_CATALOG)) {
dbName = catalogName + SEP + dbName;
}
*/

return dbName;
}
mvn clean install -DskipTests

How Hive Hook Works — Workflow

Apache Atlas Hive Hook Workflow diagram
  1. Does Apache Atlas have to be up and running to use Atlas Hive Hook or other hook in general? Seems Kafka is the middle man so Atlas doesn’t to be running. That’s correct. Atlas doesn’t need to be up & running to use Atlas Hive Hook
  2. Does Kafka have to be up & running to use Atlas Hive Hook? Actually you can still *use* Atlas Hook but the extracted metadata won’t go anywhere.

The Problem of the latest Apache Atlas

Caused by: org.apache.solr.common.SolrException: Cannot connect to cluster at localhost:2181: cluster not found/not ready
at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:385)
at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:141)
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:383)
at org.janusgraph.diskstorage.solr.Solr6Index.<init>(Solr6Index.java:218)
... 101 more

The Solution

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Learning From School of Fishes & Organizing Chaos

Exploiting Software -ATTACK PATTERN — Part 1

How to start learning Robotics? (Intro to Robotics Engineering)

SSH Deployment Task in Azure Pipelines

Membuat Smart Home Arduino Android Bluetooth HC-05

Getting started with Robot Framework + Selenium for automation testing (7) — Resource and scopes

NOTABLE FEATURES OF UNITY 2020.1

Unity 3D Game Development

Why is programming so frustrating?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Liangjun Jiang

Liangjun Jiang

More from Medium

Building Apache Livy 0.8.0 for Spark 3.x

How to Handle Nested Data in Apache Druid vs Rockset

Deploy Open Source Datahub — Part II

Ultra-Fast Data Loading and Testing in Altinity.Cloud