Deploy Atlas Hive Hook — Part I

Fix the Hive version issue in Atlas 2.1.0

create table foo;
[HiveServer2-Background-Pool: Thread-52]: HiveHook.run(): failed to process operation CREATETABLE
java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.api.Database.getCatalogName()Ljava/lang/String;
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.getDatabaseName(HiveMetaStoreBridge.java:577) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.getQualifiedName(AtlasHiveHookContext.java:201) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.init(AtlasHiveHookContext.java:293) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.<init>(AtlasHiveHookContext.java:83) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.AtlasHiveHookContext.<init>(AtlasHiveHookContext.java:64) ~[hive-bridge-2.1.0.jar:2.1.0]
at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:175) [hive-bridge-2.1.0.jar:2.1.0]
addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java#getDatabaseName()public static String getDatabaseName(Database hiveDB) {
String dbName = hiveDB.getName().toLowerCase();
/*
String catalogName = hiveDB.getCatalogName() != null ? hiveDB.getCatalogName().toLowerCase() : null;

if (StringUtils.isNotEmpty(catalogName) && !StringUtils.equals(catalogName, DEFAULT_METASTORE_CATALOG)) {
dbName = catalogName + SEP + dbName;
}
*/

return dbName;
}
mvn clean install -DskipTests

How Hive Hook Works — Workflow

Apache Atlas Hive Hook Workflow diagram
  1. Does Apache Atlas have to be up and running to use Atlas Hive Hook or other hook in general? Seems Kafka is the middle man so Atlas doesn’t to be running. That’s correct. Atlas doesn’t need to be up & running to use Atlas Hive Hook
  2. Does Kafka have to be up & running to use Atlas Hive Hook? Actually you can still *use* Atlas Hook but the extracted metadata won’t go anywhere.

The Problem of the latest Apache Atlas

Caused by: org.apache.solr.common.SolrException: Cannot connect to cluster at localhost:2181: cluster not found/not ready
at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:385)
at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:141)
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:383)
at org.janusgraph.diskstorage.solr.Solr6Index.<init>(Solr6Index.java:218)
... 101 more

The Solution

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How to install wildcard certificate from Let’s Encrypt on NGINX running on Ubuntu, Linux?

How to upload/update/delete data to Google Firebase with Python

Apache Kafka

What framework is a good choice for web developers?

Robust Server-less Performance Monitoring & Incident Alerting Platform with AWS Lambda and Google…

Advent Of Code — Science for Hungry People — Puzzle 15

Getting Started with Android Automation using Tasker

Unonomics with Dig: Stats and Maths (Part 2) — MCH revisited

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Liangjun Jiang

Liangjun Jiang

More from Medium

Apache Airflow: Understanding Operators

Apache Pulsar Edge IoT Applications with Python for TVOC

Release News! Apache DolphinScheduler 2.0.5 Optimizes The Fault Tolerance Process of Worker

Building our big data platform (part 6)