LinkIn Datahub Metadata Ingestion Scripts Unofficical Guide

  1. Python 3.7 Environment
    Only `requirements.txt` under `mce-cli` has highligted we should use Python 3.7 for `mce-cli` script. I have to modify all scripts under `metadata-ingestion` to be compatable with Python 3.7.
    The easiest way having a Python 3.7 in my macOS is using [Anaconda](
{‘auditHeader’: None, ‘proposedSnapshot’: (‘com.linkedin.pegasus2avro.metadata.snapshot.DatasetSnapshot’, {‘urn’: ‘urn:li:dataset:(urn:li:dataPlatform:mysql,datahub.metadata_aspect,PROD)’, ‘aspects’: [{‘owners’: [{‘owner’: ‘urn:li:corpuser:datahub’, ‘type’: ‘DATAOWNER’}], ‘lastModified’: {‘time’: 0, ‘actor’: ‘urn:li:corpuser:datahub’}}, {‘schemaName’: ‘datahub.metadata_aspect’, ‘platform’: ‘urn:li:dataPlatform:mysql’, ‘version’: 10, ‘created’: {‘time’: 1587320913, ‘actor’: ‘urn:li:corpuser:datahub’}, ‘lastModified’: {‘time’: 1587320913, ‘actor’: ‘urn:li:corpuser:datahub’}, ‘hash’: ‘’, ‘platformSchema’: {‘tableSchema’: “[(‘urn’, ‘varchar(500)’, ‘NO’, ‘PRI’, None, ‘’), (‘aspect’, ‘varchar(200)’, ‘NO’, ‘PRI’, None, ‘’), (‘version’, ‘bigint(20)’, ‘NO’, ‘PRI’, None, ‘’), (‘metadata’, ‘longtext’, ‘NO’, ‘’, None, ‘’), (‘createdon’, ‘datetime(6)’, ‘NO’, ‘’, None, ‘’), (‘createdby’, ‘varchar(255)’, ‘NO’, ‘’, None, ‘’), (‘createdfor’, ‘varchar(255)’, ‘YES’, ‘’, None, ‘’)]”}, ‘fields’: [{‘fieldPath’: ‘urn’, ‘nativeDataType’: ‘varchar(500)’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}, {‘fieldPath’: ‘aspect’, ‘nativeDataType’: ‘varchar(200)’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}, {‘fieldPath’: ‘version’, ‘nativeDataType’: ‘bigint(20)’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}, {‘fieldPath’: ‘metadata’, ‘nativeDataType’: ‘longtext’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}, {‘fieldPath’: ‘createdon’, ‘nativeDataType’: ‘datetime(6)’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}, {‘fieldPath’: ‘createdby’, ‘nativeDataType’: ‘varchar(255)’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}, {‘fieldPath’: ‘createdfor’, ‘nativeDataType’: ‘varchar(255)’, ‘type’: {‘type’: {‘com.linkedin.pegasus2avro.schema.StringType’: {}}}}]}]}), ‘proposedDelta’: None} has been successfully produced!




Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

5 Ways to Keep Your Code Clean and Efficient as a Developer

Bitspawn Bi-Weekly Progress Report: April 1st 2021 — May 5th 2021 We are focusing on restructuring…

The Road to Code…

Number of Paths — Pramp question Java

The Hello Freshers Story!

Project Dev Journal: Capstone (Part 1)

DeroGold Variety Hour

Be a Privacy Superhero and Put on a Mask

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Liangjun Jiang

Liangjun Jiang

More from Medium

Data Observability on Steroids

Innovation of Data Integration Technology in the Intelligent Era

Photobox New Data Platform

How to use Excel to query big data? MDX for Kylin! (Part-I)