Forum

This topic contains 4 replies, has 2 voices, and was last updated by  Anshuman 8 months, 1 week ago.

Viewing 5 posts - 1 through 5 (of 5 total)
  • Author
    Posts
  • #3061 Reply

    jitendra
    Participant

    Hi ALL,
    PLEASE FIND THE BELOW ATTACHED INFORMATION OF HIVE.

    name value
    hive.exec.job.debug.capture.stacktraces TRUE
    hive.stats.reliable FALSE
    hive.metastore.connect.retries 3
    hive.auto.progress.timeout 0
    hive.hmshandler.retry.interval 1000
    hive.insert.into.external.tables TRUE
    hive.exec.default.partition.name __HIVE_DEFAULT_PARTITION__
    hive.metastore.execute.setugi TRUE
    hive.fetch.task.conversion minimal
    hive.join.cache.size 25000
    hive.enforce.sorting FALSE
    hive.zookeeper.session.timeout 600000
    hive.mapred.mode nonstrict
    hive.lock.manager org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
    hive.limit.optimize.enable FALSE
    hive.metastore.rawstore.impl org.apache.hadoop.hive.metastore.ObjectStore
    hive.mapjoin.size.key 10000
    hive.querylog.location /tmp/root
    hive.lockmgr.zookeeper.default.partition.name __HIVE_DEFAULT_ZOOKEEPER_PARTITION__
    hive.merge.mapredfiles FALSE
    hive.metastore.ds.retry.interval 1000
    hive.metastore.client.connect.retry.delay 1
    hive.metastore.archive.intermediate.original _INTERMEDIATE_ORIGINAL
    hive.script.auto.progress FALSE
    hive.multi.insert.move.tasks.share.dependencies FALSE
    hive.optimize.ppd.storage TRUE
    hive.query.id root_20170319002424_61de6a85-fbc7-4713-bc3d-f8966a6eb94e
    hive.script.serde org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    hive.archive.har.parentdir.settable TRUE
    hive.mapjoin.followby.gby.localtask.max.memory.usage 0.55
    hive.added.jars.path file:///usr/lib/hive/lib/hive-builtins-0.10.0-cdh4.4.0.jar
    hive.test.mode.samplefreq 32
    hive.task.progress FALSE
    hive.stats.retries.max 0
    hive.stats.autogather TRUE
    hive.stats.collect.rawdatasize TRUE
    hive.debug.localtask FALSE
    hive.optimize.bucketmapjoin.sortedmerge FALSE
    hive.index.compact.query.max.entries 10000000
    hive.join.emit.interval 1000
    hive.test.mode FALSE
    hive.zookeeper.quorum localhost.localdomain
    hive.input.format org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
    hive.metastore.server.max.threads 100000
    hive.default.fileformat TextFile
    hive.metastore.server.min.threads 200
    hive.security.metastore.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
    hive.map.groupby.sorted FALSE
    hive.mapjoin.cache.numrows 25000
    hive.skewjoin.mapjoin.min.split 33554432
    hive.index.compact.binary.search TRUE
    hive.mapred.reduce.tasks.speculative.execution TRUE
    hive.limit.optimize.fetch.max 50000
    hive.merge.current.job.has.dynamic.partitions FALSE
    hue.kerberos.principal.shortname hue
    hive.exec.show.job.failure.debug.info TRUE
    hive.variable.substitute TRUE
    hive.mapjoin.check.memory.rows 100000
    hive.cli.errors.ignore FALSE
    hive.server2.thrift.min.worker.threads 5
    hive.optimize.bucketmapjoin FALSE
    hive.insert.into.multilevel.dirs FALSE
    hive.server2.allow.user.substitution TRUE
    hive.stats.dbclass jdbc:derby
    hive.query.string select count(*) from txnrecords
    hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.MemoryTokenStore
    hive.optimize.metadataonly TRUE
    hive.merge.smallfiles.avgsize 16000000
    hive.exec.plan hdfs://localhost.localdomain:8020/tmp/hive-root/hive_2017-03-19_00-24-17_864_7042588480852712331-1/-mr-10003/a3b02e67-bcdf-4887-811b-f4290b260a94
    hive.exec.reducers.max 999
    hive.security.authorization.enabled FALSE
    hive.exec.dynamic.partition.mode strict
    hive.server2.in.mem.log.size 131072
    hive.server2.thrift.sasl.qop auth
    hive.groupby.skewindata FALSE
    hive.autogen.columnalias.prefix.includefuncname FALSE
    hive.metastore.batch.retrieve.max 300
    hive.merge.mapfiles TRUE
    hive.exec.max.dynamic.partitions.pernode 100
    hive.metastore.sasl.enabled FALSE
    hive.mergejob.maponly TRUE
    hive.metastore.batch.retrieve.table.partition.max 1000
    hive.exec.mode.local.auto FALSE
    hive.server.tcp.keepalive TRUE
    hive.optimize.skewjoin.compiletime FALSE
    hive.exec.mode.local.auto.inputbytes.max 134217728
    hive.zookeeper.client.port 2181
    hive.optimize.reducededuplication TRUE
    hive.exec.perf.logger org.apache.hadoop.hive.ql.log.PerfLogger
    hive.server2.thrift.login.timeout 20
    hive.transform.escape.input FALSE
    hive.metastore.event.expiry.duration 0
    hive.server2.enable.impersonation FALSE
    hive.sample.seednumber 0
    hive.session.id 3d1e105c-a9f0-4c82-8f3f-2f000e5328a8
    hive.limit.row.max.size 100000
    hive.server2.authentication NONE
    hive.auto.convert.join FALSE
    hive.mapred.supports.subdirectories FALSE
    hive.mapred.local.mem 0
    hive.hashtable.loadfactor 0.75
    hive.ppd.recognizetransivity TRUE
    hive.exec.reducers.bytes.per.reducer 1073741824
    hive.security.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider
    hive.variable.substitute.depth 40
    hive.merge.input.format.block.level org.apache.hadoop.hive.ql.io.rcfile.merge.RCFileBlockMergeInputFormat
    hive.exec.max.created.files 100000
    hive.map.aggr TRUE
    hive.querylog.enable.plan.progress TRUE
    hive.support.concurrency TRUE
    hive.metastore.authorization.storage.checks FALSE
    hive.cli.print.header FALSE
    hive.merge.rcfile.block.level TRUE
    hive.metastore.client.socket.timeout 300
    hive.udtf.auto.progress FALSE
    hive.limit.optimize.limit.file 10
    hive.stats.dbconnectionstring jdbc:derby:;databaseName=TempStatsStore;create=true
    hive.exec.job.debug.timeout 30000
    hive.enforce.bucketmapjoin FALSE
    hive.unlock.numretries 10
    hive.enforce.sortmergebucketmapjoin FALSE
    hive.mapjoin.bucket.cache.size 100
    hive.session.silent FALSE
    hive.script.operator.truncate.env FALSE
    hive.metastore.event.clean.freq 0
    hive.conf.validation TRUE
    hive.autogen.columnalias.prefix.label _c
    hive.map.aggr.hash.force.flush.memory.threshold 0.9
    hive.lock.numretries 100
    hive.metastore.cache.pinobjtypes Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order
    hive.skewjoin.mapjoin.map.tasks 10000
    hive.metastore.failure.retries 1
    hive.ppd.remove.duplicatefilters TRUE
    hive.optimize.index.autoupdate FALSE
    hive.server2.blocking.query TRUE
    hive.exec.scratchdir /tmp/hive-root
    hive.merge.size.per.task 256000000
    hive.stats.jdbc.timeout 30
    hive.optimize.index.filter.compact.maxsize -1
    hive.warehouse.subdir.inherit.perms TRUE
    hive.metastore.thrift.framed.transport.enabled FALSE
    hive.zookeeper.namespace hive_zookeeper_namespace_hive1
    hive.exec.parallel FALSE
    hive.optimize.index.groupby FALSE
    hive.server2.authorization.external.exec TRUE
    hive.server2.table.type.mapping HIVE
    hive.optimize.listbucketing FALSE
    hive.optimize.union.remove FALSE
    hive.cli.prompt hive
    hive.jobname.length 50
    hive.server2.thrift.max.worker.threads 100
    hive.fileformat.check TRUE
    hive.mapper.cannot.span.multiple.partitions FALSE
    hive.exec.dynamic.partition TRUE
    hive.lock.sleep.between.retries 60
    hive.stats.jdbcdriver org.apache.derby.jdbc.EmbeddedDriver
    hive.stats.collect.tablekeys FALSE
    hive.exec.script.allow.partial.consumption FALSE
    hive.stats.atomic FALSE
    hive.optimize.index.filter FALSE
    hive.cli.print.current.db FALSE
    hive.hwi.listen.port 9999
    hive.exec.max.dynamic.partitions 1000
    hive.script.recordwriter org.apache.hadoop.hive.ql.exec.TextRecordWriter
    hive.stats.retries.wait 3000
    hive.hmshandler.retry.attempts 1
    hive.script.operator.id.env.var HIVE_SCRIPT_OPERATOR_ID
    hive.metastore.archive.intermediate.extracted _INTERMEDIATE_EXTRACTED
    hive.map.aggr.hash.min.reduction 0.5
    hive.optimize.index.filter.compact.minsize 5368709120
    hive.mapjoin.localtask.max.memory.usage 0.9
    hive.error.on.empty.partition FALSE
    hive.exec.script.trust FALSE
    hive.exec.concatenate.check.index TRUE
    hive.outerjoin.supports.filters TRUE
    hive.exec.mode.local.auto.input.files.max 4
    hive.server2.in.mem.logging TRUE
    hive.query.result.fileformat TextFile
    hive.archive.enabled FALSE
    hive.exec.tasklog.debug.timeout 20000
    hive.enforce.bucketing FALSE
    hive.security.metastore.authorization.manager org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider
    hive.metastore.ds.retry.attempts 1
    hive.zookeeper.clean.extra.nodes FALSE
    hive.index.compact.query.max.size 10737418240
    hive.exec.parallel.thread.number 8
    hive.metastore.local FALSE
    hive.input.format.sorted FALSE
    hive.hashtable.initialCapacity 100000
    hive.metastore.server.tcp.keepalive TRUE
    hive.metastore.archive.intermediate.archived _INTERMEDIATE_ARCHIVED
    hive.rework.mapredwork FALSE
    hive.index.compact.file.ignore.hdfs FALSE
    hive.map.aggr.hash.percentmemory 0.5
    hive.entity.separator @
    hive.hbase.wal.enabled TRUE
    hive.optimize.skewjoin FALSE
    hive.exim.uri.scheme.whitelist hdfs,pfile
    hive.entity.capture.input.URI FALSE
    hive.metadata.move.exported.metadata.to.trash TRUE
    hive.mapjoin.followby.map.aggr.hash.percentmemory 0.3
    hive.optimize.groupby TRUE
    hive.mapred.partitioner org.apache.hadoop.hive.ql.io.DefaultHivePartitioner
    hive.optimize.ppd TRUE
    hive.server2.thrift.port 10000
    hive.exec.compress.output FALSE
    hive.script.recordreader org.apache.hadoop.hive.ql.exec.TextRecordReader
    hive.heartbeat.interval 1000
    hive.skewjoin.key 100000
    hive.exec.counters.pull.interval 1000
    hive.exec.submitviachild FALSE
    hive.metastore.warehouse.dir /user/hive/warehouse
    hive.security.authenticator.manager org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator
    hive.optimize.cp TRUE
    hive.querylog.plan.progress.interval 60000
    hive.exec.local.scratchdir /tmp/root
    hive.cluster.delegation.token.store.zookeeper.znode /hive/cluster/delegation
    hive.multigroupby.singlereducer TRUE
    hive.exec.list.bucketing.default.dir HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME
    hive.fetch.output.serde org.apache.hadoop.hive.serde2.DelimitedJSONSerDe
    hive.binary.record.max.length 1000
    hive.hwi.listen.host 0.0.0.0
    hive.hadoop.supports.splittable.combineinputformat FALSE
    hive.stats.ndv.error 20
    hive.test.mode.prefix test_
    hive.downloaded.resources.dir /tmp/${hive.session.id}_resources
    hive.mapjoin.smalltable.filesize 25000000
    hive.exec.rcfile.use.explicit.header TRUE
    hive.groupby.mapaggr.checkinterval 100000
    hive.internal.ddl.list.bucketing.enable FALSE
    hive.exec.drop.ignorenonexistent TRUE
    hive.hmshandler.force.reload.conf FALSE
    hive.server.read.socket.timeout 10
    hive.exec.compress.intermediate FALSE
    hive.metastore.uris thrift://localhost.localdomain:9083
    hive.exec.script.maxerrsize 100000
    hive.exec.rowoffset FALSE
    hive.lock.mapred.only.operation FALSE
    hive.current.db jack
    hive.start.cleanup.scratchdir FALSE

    #3062 Reply

    jitendra
    Participant

    Hive properties with default value and description.

    hive.enforce.sorting–Default Value: Hive 0.x: false
    Hive 1.x: false
    Hive 2.x: removed, which effectively makes it always true.
    Added In: Hive 0.6.0
    Whether sorting is enforced. If true, while inserting into the table, sorting is enforced.

    hive.zookeeper.session.timeout Default Value: Hive 0.7.0 to 1.1.x: 600000ms
    Hive 1.2.0 and later: 1200000ms.
    Added In: Hive 0.7.0
    ZooKeeper client’s session timeout (in milliseconds). The client is disconnected, and as a result, all locks released, if a heartbeat is not sent in the timeout.

    hive.mapred.mode Default Value: Hive 0.x: nonstrict
    Hive 1.x: nonstrict
    Hive 2.x: strict
    Added In: Hive 0.3.0
    The mode in which the Hive operations are being performed. In strict mode, some risky queries are not allowed to run. For example, full table scans are prevented (see HIVE-10454) and ORDER BY requires a LIMIT clause.

    hive.lock.manager Default Value: org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
    Added In: Hive 0.7.0
    The lock manager to use when hive.support.concurrency is set to true.

    hive.limit.optimize.enable Default Value: false
    Added In: Hive 0.8.0
    Whether to enable to optimization to trying a smaller subset of data for simple LIMIT first.

    hive.metastore.rawstore.impl Default Value: org.apache.hadoop.hive.metastore.ObjectStore
    Added In: Hive 0.8.1
    Name of the class that implements org.apache.hadoop.hive.metastore.rawstore interface. This class is used to store and retrieval of raw metadata objects such as table, database.

    hive.querylog.location /tmp/root
    USED TO STORE THE LOG OF QUERY.

    hive.lockmgr.zookeeper.default.partition.name Default Value: __HIVE_DEFAULT_ZOOKEEPER_PARTITION__
    Added In: Hive 0.7.0 with HIVE-1293
    The default partition name when ZooKeeperHiveLockManager is the hive lock manager.

    hive.merge.mapredfiles Default Value: false
    Added In: Hive 0.4.0
    Merge small files at the end of a map-reduce job.

    hive.metastore.ds.retry.interval
    Default Value: 1000
    Added In: Hive 0.6.0
    The number of milliseconds between metastore retry attempts.

    hive.metastore.client.connect.retry.delay
    Default Value: 1
    Added In: Hive 0.7.0
    Number of seconds for the client to wait between consecutive connection attempts.

    #3124 Reply

    RahulPainuly

    PROPERTIES CDH4 21-30

    hive.merge.mapredfiles
    Default value: FALSE
    Merge small files at the end of a map-reduce job.

    hive.metastore.ds.retry.interval
    Default value: 1000
    The number of miliseconds between metastore retry attempts

    hive.metastore.client.connect.retry.delay
    Default value:1
    Number of seconds for the client to wait between consecutive connection attempts

    hive.script.auto.progress
    Default value:FALSE
    Whether Hive Tranform/Map/Reduce Clause should automatically send progress information to TaskTracker to avoid the task getting killed because of inactivity. Hive sends progress information when the script is outputting to stderr. This option removes the need of periodically producing stderr messages, but users should be cautious because this may prevent infinite loops in the scripts to be killed by TaskTracker.

    hive.multi.insert.move.tasks.share.dependencies
    Default value:FALSE
    If this is set all move tasks for tables/partitions (not directories) at the end of a multi-insert query will only begin once the dependencies for all these move tasks have been met. Advantages: If concurrency is enabled, the locks will only be released once the query has finished, so with this config enabled, the time when the table/partition is generated will be much closer to when the lock on it is released. Disadvantages: If concurrency is not enabled, with this disabled, the tables/partitions which are produced by this query and finish earlier will be available for querying much earlier. Since the locks are only released once the query finishes, this does not apply if concurrency is enabled.

    hive.optimize.ppd.storage
    Default value:TRUE
    Whether to push predicates down into storage handlers. Ignored when hive.optimize.ppd is false.

    hive.script.serde
    Default value:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
    The default serde for trasmitting input data to and reading output data from the user scripts.

    hive.archive.har.parentdir.settable
    value:TRUE(bydefault false)
    In new Hadoop versions, the parent directory must be set while creating a HAR. Because this functionality is hard to detect with just version numbers, this conf var needs to be set manually.

    hive.metastore.archive.intermediate.original
    Default value:_INTERMEDIATE_ORIGINAL
    Intermediate dir suffixes used for archiving. Not important what they are, as long as collisions are avoided

    hive.query.id
    Default value:root_20170319002424_61de6a85-fbc7-4713-bc3d-f8966a6eb94e
    unique id given when a job run.

    #3208 Reply

    Anshuman

    Useful Hive Properties:

    Description Command

    To Print Current DB in use set hive.cli.print.current.db=true; (or)
    set hiveconf:hive.cli.print.current.db=true;

    To remove current db name display in hive shell set hiveconf:hive.cli.print.current.db=false;

    Specifying Metastore location for each user set hive.metastore.warehouse.dir=/user/myname/hive/warehouse;

    System Namespace (provides read-write access to
    -Java system properties) set system:user.name; (or)
    set system:user.name=yourusername;

    env Namespace (provides read-only access to
    -environment variables) set env:HOME;

    Print Column Headers SET hive.cli.print.header=true;

    #3209 Reply

    Anshuman

    Useful Hive Properties:

    Description Command
    To Print Current DB in use set hive.cli.print.current.db=true; (or)
    set hiveconf:hive.cli.print.current.db=true;

    To remove current db name display in hive shell set hiveconf:hive.cli.print.current.db=false;

    Specifying Metastore location for each user set hive.metastore.warehouse.dir=/user/myname/hive/warehouse;
    System Namespace (provides read-write access to
    -Java system properties) set system:user.name; OR set system:user.name=yourusername;

    env Namespace (provides read-only access to
    -environment variables) set env:HOME;

    Print Column Headers SET hive.cli.print.header=true;

Viewing 5 posts - 1 through 5 (of 5 total)
Reply To: CDH 4 HIVE PROPERTIES.
Your information:




cf22

Your Name (required)

Your Email (required)

Subject

Phone No

Your Message

Cart

  • No products in the cart.