Migrating from Cloudera to Open-Source Hadoop: A Technical Guide

Moving a Cloudera cluster to open-source Hadoop is mostly a management plane problem. The components that run the workloads (HDFS, YARN, Hive, Spark, HBase, Kafka, and the rest) are Apache projects in both cases. What is specific to Cloudera is the layer around them: Cloudera Manager renders every configuration file, starts every daemon, holds the Kerberos identities and TLS material, schedules replication, and exposes the API that operations automation depends on.

This guide describes, at the level of directories, configuration keys, and identities, what changes when that layer is replaced, and how Bifrost handles each part in an in-place Classic migration. The target is the Ilum Bigtop distribution, managed by Apache Ambari (Ilum build), operated directly or through Ilum Cluster Ops. The hosts, the data, and the identities of the cluster stay the same.

What Cloudera Manager Does on a Cluster

Responsibility	On Cloudera CDP and CDH	After the migration
Software distribution	Parcels, unpacked under /opt/cloudera/parcels/ and activated by symlink	RPM packages of the Ilum Bigtop distribution, installed per version under /usr/bigtop/
Process supervision	The Cloudera Manager agent with its own supervisord launches every role	The Ambari agent, a systemd service, starts each daemon as its service user and restarts it automatically
Configuration rendering	Every launch renders configuration files and keytabs into a generated process directory	Static configuration in /etc/<component>/conf, written by Ambari from the desired configuration
Kerberos	Cloudera Manager can create principals and generate keytabs	The existing keytabs are placed under /etc/security/keytabs/; no KDC administrator account is used
TLS	Auto-TLS with the Cloudera Manager CA, or corporate PKI	The existing keystores and truststores are reused under /etc/security/serverKeys/
Monitoring and alerts	Service Monitor, Host Monitor, and the alert publisher	Ambari alerts, Prometheus, Grafana, and Loki
Replication	Replication Manager	Replication schedules in Apache Ambari (Ilum build)
Queue management	YARN Queue Manager	YARN Queue Manager view in Apache Ambari (Ilum build)
API	Cloudera Manager REST API	Ambari REST API, with an optional Cloudera Manager API compatibility layer
Console access	Cloudera Manager users and LDAP login	Ambari LDAP login with group-to-role mapping, or Ilum Cluster Ops

What Happens to Cloudera Manager

• Stopping. In phase 4, the services are stopped through the Cloudera Manager API. The cloudera-scm-agent, cloudera-scm-supervisordet cloudera-scm-server units are then stopped, disabled, and masked on every host. Their previous state is recorded for rollback, and the run stops if any installed unit is left unmasked.
• Packages and database. The Cloudera Manager packages and the Cloudera Manager database are not modified, so that a rollback can restart Cloudera Manager against its own database.
• Process directories. The generated process directories under /var/run/cloudera-scm-agent/process/ live on tmpfs and disappear at the next stop or reboot. Phase 3 archives them as a root-only rollback and evidence asset. Every configuration value that points into them is re-pointed to a stable path.
• Parcels. Phase 5 removes only the activation symlinks. The parcel content is kept as a rollback asset until finalize.
• Leftovers. A parcel daemon or a process still supervised by Cloudera Manager stops the takeover. A write fence keeps Ambari from writing into the conf.cloudera.* directories, and validate-post reports every Cloudera Manager unit that is not masked.

Package and Directory Layout

Item	Cloudera CDP	Ilum Bigtop distribution
Component installation	/opt/cloudera/parcels/CDH/lib/<component>	/usr/bigtop/<version>/usr/lib/<component>
Active version	Parcel activation symlink	/usr/bigtop/current/<component>, set with distro-select
Daemon configuration	/var/run/cloudera-scm-agent/process/<id>-<service>-<ROLE>/	/etc/<component>/conf
Client configuration	/etc/<component>/conf, an alternatives link to conf.cloudera.<service>	/etc/<component>/conf, an alternatives link to the configuration Ambari writes
Keytabs	Rendered into the process directory at each launch	/etc/security/keytabs/
TLS material	Auto-TLS agent certificate directory or corporate paths	/etc/security/serverKeys/
Process supervision	cloudera-scm-agent et cloudera-scm-supervisord	ambari-agent

Packages of several versions can be installed side by side; distro-select switches the active version for every component at once.

Configuration Translation

Reading the Effective Configuration

Discovery reads the full configuration view of every service, role, role configuration group, and host through the Cloudera Manager API, together with the client configuration bundles, the HDFS nameservices, the replication peers and schedules, and the external accounts. Over SSH, it also captures the files the daemons actually run with: the gateway configuration in /etc/hadoop/conf, hive-site.xml from the HiveServer2 process directory, and capacity-scheduler.xml from the ResourceManager process directory.

When the Cloudera Manager field and the rendered file disagree, the rendered file wins: the Cloudera Manager field describes the intent, the rendered file describes what the daemon really ran with.

Translating Properties

Each property passes through the same ordered rules:

1. Properties whose names indicate a secret are replaced by a vault reference.
2. Cloudera-internal settings (com.cloudera.*, Navigator settings, health thresholds, and similar) are filtered out and recorded in the review report.
3. A property that has no open-source equivalent is dropped when it holds the Cloudera default, and placed in the review queue when it was set by the cluster owner.
4. Every other property is routed to an Ambari configuration type by its name. A property without a route is placed in the review queue.

Property prefix	Ambari configuration type
dfs.	hdfs-site
yarn.scheduler.capacity.	capacity-scheduler
yarn.app.mapreduce.	mapred-site
hbase.	hbase-site
spark.	spark-defaults
tez.	tez-site
atlas.	application-properties

Cloudera Manager settings that have no direct property name are translated to the properties they control:

Cloudera Manager setting	Résultat
hdfs_hadoop_ssl_enabled set to true	dfs.http.policy se mettre à HTTPS_ONLY
yarn_service_lce_always set to true	yarn.nodemanager.container-executor.class set to the LinuxContainerExecutor classe
dfs_encrypt_data_transfer_algorithm with an AES cipher suite	Written to dfs.encrypt.data.transfer.cipher.suites
hiveserver2_webui_bind_wildcard set to true	hive.server2.webui.host se mettre à 0.0.0.0
hbase_cell_acl_authorization	Carried, inverted, into hbase.security.access.early_out

Ports that Cloudera Manager keeps at the role configuration group level are carried as well, for example the HBase Thrift port 9090 and the Spark history server ports 18088 and 18089. After the cutover, every port a source role listened on is checked on the same host.

Safety Valves

Advanced configuration snippets (safety valves) are expanded into individual properties of the configuration type they target:

<!-- HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml -->
<property>
  <nom>dfs.namenode.handler.count</nom>
  <valeur>200</valeur>
  <final>vrai</final>
</property>

The snippet above is carried as dfs.namenode.handler.count with the value 200 dans hdfs-site, and the final flag is kept.

• Environment snippets (*_env_safety_valve) are parsed as key=value lines. Dedicated handlers carry the Spark Python interpreter, the Hue settings, and the KMS JCE settings.
• Role-level snippets for the NameNode, DataNode, JournalNode, HBase Master, and RegionServer are carried cluster-wide, or as a configuration group when another layer sets the same key differently.
• A snippet that cannot be parsed unambiguously stops the plan; it is never read as empty.

Role Configuration Groups

Role configuration groups are not copied one to one. Their values are folded into the cluster-wide Ambari configuration types. An Ambari configuration group, named bifrost-<service>-<host>, is created only where the same role runs with different values on different hosts and the difference survives the cutover. Values that exist only because of Cloudera Manager, such as process directory paths and Cloudera Manager macros, never become configuration groups. Plan validation reports every value that was folded.

Guards on Carried Values

• A carried value that still points into the Cloudera installation (/opt/cloudera, PARCELS_ROOT, /var/run/cloudera, ou un conf.cloudera directory) is refused before the cutover.
• After the takeover, validate-post checks every Ambari configuration type for references to the old installation tree.
• A carried key can be dropped only with a per-key acknowledgement that records who dropped it, why, and when.

Data and Metadata

Data never moves. Before the package swap, the data directories of the NameNode, DataNodes, JournalNodes, Kafka brokers, Solr, Oozie, ZooKeeper, and the KMS are recorded by inode and modification time. Any change after the swap stops the migration and triggers rollback, and so does a Cloudera package that owns a data directory.

Component	What happens
HDFS	The on-disk layout version is compared with the target software. On the version-aligned target they are equal and no NameNode metadata is touched. When they differ, the active NameNode runs the layout upgrade and the standby bootstraps from it.
Ruche Metastore	The same metastore database is re-pointed. schematool -validate is a gate before the metastore starts. On CDH 6 sources, the schema is upgraded to the Hive 3 schema.
HBase	The balancer is turned off, tables are flushed, and HBase is shut down cleanly before the swap. The data stays on HDFS, and the balancer is turned back on after the takeover. The Phoenix indexing WAL codec is carried, and the Phoenix library is installed wherever it is configured.
ZooKeeper	The data directory, with every application znode, is kept and never reinitialized. A snapshot is taken for rollback. zoo.cfg tuning and security settings are carried.
Kafka	Log directories, including __consumer_offsets, stay in place. Offsets, topic overrides, ACLs, and the topic inventory are recorded for parity checks. ZooKeeper and KRaft modes are detected, and broker IDs and listeners are kept.
Solr	Cores stay in place and their replicas are activated under the new service. The Atlas index is never rebuilt before finalize.
Atlas	The graph tables stay in HBase. HBase snapshots are taken as a recovery seed and removed at finalize.
Oozie	The same database is re-pointed. The new share library is installed next to the existing one, which is kept until finalize, and customer jars in it are checked for a counterpart.
Hue	The database is re-pointed, and the Hue secret key is carried so that user sessions survive.
Ranger and Ranger KMS	The databases are re-pointed, and policies are exported before the cutover. Finalize refuses until every encryption zone key is proven decryptable on the target.

Identities and Certificates

Kerberos

• Keytabs are copied, never regenerated. Phase 3 stages the keytabs from the Cloudera Manager process directories. The takeover places them under /etc/security/keytabs/ with the file names the stack expects, such as nn.service.keytab et spnego.service.keytab. Shared keytabs are split with ktutil, which keeps the key material and the key version number byte for byte. Existing files are moved aside, never overwritten.
• Key versions are recorded. Discovery records the principal, host, and key version number of every keytab, so that a change is detected.
• Ambari does not manage identities. Ambari Kerberos is enabled in a mode that asks the KDC or Active Directory for nothing:

kerberos-env:
  kdc_type: none
  manage_identities: « faux »
  manage_auth_to_local: « faux »
  install_packages: « faux »
  create_ambari_principal: « faux »
krb5-conf:
  manage_krb5_conf: « faux »

Principals, key versions, auth_to_local rules, and /etc/krb5.conf stay as they are. Only service identities are carried; the principals Cloudera Manager used for its own monitoring are left out. The HTTP authentication signer secret, which Cloudera Manager kept per role, is replaced by one cluster-wide /etc/security/http_secret.

TLS

• Certificates are reused. The estate's keystores and truststores, including those issued by Cloudera Manager Auto-TLS, are copied to /etc/security/serverKeys/. Configuration values that pointed at the Cloudera Manager agent certificate directory or the {{CM_AUTO_TLS}} macro are re-pointed there.
• Trust is checked. Whether each JDK in use trusts the CA of the estate is checked and reported.
• Renewal. Certificates are renewed later with bifrost classic renew-certificates from PEM, JKS, or PKCS12 material issued by the corporate PKI, or with the certificate management of Apache Ambari (Ilum build). See Day-2 operations.
• Ranger plugins. The truststore, credential file, and audit keytab paths of every Ranger plugin are moved off the process directories.

Clients and Applications

Item	After the migration
Hostnames and service ports	Unchanged. Every port a source role listened on is checked on the same host after the cutover.
Principals and auth_to_local rules	Unchanged.
HDFS nameservice, ZooKeeper quorum, fs.defaultFS	Taken from discovery, not from stack defaults.
Operating system users, groups, UIDs, GIDs	Unchanged. Ambari never creates or modifies them.
Gateway hosts	Cloudera Manager gateway roles become Ambari client components. bifrost classic refresh-clients rewrites /etc/<component>/conf from the desired configuration.
Scripts that call parcel paths	Le bin et lib entries of every active parcel are re-linked into the new installation, so /opt/cloudera/parcels/<product>/bin/<tool> keeps resolving.
Hive command line	Le ruche command uses Beeline, as on CDP.
Automation on the Cloudera Manager API	Re-pointed to the Ambari REST API, or served by the Cloudera Manager API compatibility layer of Apache Ambari (Ilum build). With ambari_cm_endpoint_takeover: true, Ambari also answers on the Cloudera Manager ports 7180 and 7183.

Spark commands keep their meaning:

Command	After the migration
spark3-submit, spark3-shell, pyspark3	Spark 3, where the source ran the Cloudera Spark 3 parcel
étincelle-soumission, spark-shell, Pyspark	Spark 2, where the source ran the Spark 2 included in CDP
spark2-submit	Spark 2, as an additional name

Common Pitfalls and How They Are Handled

• Cloudera paths in configuration. Refused before the cutover and checked again by validate-post. Parcel paths used by scripts keep resolving through the re-linked client entries.
• Classes missing from the target build. bifrost classic scan-class-coverage lists every class referenced in configuration that the target jars do not contain, before the window. HBase coprocessor classes are included. The takeover stops on a missing class instead of starting a daemon that fails later.
• Customer jars. Hive hooks, SerDes, and UDF jars are carried to a Bifrost-managed directory on HIVE_AUX_JARS_PATH. Kafka plugins and the Oozie libext directory are carried the same way.
• Queues edited in Queue Manager. When queue definitions live in the ZooKeeper mutation store, the live store is compared with the captured capacity-scheduler.xml, and any difference stops the run before the services are stopped. Queues are then managed in the YARN Queue Manager view.
• Cloudera lineage listeners. Navigator and other Cloudera lineage listeners are removed from the Spark configuration; the Atlas Spark hook is kept wherever Atlas runs.
• Process limits. Every daemon gets at least the nproc et nofile limits it had on Cloudera, measured from /proc/<pid>/limits, instead of the lower stack defaults.
• Log locations. Log directories are carried. After the cutover, the open files of every daemon are checked to confirm that nothing writes into the old installation tree.
• Port collisions. Ambari serves on port 8085 by default, because port 8080 is used by the ZooKeeper admin server. Ports that differ between Cloudera and the stack defaults, such as the HBase Thrift port, are carried.
• Dependencies between services. Plan validation refuses a plan in which a migrated service still points at a service that is not part of the migration.
• Java. Components run on JDK 11 by default. The package swap stops if no JDK 11 is available to keep /usr/bin/java working.

Replication

Replication Manager schedules move with the cluster:

1. Capture. The replication peers, the schedules of every service, the snapshot policies, and the external accounts are captured at discovery and again in phase 3.
2. Import. The takeover registers the peers and accounts in Apache Ambari (Ilum build) and creates the HDFS schedules (from a peer cluster, between clusters of the same Cloudera Manager, and to Amazon S3) and the Hive schedules (from a peer cluster). Every schedule is created paused. The schedules keep the Cloudera field names, so no schedule is rewritten by hand.
3. Decisions. Where a schedule moved deleted files to the trash, the owner chooses per schedule whether files are deleted or kept.
4. Activation. After validation, bifrost classic replication-activate --execute runs a dry run of each schedule, resumes only the schedules whose dry run succeeded, and confirms that Ambari has scheduled the next run. Schedules that were paused in Cloudera Manager stay paused, and one-time schedules are not activated.

Peer credentials are kept in a credential store on the Ambari server; cloud credentials are kept in a credential store in HDFS.

Apache Ambari (Ilum Build)

Apache Ambari (Ilum build) extends upstream Apache Ambari with the operations that Cloudera Manager users rely on. It manages the services added by the Ilum Bigtop distribution, such as Impala, Atlas, Hue, Oozie, and Spark 2, as first-class services.

Replication Manager

• Un Replication menu with replication schedules, snapshot policies, and replication peers.
• HDFS replication between clusters, including pulling from a peer managed by Ambari or by Cloudera Manager, with high availability, cross-realm Kerberos, and encryption zones.
• HDFS replication to cloud object storage: Amazon S3 and Azure Data Lake Storage.
• Hive replication of tables and data from a peer, with functions reloaded on every HiveServer2.
• Scheduling by cron expression or by interval, with pause and resume, run now, dry run, abort, and a run history with the copy counters of each run.
• Snapshot policies for HDFS and HBase, with daily, weekly, and monthly retention.
• Alerting on failed and aborted runs.
• Safe copies. The copy engine supports Cloudera-style path rebasing and closes upstream cases in which the delete option removes data unexpectedly.

Component Operations

Service	Operations
HDFS	Roll edits, enter and leave safe mode, save namespace, rebalance, decommission, bootstrap the standby NameNode
FIL	Reconfigure the ResourceManager, refresh queues, format the state store, clean NodeManager recovery and local directories
HBase	Enable and disable the balancer, create the Ranger repository
Ruche	Validate and upgrade the metastore schema
Any role	Thread dump, heap histogram, heap dump, and open files, from the host component menu

Workloads are managed from the service pages: the YARN Applications tab lists and kills applications, and the Impala Queries tab lists and cancels queries. Diagnostic bundles can be collected for support.

Restarts and Availability

• Rolling restart with batches, health gates, and a failure budget. High-availability pairs are never restarted in the same batch, and the HDFS edit log is rolled after each JournalNode batch.
• Restart review that shows the reason for every pending restart, and refresh instead of restart where a configuration change can be applied live.
• Automatic actions that restart a component when an alert fires, with a rate limit.
• Role migration between hosts and a graceful stop of all services on a host.
• High availability for the NameNode, ResourceManager, Impala statestore and catalog, Hive Metastore, Oozie, Atlas, Hue, Livy, and Ranger UserSync and TagSync.

Monitoring and Logs

• Prométhée as the metrics source, built in or external, with remote_write to another store.
• Tableaux de bord Grafana for the cluster and its services, and metrics tabs for Hive, ZooKeeper, Impala, Oozie, Ranger, Ranger KMS, and Spark. HDFS, YARN, and Kafka show the charts Cloudera Manager users know.
• Log search across the cluster with Loki.
• Alert definitions for TLS certificate expiry, replication failures, agent recovery, the Ranger audit spool, HBase regions in transition and compaction queues, and more, with heatmaps colored by alert thresholds.
• Alert targets for email, SNMP, and webhooks.

Security and Governance

• Certificate management: generate a CA, issue, rotate, and place certificates; certificates the cluster already serves are recorded and tracked; expiry raises an alert.
• Directory access: scheduled LDAP synchronization, group-to-role mapping, and custom roles.
• Governance page that shows whether Ranger enforcement, audit delivery, lineage, and tag synchronization work.
• Audit trail of management operations, with a dedicated permission to view it.

Automatisation

• An OpenAPI specification with an API explorer, and a Python SDK.
• Un batch API that applies several operations in one call.
• Un Cloudera Manager API compatibility layer for reads, configuration changes, and start, stop, and restart commands, so that existing automation can be moved step by step.

Next Steps

• Follow the procedure in Classic migration.
• Read how the takeover works in Management plane, and how identities and keys are carried in Security and encryption.
• Check components and platforms in the Support matrix.
• To discuss a migration, contact [email protected].