As enterprises race to deploy artificial intelligence at scale, one problem is becoming increasingly expensive: preparing enormous amounts of data before AI models and analytics applications can actually use it.
Cloudera is now working with NVIDIA to address that bottleneck.
The data and AI platform company has announced native NVIDIA GPU acceleration for Apache Spark 4.1 in Cloudera Data Engineering, using NVIDIA's CUDA-X-powered cuDF technology.
The integration is designed to help organisations process large Spark workloads faster while continuing to use their existing PySpark and SQL applications.
In simple terms, companies will not have to rewrite their Spark applications just to take advantage of GPU computing.
What Has Cloudera Announced?
Cloudera Data Engineering will support the NVIDIA cuDF plugin for Apache Spark, bringing GPU acceleration directly into Spark data-processing workloads.
The capability will also form part of Cloudera Anywhere Cloud, the company's hybrid data and AI platform announced on August 20.
According to Cloudera, the integration can deliver up to four times faster workload performance on NVIDIA GPUs compared with traditional CPU-based infrastructure, depending on the workload.
For enterprises running large ETL, analytics and AI data-preparation jobs, shorter processing times could also translate into lower infrastructure costs because cloud compute resources need to remain active for less time.
Why GPU-Accelerated Spark Matters for AI
Apache Spark sits behind many of the data pipelines used by large enterprises.
Before businesses can train AI systems, run analytics or create machine-learning applications, raw information often needs to be cleaned, transformed, combined and prepared.
For organisations dealing with very large datasets, those Spark jobs can run for hours.
That creates two problems.
The first is speed. AI and analytics teams have to wait longer before usable data becomes available.
The second is cost. Longer-running cloud workloads consume additional computing resources and increase infrastructure bills.
GPU acceleration attempts to solve both by moving supported data-processing operations from traditional CPUs to GPUs, which are designed to perform large numbers of calculations in parallel.
NVIDIA's growing role in AI workloads
No PySpark or SQL Rewrite Required
One of the more important parts of the Cloudera NVIDIA Apache Spark integration is its focus on existing enterprise applications.
Rather than asking data engineering teams to rebuild pipelines around an entirely different framework, the companies say organisations will be able to accelerate compatible Spark jobs without changing their existing PySpark or SQL code.
Cloudera also plans to handle GPU deployment within the platform, reducing the need for teams to manually configure GPU drivers for these workloads.
The integration is expected to provide:
- Zero-code GPU acceleration for Apache Spark 4.1
- Faster ETL and AI data-preparation workloads
- Shorter cloud compute runtimes
- Built-in GPU deployment
- Enterprise security and data governance
- Support across hybrid and multi-cloud environments
Hybrid Cloud Is a Key Part of the Strategy
The partnership is not only about raw processing speed.
Cloudera is positioning the capability around enterprises whose data is spread across different infrastructure environments.
The company says accelerated Spark workloads will be supported across public cloud, private cloud, sovereign cloud and on-premises environments, with governance provided through its Unified Data Fabric.
That could be particularly important for banks, governments, healthcare organisations and other regulated enterprises that cannot simply move every dataset into a single public cloud.
Instead, businesses could potentially run accelerated data workloads closer to where their information already resides.
NVIDIA Announces RTX Spark AI Chip for Windows Laptops and Personal AI PCs
AI Infrastructure Costs Are Becoming a Bigger Concern
The announcement arrives as enterprises confront the rising infrastructure demands associated with AI.
Cloudera's recent Great AI Re-Architecture research found that 84% of respondents said AI workloads had increased their infrastructure costs.
The research also suggested that enterprises are reconsidering how and where AI workloads operate as they balance performance, governance and spending.
Faster data processing will not eliminate the wider cost of running AI infrastructure, but reducing the amount of time required to prepare large datasets could help organisations use their computing resources more efficiently.
Data Preparation Is Becoming an AI Bottleneck
The AI industry often focuses on increasingly powerful models and GPUs used to train or run them. But enterprise AI depends just as heavily on the data feeding those systems.
Leo Brunnick, Chief Product Officer at Cloudera, said many organisations are being constrained not by AI models themselves but by the time required to transform raw information into trusted and usable data.
By accelerating Spark workloads within Cloudera Data Engineering, the company wants organisations to move more quickly from data preparation to analytics and AI while retaining governance and security controls.
NVIDIA also sees the ability to accelerate existing enterprise workflows as an important part of broader AI adoption.
Pat Lee, Vice President of Strategic Enterprise Partnerships at NVIDIA, said integrating NVIDIA AI infrastructure and CUDA-X libraries into Cloudera Data Engineering could help businesses accelerate Spark pipelines without requiring changes to existing PySpark or SQL applications.
What Is NVIDIA cuDF?
NVIDIA cuDF is a GPU-accelerated data-processing library built on the company's CUDA platform.
It allows supported dataframe and data-processing operations to run on NVIDIA GPUs instead of relying entirely on CPUs.
For Apache Spark, NVIDIA provides a cuDF-based acceleration plugin that can move compatible Spark SQL and DataFrame operations onto GPUs while maintaining the familiar Spark environment.
That means data teams can potentially gain GPU performance without replacing the tools and interfaces they already use.
When Will It Be Available?
Cloudera says GPU acceleration for Apache Spark will be available through Cloudera Data Engineering as part of Cloudera Anywhere Cloud.
The capability was announced alongside Cloudera Anywhere Cloud at EVOLVE Singapore on August 20, 2026.
The companies are also expected to showcase related technology and demonstrations at NVIDIA GTC Berlin and Cloudera EVOLVE New York later in 2026.
Also Read | China's Peking University Builds Neuromorphic Brain Chip; 478x Faster Than Nvidia A100
Why This Development Matters
Generative AI may be attracting most of the attention, but enterprises cannot scale AI effectively without fast and reliable data infrastructure.
That makes technologies improving the less-visible data preparation layer increasingly important.
If Cloudera's claimed performance improvements translate consistently to enterprise workloads, GPU-accelerated Spark could help organisations shorten data-processing cycles while making better use of expensive computing infrastructure.
Perhaps more importantly, Cloudera and NVIDIA are trying to deliver those gains without forcing businesses to rebuild the Spark pipelines they already depend on.
For enterprises facing growing AI infrastructure bills, that could make GPU acceleration easier to adopt — and turn faster data engineering into another important part of the AI infrastructure race.