Hi, I'm

Rahul Joshi.

Distinguished Data Engineer building open lakehouse systems at scale.

Distinguished Data Engineer and Director at Capital One — an elite technical role held by fewer than 0.5% of associates. 19+ years architecting cloud-native data platforms for financial services at enterprise scale. Open source contributor to Apache Polaris and Apache Iceberg. Published researcher. IEEE Senior Member. BCS Fellow.

Download CV

About

I architect data platforms that power intelligent financial products at enterprise scale — production systems processing hundreds of petabytes across Capital One’s most critical workloads.

My current focus is the open lakehouse: Apache Iceberg, Apache Polaris, and the catalog and compute layers that make multi-engine data access reliable and governable. I contribute to both projects upstream and apply them in production.

M.Tech from IIT Kharagpur (GPA 9.11/10). Fellow of the British Computer Society. Senior Member, IEEE. Forbes Technology Council. Published in peer-reviewed journals on data-centric AI and platform architecture.

Core technical areas:

Apache Spark
Apache Iceberg
Apache Polaris
Python / Scala
AWS
Data Lakehouse
Apache Flink
Apache Kafka
Distributed Systems
Cloud-Native Architecture

Experience

Capital One
EY
IBM
Persistent Systems
NVIDIA

Distinguished Data Engineer & Director - Capital One

2022 - Present

Elite technical leadership role held by fewer than 0.5% of associates. Leading data architecture across Capital One’s most critical platforms — Enterprise Data Tech and Card Tech.

Architected the ML Zone for production-scale model training
Drove $10M+ in annual cloud cost savings
Led architecture of the enterprise-scale data lake (300+ PB) and multi-tenant cloud warehouse (50+ PB)
Represents Card Tech in Capital One’s Distinguished Engineer community
Active contributor to Apache Polaris and Apache Iceberg upstream

Senior Manager, Data & Analytics — Financial Services - EY

2019 - 2022

Led high-impact data architecture engagements across Banking, Capital Markets, Wealth Management, and Insurance.

Architected cloud-based enterprise data lakes and analytics hubs on AWS and Snowflake for major U.S. banks
Designed a graph-based resiliency framework for a global investment bank using Neo4j
Led architecture reviews presented at board level

Sr. Managing Consultant, Big Data & Analytics - IBM

2015 - 2019

Architected hybrid cloud data lakes integrating 100+ source systems and 5B+ daily records for a major U.S. insurer
Designed a Kafka-based transport platform handling 4B+ daily events
Built Spark/MapReduce pipelines supporting 10,000+ daily workloads
Deployed predictive models that automated 1.5M+ IVR actions and reduced call volumes

Software Engineer → Architect - Persistent Systems

2007 - 2015

Led architecture of next-gen hybrid data lake platforms in partnership with IBM
Ported Apache Hadoop to Windows, enabling the launch of Microsoft HDInsight on Azure
Engineered Microsoft’s ODBC Driver for Apache Hive
Built bi-directional Sqoop connectors for SQL Server / PDW integration

Software QA Engineer Trainee - NVIDIA

2006 - 2007

Quality assurance and validation of GPU device drivers and media decoders. DirectX and OpenGL compliance testing.

Education

2009 - 2011

M.Tech, Computer Science & Engineering

Indian Institute of Technology (IIT) Kharagpur

GPA: 9.11 / 10

Research in information retrieval, machine learning, and rank aggregation. Published at Springer PReMI 2011.

Rank Aggregation Paper

2002 - 2006

B.E., Computer Engineering

Pune University

GPA: First Class with Distinction

Best Outgoing Student, Computer Engineering Department.

Projects

Java Iceberg Catalog Concurrency

Apache Polaris

7 PRs merged into the REST-compatible open catalog for Apache Iceberg. Contributions span thread safety in the event buffer, storage location validation, and credential rotation correctness.

View PRs

Java REST API Lakehouse

Apache Iceberg

IRC Events endpoint implementation covering all 13 catalog operation types with full round-trip serialization, pagination support, and forward-compatibility parsing.

View PR

Spark Scala Search

IndexTables

Open table format for Apache Spark enabling fast retrieval and full-text search across large-scale datasets. Documentation fix replacing deprecated configuration keys.

View PR

Apps Script JavaScript Automation

Gmail Inbox Sweeper

Automatically discovers spam senders, bulk-trashes by domain, and runs daily cleanup on autopilot. 56-test Jest suite with mocked Google APIs and GitHub Actions CI.

View Repo

Apache Polaris

Contributor to Apache Polaris — the open source, REST-compatible catalog for Apache Iceberg. 7 PRs …

Know more

Apache Iceberg

Contributor to Apache Iceberg — the open table format for large-scale analytic datasets. Working on …

Know more

IndexTables

Contributor to IndexTables — an open table format for Apache Spark enabling fast retrieval and …

Know more

Gmail Inbox Sweeper

Google Apps Script that automatically discovers spam senders, bulk-trashes by domain, and schedules …

Know more

Achievements

Fellow, British Computer Society (BCS)

Elected Fellow of the British Computer Society — the UK's chartered body for IT.

Senior Member, IEEE

IEEE Senior Member — recognizing significant performance over a period of at least ten years.

Forbes Technology Council

Invitation-only community for senior technology executives and CTOs.

Fellow, IETE

Fellow of the Institution of Electronics and Telecommunication Engineers.

Full Member, Sigma Xi

The Scientific Research Honor Society. Recognizes scientific achievement and encourages research.

Judge, Globee Awards for Technology (2025)

Peer reviewer and judge for the Globee Technology Awards.

Peer Reviewer, IEEE Conferences

Active peer reviewer for IEEE conference submissions in data engineering and AI.

Service Excellence Award, IBM (2016)

Recognized for exceptional client delivery and technical leadership.

Get in Touch

Open to conversations on data platform architecture, open lakehouse systems, and OSS collaboration.

Get in touch