Apache Spark · Databricks · Delta Lake
Probatum helps data teams find the real bottleneck in slow, failing, or expensive Spark and Databricks pipelines — and fix it with changes your engineers understand and can maintain.
Independent practice based in São Paulo · English and Portuguese
Where an assessment starts
Evidence first. Recommendations second.
I. Problems we solve
…and nobody can say with confidence which jobs drive it, or why.
Downstream reports and teams wait while a job runs past its completion deadline.
Jobs that used to finish comfortably now run for hours.
Out-of-memory errors, lost executors, SQL timeouts — often “fixed” by a retry, until the next time.
The move is done, but runtimes and costs aren’t where the team expected.
Your engineers are capable — they need a second specialist on a hard problem.
II. Services
Most engagements begin with a focused assessment, so changes are based on evidence instead of guesses.
A focused review of the workloads that hurt most. Execution plans, Spark UI evidence, logs, configuration, and data layout — read to find what is actually slowing them down.
You receive
— Bottleneck analysis per workload
— Prioritized recommendations
— Likely impact and tradeoffs of each
— A practical implementation plan
Implement and validate the agreed improvements on a defined set of pipelines, measured against a baseline set together.
Typical areas
— Join strategies and broadcast joins
— Shuffle, partitioning, and skew
— Memory pressure and spill
— File sizes and Delta table layout
— Right-sized compute configuration
For a slow, failing, or unpredictable workload your team is stuck on. We find the cause together, and you keep a repeatable diagnostic process for the next one.
Good fit for
— Recurring or intermittent failures
— Hard-to-explain slowdowns
— A second opinion on a fix in progress
Scheduled engagement — not on-call or 24/7 incident support.
III. Experience
That combination is the point: reading what a job does at execution time, and turning it into a change an engineering team can actually ship.
Investigating and improving mission-critical Spark and Databricks workloads — including jobs that read petabyte-scale tables. Long-running jobs, shuffle and partition sizing, join strategy, UDF memory issues, SQL timeouts, JDBC, Unity Catalog access, and Delta Lake layout.
Building and migrating pipelines for large banking platforms: batch frameworks across DB2, Oracle, and Delta Lake; Medallion and Data Vault models for risk and regulatory reporting; Delta Live Tables; and the migration of legacy Cloudera pipelines to Azure Databricks.
Selected results from in-house roles
Outcomes of team initiatives from earlier in-house roles in banking, as measured internally. They describe past work, not consulting results, and are not a forecast for any client. No customer or employer confidential information is shared.
IV. How I work
The workload, who depends on it, and what the problem costs the business.
Agree on scope and the success metric, and collect the technical evidence.
Find the bottlenecks and rank changes by impact, effort, and risk.
Implement the agreed changes, or guide your team through them.
Measure against the baseline and document everything for your team.
No number is quoted before the workload has been reviewed. After the assessment, you’ll know what’s realistic and why.
V. About
Probatum is Latin for “tested, proven” — the standard every recommendation here has to meet.
Probatum is a one-person practice. I’m a data engineer in São Paulo who has spent his career on Spark pipelines — first building and migrating them for one of Brazil’s largest banks, now helping teams investigate and fix them in production. I like problems where the answer is in the execution plan, not in a bigger cluster.
You work directly with the person doing the analysis, in English or Portuguese.
Current certifications
Databricks Certified Associate Developer for Apache Spark
Databricks Certified Data Engineer Associate
Databricks Certified Data Analyst Associate
Microsoft Azure Data Fundamentals · Azure Fundamentals
Credential verification available on request.
Education
B.Sc. Computer Engineering
MBA, Data Engineering
Probatum is independent. It is not affiliated with, endorsed by, or a partner of Databricks, and engagements do not include vendor support, privileged access, or any employer or customer confidential information.
VI. FAQ
Apache Spark (PySpark and Spark SQL) and Databricks, including Delta Lake and Databricks SQL. Most platform experience is on Azure. If you run Spark elsewhere, describe your setup and you’ll get an honest answer on fit.
Yes — that’s the default. I work with your engineers, explain the reasoning behind each change, and leave documentation so the team owns the result.
A short description of the workload, what’s going wrong, and why it matters. Useful evidence includes Spark UI or event logs, query plans, job and cluster configuration, and table sizes. Exactly what is shared, and how, is agreed during scoping.
Against a baseline agreed before anything changes — runtime, compute cost per run, failure frequency, throughput, or completion within your processing window, depending on the problem.
Both. An assessment ends with prioritized recommendations and a plan; an optimization sprint implements and validates them. Choose either, or both in sequence.
VII. Contact
Send a short note: what the pipeline does, what’s going wrong, and what it’s costing you. You’ll get a reply on whether Probatum can help and a suggested first step.