Micro-batching workaround

Problem. A platform enforces a hard query-time cap (e.g. 10 minutes in AEP — constraint.aep-adhoc-query-timeout-10min) that prevents complex JOINs across large fact tables from completing.

Workaround. Break the query into specific time windows or partitions that each complete within the cap. Merge results in a downstream step. Often paired with CTAS (Create Table As Select) to materialize intermediate results.

Cost.

• Operational complexity — what was one query is now an orchestration with retries, partial-failure handling, and merge logic.
• Storage clutter — CTAS tables proliferate, fragmenting the data lake.
• Stifled exploration — analysts and data scientists who would normally probe the data interactively avoid complex queries entirely (source.packaged-vs-composable-md).

When it's the right answer. When the platform constraint is genuinely binding and the cost of moving the workload off-platform exceeds the operational tax of micro-batching.

When it's an anti-pattern. When teams adopt micro-batching as the default rather than escalating the underlying constraint. The workaround makes the limit invisible, which makes the case for migration harder to articulate to leadership.