SPIKE fix: stop corrupting group header colspans

Reported symptom: with split_rollup_mode on, columns looked shuffled and
some month columns had no year header above them. That was this spike, not
Perspective.

Two mistakes, either enough on its own:

- The original span was stashed on the cell as `_pfFullSpan` the first time
  it was seen. regular-table recycles header cells between draws, so the
  stashed value reappeared on an unrelated group and truncated it.
- The work was guarded on `colSpan > 1`, so once a cell had been set to 1 it
  could never grow back.

Both fired even with nothing collapsed, which is why merely turning rollup
on was enough to break the header.

Now the span is derived from the column paths every draw, and colSpan is
only touched while something is actually collapsed — with nothing hidden,
regular-table's own merge of adjacent equal header values is the right
answer and is left alone.

Measured on Reason x (Year, Month), 18 leaf columns, across first draw,
sort, sort-cleared and horizontal scroll:

  before fix   2025 spans 3,  group row covers  8/18  (10 orphaned)
  after fix    2025 spans 13, group row covers 18/18
  no spike     2025 spans 13, group row covers 18/18

Collapse behaviour is unchanged: 8 -> 6 -> 3 -> 5 cells across collapse
2025, collapse 2026, expand 2025, with header and body width-aligned
throughout and the state surviving a sort.

Not fixed, because it is not a bug: rollup emits totals BEFORE their
members, so the grand total sits far left and each year's subtotal precedes
its own months. SplitRollupMode documents this as "totals before" order and
offers only flat|rollup, so it cannot be moved to Excel's convention.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LoxNi8cFsQLPUSw3obb5NH

CLAUDE.md

@@ -95,6 +95,43 @@ Turning a region back into slices re-derives, per cell, the same filters Perspec
 
 ---
 
+## Column hierarchy (collapse / expand)
+
+The two pivot axes collapse by completely different mechanisms, and the asymmetry is a
+Perspective constraint, not a choice:
+
+- **Rows.** The `GROUP BY ROLLUP` view holds every level at once; `view.set_depth()` — which
+  lives on the view, not the config — hides the deeper ones. That is what the `EXPAND 0 1 2 3`
+  buttons drive, via `applyDepth()`.
+- **Columns.** There is no equivalent. `expand()` / `collapse()` take a **row index**,
+  `ViewConfig` has `group_by_depth` but no `split_by_depth`, and `split_rollup_mode`
+  (`'flat' | 'rollup'`) only chooses whether subtotal column groups are *emitted* — it is a
+  view shape, not an interaction. So `applySplitDepth(n)` collapses by restoring a
+  **truncated `split_by`**, which rebuilds the view.
+
+Three things follow from the rebuild, and each is handled:
+
+1. The full hierarchy has to be remembered separately — once collapsed, `viewer.save()`
+   only reports the short `split_by`. `splitFullRef` / `splitFull` hold it, and it is
+   persisted into the saved layout as `split_full` so a reload while collapsed can still
+   expand back. `adoptSplit()` is the single place it is set.
+2. `perspective-config-update` fires for our own restore as well as the user rearranging
+   the pivot. `collapsingRef` distinguishes them — without it, a collapse would overwrite
+   the full hierarchy with the truncated one and the deeper levels would be unreachable.
+3. Row depth lives on the discarded view, so `applyDepth(expandDepthRef.current)` is
+   re-applied afterwards — the same wart as the refocus re-apply.
+
+The selection is cleared on every change: slices name the split_by dimensions they were
+cut from, and the highlight is keyed on grid coordinates. Neither survives a column axis
+that just changed shape.
+
+**Limitation:** this is whole-axis, not per-branch. Excel can collapse 2025 while 2026
+stays expanded; truncating `split_by` collapses every column group at that level together.
+Per-branch is not reachable — `columns` selects which *measures* appear, not individual
+split combinations.
+
+---
+
 ## Operation SQL patterns
 
 All three operations follow the same structure: insert a `pf.log` row in a CTE, then insert forecast rows referencing its id. `{{where_clause}}` is built from the slice; `{{exclude_clause}}` blocks `exclude_iters` rows.

ui/src/branchCollapse.js

@@ -0,0 +1,178 @@
+// SPIKE — per-branch column collapse without forking the datagrid plugin.
+//
+// The question this answers: Excel can collapse 2025 while 2026 stays expanded.
+// Perspective cannot, because the column axis has no expand/collapse of any kind
+// (see the "Column hierarchy" section of CLAUDE.md). But under
+// `split_rollup_mode: 'rollup'` the engine already emits the whole column tree in
+// pre-order — grand total, then each branch's subtotal immediately followed by its
+// own leaves — so every number Excel would show is already on the client. What is
+// missing is only the ability to *not show* some of them.
+//
+// So this hides the leaves of a collapsed branch in the DOM, from a style listener,
+// and shrinks the spanning header cells to match. No fork, no engine change.
+//
+// What it cannot fix, and what the spike is really measuring: the plugin still
+// fetches and lays out the hidden columns. Its horizontal virtualisation asks the
+// view for a contiguous [start_col, end_col) window and indexes everything by that
+// x, so a hidden column still costs a fetch and still occupies an index. At small
+// column counts that is invisible; the open question is where it stops being so.
+
+const ZERO_WIDTH = /^\u200b*$/
+
+// The split_by levels a column actually belongs to, dropping the measure name.
+// 'Amount' -> [],  '2025|Amount' -> ['2025'],  '2025|AI|Amount' -> ['2025','AI']
+export function levelsOf(columnPath) {
+  if (!columnPath) return []
+  return columnPath.split('|').slice(0, -1)
+}
+
+// Same, from a cell's column_header, where rolled-up levels arrive as runs of
+// zero-width spaces rather than being absent.
+export function levelsOfHeader(columnHeader, splitLen) {
+  const out = []
+  for (const level of (columnHeader || []).slice(0, splitLen)) {
+    if (level == null || ZERO_WIDTH.test(level)) break
+    out.push(level)
+  }
+  return out
+}
+
+// How many consecutive columns from x share this cell's first chy+1 levels —
+// the span the group header would have if nothing were hidden.
+//
+// This has to be derived from the column paths on every draw. The obvious
+// shortcut, stashing the original span on the cell the first time we see it, is
+// wrong: regular-table recycles header cells between draws, so the stashed value
+// reappears on an unrelated group and silently truncates it.
+function fullSpan(columnPaths, x, chy) {
+  const levels = levelsOf(columnPaths[x])
+  if (levels.length < chy + 1) return 1   // a rollup placeholder covers only itself
+  const prefix = levels.slice(0, chy + 1)
+  let n = 0
+  for (let i = x; i < columnPaths.length; i++) {
+    const l = levelsOf(columnPaths[i])
+    if (l.length < prefix.length || !prefix.every((v, k) => l[k] === v)) break
+    n++
+  }
+  return n
+}
+
+function isDescendant(levels, prefix) {
+  return levels.length > prefix.length && prefix.every((v, i) => levels[i] === v)
+}
+
+// Engine column indices to hide: the strict descendants of any collapsed branch.
+// The branch's own subtotal column survives — that is what you are collapsing to.
+export function hiddenColumns(columnPaths, collapsed) {
+  const hidden = new Set()
+  if (!collapsed.size) return hidden
+  const prefixes = [...collapsed].map(c => c.split('|'))
+  columnPaths.forEach((path, x) => {
+    if (!path) return
+    const levels = levelsOf(path)
+    if (prefixes.some(p => isDescendant(levels, p))) hidden.add(x)
+  })
+  return hidden
+}
+
+// The branch a header cell at column x, header row `chy`, would collapse: the
+// first chy+1 levels of its column path. Null when that is not a collapsible
+// group — the grand total, or the deepest level, which has no leaves to hide.
+export function branchAt(columnPaths, x, chy, splitLen) {
+  const levels = levelsOf(columnPaths[x])
+  if (levels.length < chy + 1) return null
+  if (chy + 1 >= splitLen) return null
+  return levels.slice(0, chy + 1).join('|')
+}
+
+/**
+ * Attach per-branch collapse to a datagrid plugin element.
+ * Returns { toggle, collapsed, clear, detach } — `onChange` fires after a toggle
+ * so the host can persist or re-render.
+ */
+export function attachBranchCollapse(grid, { onChange } = {}) {
+  const collapsed = new Set()
+  const rt = grid.regular_table
+
+  const splitLen = () => grid.model?._config?.split_by?.length || 0
+  const paths    = () => grid.model?._column_paths || []
+
+  function styleListener() {
+    const len = splitLen()
+    if (!len) return
+    const hidden = hiddenColumns(paths(), collapsed)
+
+    const columnPaths = paths()
+
+    for (const cell of rt.querySelectorAll('tbody td, tbody th, thead th')) {
+      const meta = rt.getMeta(cell)
+      if (!meta) continue
+
+      // A group header stays visible but must shrink to the columns still
+      // showing under it, or the header row and the body stop lining up.
+      //
+      // Only while something is actually collapsed. With nothing hidden this
+      // must not touch colSpan at all — regular-table computes it each draw by
+      // merging adjacent equal header values, and that answer is the right one.
+      if (hidden.size && meta.type === 'column_header' && meta.x != null) {
+        const chy = meta.column_header_y ?? 0
+        if (chy < len) {
+          const full = fullSpan(columnPaths, meta.x, chy)
+          let visible = 0
+          for (let i = meta.x; i < meta.x + full; i++) if (!hidden.has(i)) visible++
+          cell.colSpan = Math.max(1, visible)
+        }
+      }
+
+      const hide = meta.x != null && hidden.has(meta.x)
+      cell.style.display = hide ? 'none' : ''
+
+      // mark what can be clicked to collapse, and which way it would go
+      if (meta.type === 'column_header' && meta.x != null && !hide) {
+        const branch = branchAt(columnPaths, meta.x, meta.column_header_y ?? 0, len)
+        if (branch) {
+          cell.dataset.pfBranch = branch
+          cell.style.cursor = 'pointer'
+          cell.title = collapsed.has(branch)
+            ? `Expand ${branch}`
+            : `Collapse ${branch} to its subtotal`
+        } else {
+          delete cell.dataset.pfBranch
+          cell.style.cursor = ''
+        }
+      }
+    }
+  }
+
+  function onClick(event) {
+    const target = (event.composedPath?.()[0]) || event.target
+    const th = target?.closest?.('th[data-pf-branch]')
+    if (!th) return
+    event.stopPropagation()   // the plugin would otherwise sort or open the menu
+    event.preventDefault()
+    toggle(th.dataset.pfBranch)
+  }
+
+  function toggle(branch) {
+    if (collapsed.has(branch)) collapsed.delete(branch)
+    else collapsed.add(branch)
+    rt.draw({ preserve_width: true })?.catch?.(() => {})
+    onChange?.(new Set(collapsed))
+  }
+
+  function clear() {
+    collapsed.clear()
+    rt.draw({ preserve_width: true })?.catch?.(() => {})
+    onChange?.(new Set(collapsed))
+  }
+
+  rt.addStyleListener(styleListener)
+  rt.addEventListener('click', onClick, true)
+
+  return {
+    toggle,
+    clear,
+    collapsed: () => new Set(collapsed),
+    detach() { rt.removeEventListener('click', onClick, true) },
+  }
+}

ui/src/views/Forecast.jsx

@@ -1,5 +1,6 @@
 import { useState, useEffect, useRef } from 'react'
 import useTheme from '../theme.jsx'
+import { attachBranchCollapse } from '../branchCollapse.js'   // SPIKE
 import OperationPanel from '../components/OperationPanel.jsx'
 import BridgeView from '../components/BridgeView.jsx'
 
@@ -23,6 +24,8 @@ function cleanLayout(cfg, validCols) {
   if (c.columns)  c.columns  = c.columns.filter(col => col == null || ok(col))
   if (c.group_by) c.group_by = c.group_by.filter(ok)
   if (c.split_by) c.split_by = c.split_by.filter(ok)
+  // the uncollapsed column hierarchy travels with the layout (see applySplitDepth)
+  if (c.split_full) c.split_full = c.split_full.filter(ok)
   if (c.sort)     c.sort     = c.sort.filter(([col]) => ok(col))
   if (c.filter)   c.filter   = c.filter.filter(([col]) => ok(col))
   return c
@@ -42,6 +45,12 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
   const [saveAsName,     setSaveAsName]     = useState('')
 
   // operation panel — a selection is a LIST of slices; one entry is the common case
+  // The column hierarchy and how many of its levels are showing. Mirrored into
+  // splitFullRef for handlers registered once; held as state so the toolbar
+  // re-renders when either changes.
+  const [splitFull,      setSplitFull]      = useState([])
+  const [splitDepth,     setSplitDepth]     = useState(null)
+
   const [slices,         setSlices]         = useState([])
   const [applyMode,      setApplyMode]      = useState('prorate')  // 'prorate' | 'each'
   const [activeOp,       setActiveOp]       = useState('scale')
@@ -167,6 +176,13 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
   const tableRef       = useRef(null)
   const colMetaRef     = useRef([])
   const expandDepthRef = useRef(null)
+  // The column axis has no set_depth() — collapsing it means restoring a shorter
+  // split_by, so the full hierarchy has to be remembered separately to expand again.
+  const splitFullRef   = useRef([])
+  // set while our own restore is in flight, so the config-update listener can tell
+  // a collapse from the user rearranging split_by themselves
+  const collapsingRef  = useRef(false)
+  const branchRef      = useRef(null)   // SPIKE: per-branch column collapse
   const initIdRef      = useRef(0)
   const modifierRef    = useRef(false)
   // the datagrid plugin element, for reading cell coordinates and driving its
@@ -536,6 +552,7 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
     setLoadProgress(null)
     setSlices([])
     expandDepthRef.current = null
+    adoptSplit([], 0)
     try {
       const [dataResult, meta] = await Promise.all([
         fetch(`/api/versions/${vid}/data`).then(async r => {
@@ -616,6 +633,9 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
         const { table: _t, ...rest } = cleanLayout(JSON.parse(saved), validCols)
         const cfg = { ...rest, plugin_config: { ...(rest.plugin_config || {}), edit_mode: 'SELECT_REGION' } }
         await viewer.restore(cfg)
+        // split_full outlives split_by: a layout saved while collapsed still knows
+        // the levels it was collapsed from
+        adoptSplit(cfg.split_full?.length ? cfg.split_full : cfg.split_by, (cfg.split_by || []).length)
         if (cfg.expand_depth != null) await applyDepth(cfg.expand_depth)
       } else {
         const sourceDefault = sources.find(s => String(s.id) === String(sid))?.default_layout
@@ -633,12 +653,19 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
           }
         }
         await viewer.restore(cfg)
+        adoptSplit(cfg.split_full?.length ? cfg.split_full : cfg.split_by, (cfg.split_by || []).length)
       }
 
       // auto-persist viewer state (formatting, columns, etc.) to the last-used cache
       if (viewer._pspUpdate) viewer.removeEventListener('perspective-config-update', viewer._pspUpdate)
       viewer._pspUpdate = async () => {
         try {
+          // A split_by change that is not ours is the user rearranging the pivot, and
+          // it redefines the hierarchy. Ours is a collapse, and must not overwrite it.
+          if (!collapsingRef.current) {
+            const live = await viewer.save()
+            adoptSplit(live.split_by || [], (live.split_by || []).length)
+          }
           const cfg = await captureConfig()
           if (cfg) await persistLayout(vid, cfg)
         } catch {}
@@ -680,6 +707,14 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
       viewer.addEventListener('perspective-select', viewer._pspSelect)
 
       gridRef.current = await viewer.getPlugin()
+
+      // SPIKE: per-branch column collapse. Exposed on window so the spike can be
+      // driven without building UI for it yet.
+      try {
+        branchRef.current?.detach()
+        branchRef.current = attachBranchCollapse(gridRef.current)
+        window.__pfBranch = branchRef.current
+      } catch (err) { console.error('[branchCollapse]', err) }
       setLargeDataset(false)
 
     } catch (err) {
@@ -690,6 +725,54 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
     }
   }
 
+  // Record the column hierarchy as the uncollapsed truth. Called whenever a layout
+  // arrives or the user rearranges split_by themselves — but never for our own
+  // collapse, which would otherwise overwrite the full list with the short one.
+  function adoptSplit(full, depth) {
+    const list = Array.isArray(full) ? full : []
+    splitFullRef.current = list
+    setSplitFull(list)
+    setSplitDepth(depth == null ? list.length : Math.min(depth, list.length))
+  }
+
+  // Collapse or expand the column axis to `n` split_by levels.
+  //
+  // The row axis gets this for free: its GROUP BY ROLLUP view holds every level at
+  // once and view.set_depth() hides the deeper ones. The column axis has no
+  // equivalent — there is no split_by_depth in ViewConfig and expand()/collapse()
+  // take a row index — so collapsing means restoring a truncated split_by, which
+  // rebuilds the view. Two consequences fall out of that: the row depth has to be
+  // re-applied afterwards (it lives on the discarded view), and it is whole-axis,
+  // not per-branch — every column group collapses to the same level together.
+  async function applySplitDepth(n) {
+    const viewer = viewerRef.current
+    const full   = splitFullRef.current
+    if (!viewer || !full.length) return
+    const depth = Math.max(0, Math.min(n, full.length))
+    collapsingRef.current = true
+    try {
+      await viewer.restore({ split_by: full.slice(0, depth) })
+      setSplitDepth(depth)
+      // restore() rebuilt the view, so the row depth that lived on the old one is gone
+      if (expandDepthRef.current != null) await applyDepth(expandDepthRef.current)
+    } catch (err) {
+      console.error('[applySplitDepth]', err)
+      flash(err.message || String(err), 'error')
+      return
+    } finally {
+      collapsingRef.current = false
+    }
+    // a slice names the split_by dimensions it was cut from, and the highlight is
+    // keyed on grid coordinates — neither survives a column axis that just changed
+    setSlices([])
+    try {
+      const cfg = await captureConfig()
+      if (cfg) await persistLayout(versionId, cfg)
+    } catch (err) {
+      console.error('[applySplitDepth persist]', err)
+    }
+  }
+
   async function applyDepth(d) {
     const viewer = viewerRef.current
     if (!viewer) return
@@ -704,7 +787,7 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
     const viewer = viewerRef.current
     if (!viewer) return null
     const cfg = await viewer.save()
-    return { ...cfg, expand_depth: expandDepthRef.current }
+    return { ...cfg, expand_depth: expandDepthRef.current, split_full: splitFullRef.current }
   }
 
   async function persistLayout(vid, cfg) {
@@ -762,6 +845,7 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
     const cfg = cleanLayout(layout.config, validCols)
     cfg.plugin_config = { ...(cfg.plugin_config || {}), edit_mode: 'SELECT_REGION' }
     await viewer.restore(cfg)
+    adoptSplit(cfg.split_full?.length ? cfg.split_full : cfg.split_by, (cfg.split_by || []).length)
     if (cfg.expand_depth != null) await applyDepth(cfg.expand_depth)
     setActiveLayoutId(layout.id)
     await persistLayout(versionId, cfg)
@@ -1087,6 +1171,30 @@ export default function Forecast({ sources = [], sourceId, versions = [], versio
           ))}
         </div>
 
+        {splitFull.length > 0 && (
+          <>
+            <div className="w-px h-4 bg-gray-200 shrink-0" />
+
+            {/* Column hierarchy group — the split_by equivalent of Expand */}
+            <div className="flex items-center gap-1.5">
+              <span className="text-gray-400 uppercase tracking-wide" style={{fontSize:'10px'}}>Columns</span>
+              {Array.from({ length: splitFull.length + 1 }, (_, n) => {
+                const label = n === 0 ? 'Total' : splitFull[n - 1]
+                return (
+                  <button key={n} onClick={() => applySplitDepth(n)}
+                    title={n === 0
+                      ? 'Collapse the columns to a single total'
+                      : `Show columns down to ${splitFull.slice(0, n).join(' › ')}`}
+                    className={`border rounded px-1.5 py-0.5 transition-colors max-w-[9rem] truncate
+                      ${splitDepth === n ? 'border-blue-300 text-blue-600 bg-blue-50' : 'border-gray-200 text-gray-500 hover:border-gray-400'}`}>
+                    {label}
+                  </button>
+                )
+              })}
+            </div>
+          </>
+        )}
+
         <div className="w-px h-4 bg-gray-200 shrink-0" />
 
         {/* Data group */}