fix: support melting empty DataFrames without crashing#2509
fix: support melting empty DataFrames without crashing#2509
Conversation
| assert table.labels | ||
| assert table.labels["test"] == "labels" | ||
|
|
||
|
|
There was a problem hiding this comment.
nit: move test_dataframe_melt_multiindex to tests/system/small/test_multiindex.py file, which has tests related to multi index.
bigframes/core/blocks.py
Outdated
| rows.append(row) | ||
|
|
||
| return core.ArrayValue.from_pyarrow(pa.Table.from_pylist(rows), session=session) | ||
| import pyarrow as pa |
There was a problem hiding this comment.
IIUC, this import can be removed because the header of this file has the pyarrow importing?
bigframes/core/blocks.py
Outdated
| import pyarrow as pa | ||
|
|
||
| if not rows: | ||
| from bigframes.dtypes import bigframes_dtype_to_arrow_dtype |
There was a problem hiding this comment.
we have bigframes.dtypes import in the head of this file.
There was a problem hiding this comment.
This testcase has been cleaned up.
tests/system/small/test_dataframe.py
Outdated
| assert table.labels["test"] == "labels" | ||
|
|
||
|
|
||
| def test_dataframe_melt_multiindex(session): |
There was a problem hiding this comment.
It looks like I cannot reproduce the issue on the main branch. See screenshot/AXRSuFmXPTt5gzj
There was a problem hiding this comment.
Good catch! test_dataframe_melt_multiindex didn't catch the syntax error because the DataFrame wasn't actually empty. The core issue only happens when we try to unpivot an empty selection, causing an untyped NULL compilation.
To properly cover this edge case, I added test_count_empty_multiindex_columns in test_multiindex.py. Note that the test explicitly asserts the generated .to_frame().sql locally. If we run .to_pandas() directly, it trips over an unrelated Ibis compiler bug with empty [STRUCT()] layouts on the BigQuery backend, which I am fixing in a separate upcoming PR. This test safely completely isolates the unpivot fix in this branch..
32ce90c to
f03758d
Compare
77dff89 to
2d1c6d4
Compare
2d1c6d4 to
f8241d7
Compare
Allows alignment melts over zero-row offset layouts
Fixes #<452681068> 🦕