Skip to content

feat(spark): Adds negative spark function #20006

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
SubhamSinghal wants to merge 26 commits into
base: main
Choose a base branch
from
Open

feat(spark): Adds negative spark function #20006

SubhamSinghal wants to merge 26 commits into from
+650 −2

Conversation

@SubhamSinghal
Copy link

@SubhamSinghal SubhamSinghal commented Jan 26, 2026 •
edited
Loading

What changes are included in this PR?

Adds support for negative spark function in data fusion.

Are these changes tested?

yes, using UTs

Are there any user-facing changes?

yes, adds new function.

@SubhamSinghal SubhamSinghal changed the title Adds negative spark function feat(spark): Adds negative spark function Jan 26, 2026
@github-actions github-actions bot added the spark label Jan 26, 2026
@comphead
Copy link
Contributor

comphead commented Jan 26, 2026

Running CI

comphead
comphead reviewed Jan 26, 2026
View reviewed changes
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SubhamSinghal

Please add .slt tests for negative functions, the simplest case is https://spark.apache.org/docs/latest/api/sql/#negative

However you can generate bunch of other edge cases with LLM

Jefffrey
Jefffrey reviewed Jan 27, 2026
View reviewed changes
SubhamSinghal and others added 4 commits January 27, 2026 18:39
@SubhamSinghal @Jefffrey
Update datafusion/spark/src/function/math/negative.rs
897fbef 
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
@SubhamSinghal @Jefffrey
Update datafusion/spark/src/function/math/negative.rs
34819a0 
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
@SubhamSinghal @Jefffrey
Apply suggestion
76aa1d3 
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
Adds slt file
2901ca5
@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Jan 27, 2026
comphead
comphead reviewed Jan 27, 2026
View reviewed changes
@comphead comphead mentioned this pull request Jan 27, 2026
Open
comphead
comphead reviewed Jan 27, 2026
View reviewed changes
comphead
comphead reviewed Jan 27, 2026
View reviewed changes
comphead
comphead reviewed Jan 27, 2026
View reviewed changes
SubhamSinghal and others added 5 commits January 27, 2026 22:27
@SubhamSinghal @comphead
Update datafusion/spark/src/function/math/negative.rs
30f000b 
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
@SubhamSinghal @comphead
Update datafusion/spark/src/function/math/negative.rs
89b3059 
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
@SubhamSinghal @comphead
Update datafusion/spark/src/function/math/negative.rs
77f63af 
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
Adds interval type
ddd30d0
Lint fix
9191b3a
martin-g
martin-g reviewed Jan 28, 2026
View reviewed changes
SubhamSinghal and others added 5 commits January 28, 2026 18:51
@SubhamSinghal @martin-g
Update datafusion/spark/src/function/math/negative.rs
da620bb 
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
@SubhamSinghal @martin-g
Update datafusion/spark/src/function/math/negative.rs
1b971c8 
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Adds arg
99cba0a
Add unsigned values for scalar
cb79d2c
Adds UT for interval and unsigned type
aaed6d2
@SubhamSinghal
Copy link
Author

SubhamSinghal commented Jan 28, 2026

Could you please also add some tests for them to negative.slt ?

Done

comphead
comphead reviewed Jan 28, 2026
View reviewed changes
comphead
comphead reviewed Jan 28, 2026
View reviewed changes
comphead
comphead previously approved these changes Jan 28, 2026
View reviewed changes
Copy link
Contributor

@comphead comphead left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SubhamSinghal IMO the PR is good to go, appreciate the contribution

UPD: CI is failing now for tests, I'll wait for green CI, however as I said before the PR looks good overall

@comphead comphead self-requested a review January 28, 2026 18:15
@comphead comphead dismissed their stale review January 28, 2026 18:15

waiting for CI

SubhamSinghal and others added 2 commits January 29, 2026 08:33
@SubhamSinghal @comphead
Update datafusion/spark/src/function/math/negative.rs
6af34c4 
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
Fix UT
b743145
@SubhamSinghal
Copy link
Author

SubhamSinghal commented Jan 29, 2026

@comphead Changes Signature from Signature::numeric to Signature::any to allow IntervalType. Can you run CI again?

comphead
comphead reviewed Jan 29, 2026
View reviewed changes
datafusion/sqllogictest/test_files/spark/math/negative.slt
SELECT negative(1.7976931348623157e308::double);
----
-1.7976931348623157e308
-179769313486231570000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Copy link
Contributor

@comphead comphead Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why this would change? is it the same datatype returned?

Copy link
Author

@SubhamSinghal SubhamSinghal Jan 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think issue is with scientific notation representation

comphead
comphead reviewed Jan 29, 2026
View reviewed changes
xanderbailey
xanderbailey reviewed Jan 29, 2026
View reviewed changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@comphead comphead comphead left review comments

@Jefffrey Jefffrey Awaiting requested review from Jefffrey

+2 more reviewers

@martin-g martin-g martin-g left review comments

@xanderbailey xanderbailey xanderbailey left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@SubhamSinghal @comphead @martin-g @xanderbailey @Jefffrey