Python 3, 64 56 48 bytes

a,b,y=map(int,open(0))
while y*y%b-a:y-=1//(y>0)

Try it online!

This outpus via exit code.

Deciding whether there exists some \$x\$ such that \$x^2\equiv a\mod b\$ with \$0\le x\le y\land x\in\Bbb Z\$ is NP-complete. See https://www.sciencedirect.com/science/article/pii/0022000078900442 for more details.

Explanation

The first line attempts to split the input and convert it into 3 integers and assign them to a, b and y. If this fails due to the input being split into a number of parts that is not exactly 3 or a part not being convertable integer, an error is raised.

So if there in no error on line 1, then a, b, y are integers.

If b is 0 an error is raised.

If y*y%b-a is 0 and b!=0 then the while loop runs so there in no error.

If y*y%b-a is not 0 and b!=0 then the while loop runs. The loop check the loop condition each iteration. If the loop condition ever becomes true then the while loop ends with no error. Each time the loop runs, if y is positive then y decreases by 1, otherwise an error occurs.

Verification of small cases

JavaScript, 70 bytes

s=>s.split(",").reduce((a,x)=>a.flatMap(y=>[x-y,y-x]),[0]).includes(0)

Try it online!

Uses the partition problem.

Each step of the reduce produces an array of values obtained by either adding or subtracting each element processed so far. Note that those values are always symmetric about 0, so that the y-x part produces the same values as adding would (just in a different order), but using subtraction in both parts ensures that the strings are converted to numbers (whereas + would concatenate). This conversion never throws an exception for strings; it produces NaN on parsing errors (and NaN values just stay NaN when subtracted).

Charcoal, 59 48 bytes

≔⪪Ｓ-θ⊞υ⁰Ｆθ≔⁺υ⁺υ⍘ι⁺-⁻γ-υＵＭυ↔⁻⌈υ⊗ι⪫Ｅ⌕Ａ⮌⍘⌕υ⌊υ²1§θι-

Try it online! Link is to verbose version of code. Explanation: Solves the balanced partition problem, which is already known to be NP-complete.

≔⪪Ｓ-θ

Split the input string on -s, as Charcoal's custom base decoding function does not work on those (it always considers them to be signs).

⊞υ⁰

Start with the empty set whose sum is zero.

Ｆθ

Loop over each substring.

≔⁺υ⁺υ⍘ι⁺-⁻γ-υ

Turn the substring into an integer using a custom base where - is zero (so it's not actually possible to encode all integers but you can get a lot of them) and append the subset sums of all the new subsets that include this integer to the list.

ＵＭυ↔⁻⌈υ⊗ι

Calculate the subset difference between each subset and its complement.

⪫Ｅ⌕Ａ⮌⍘⌕υ⌊υ²1§θι-

Output a subset with the minimum difference.