SAKO, 41 bytes

PODPROGRAM:S(A,K,O)
S()=O×(A+O)/B-B
WROC

Python 3, 24 bytes

lambda a,b,c:c*(a+c)/b-b

Try it online!

Note that this fails when \$ a=0 \$ or \$ b=0 \$, but that seems to be OK for this challenge :). We can verify its correctness:

• For an arithmetic sequence \$ b-a = c-b \implies 2b = a+c \$, the formula rightfully produces \$ 2c-b \$: $$ \frac{c*(a+c)}{b}-b = \frac{c*2b}{b}-b = 2c-b $$
• Likewise, for a geometric sequence \$ b/a = c/b \implies a = b^2/c \$, the formula rightfully produces \$ c^2/b \$: $$ \frac{c*(a+c)}{b}-b = \frac{c*(b^2/c+c)}{b}-b = \frac{b^2+c^2}{b}-b = b+\frac{c^2}{b}-b = \frac{c^2}{b} $$

Vyxal, 6 bytes

+∇/*⁰-

Try it Online!

Port of dingledooper's clever formula, go look at that for an explanation of why this works.

       # input c, a, b
+      # c+a
 ∇     # rotate stack to b, c, c+a
  /    # c/b
   *   # (c+a) * c/b
    ⁰  # last input (b)
     - # (c+a) * c/b - b

This uses four necessary operations (+/-*) and two stack-manipulation operations (⁰∇). I'm fairly sure that a solution with only one stack-manipulation op is impossible but I'd love to be proven wrong.

Charcoal, 16 bytes

ＮθＮηＮζＩ⁻÷×⁺θζζηη

Try it online! Link is to verbose version of code. Explanation: Trivial implementation of @dingledooper's formula.

35 bytes to validate the input:

ＮθＮηＮζ¿⁼⊗η⁺θζＩ⁻⁺ζηθ¿∧θ⁼×ηη×θζＩ÷×ζηθ

Try it online! Link is to verbose version of code. Works with edge cases such as 0 0 0, 1 0 0, 1 0 -1. Doesn't output anything if the input is not an arithmetic or geometric sequence.

Notes for both solutions:

• 6 bytes (ＮθＮηＮζ) could be saved by requiring the input to be in JSON format.
• Floating-point arithmetic could be used by replacing ÷ with ∕.

Retina 0.8.2, 65 bytes

.+
$*
¶1+¶
$'$&
1(?=1*¶1+¶(1+))
$1
(?=1+¶(1+)¶)\1
1
(1+)¶\1¶1+

1

Try it online! Takes input on separate lines but link is to test suite that splits on non-digits for convenience. Limited to positive integers due to use of unary arithmetic. Explanation: Uses @dingledooper's formula.

.+
$*

Convert to unary.

¶1+¶
$'$&

Add c to a.

1(?=1*¶1+¶(1+))
$1

Multiply a+c by c.

(?=1+¶(1+)¶)\1
1

Divide that by b.

(1+)¶\1¶1+

Subtract b and remove b and c.

1

Convert to decimal.

C (gcc), 24 bytes

f(a,b,c){a=c*(a+c)/b-b;}

Another port of dingledooper's answer!

Try it online!

MathGolf, 6 bytes

+╠*k;,

Port of dingledooper's Python 3 answer, so make sure to upvote that answer as well!

I/O as floats, and inputs are in the order \$c,a,b\$.

Try it online.

Explanation:

+       # Add the first two (implicit) inputs together: c+a
 ╠      # Divide it by the third (implicit) input: (c+a)/b
  *     # Multiply it by the first (implicit) input: (c+a)/b*c
   k;   # Push the second input, and discard it
     ,  # Subtract it by the third (implicit) input: (c+a)/b*c-b
        # (after which the entire stack is output implicitly as result)

The k;, could also be ?-Þ for the same byte-count: Try it online.

   ?    # Triple-swap the top three values: a,(c+a)/b,b
    -   # Subtract the top two: a,(c+a)/b-b
     Þ  # Only keep the top value of the stack: (c+a)/b-b
        # (after which the entire stack is output implicitly as result)

Desmos, 19 bytes

f(a,b,c)=c(a+c)/b-b

Try It On Desmos!

Port of dingledooper's Python 3 solution, so make sure to upvote that one too!

Alice, 29 bytes

3&/O
?+\M@/!]!?-R.n$n?[?-.*~:

Try it online!

-4 bytes thanks to Nitrodon!

Uses loopy walt's solution, go upvote them!

3&/M\                        Read 3 arguments a b c on the stack
     !]!                     Pop and write c then b on the tape
        ?-R                  Push b on the stack and calculate b-a
           .n$n              if b-a is 0, replace with 1
               ?[?           Push b and c on the stack
                  -.*        Calculate (c-b)^2
                     ~:      Calculate (c-b)*(c-b)/(b-a or 1)
                       ?+    Pop c and calculate c+(c-b)*(c-b)/(b-a or 1)
                         \O@ Output the result

Vyxal 3, 13 bytes

¯≈[Ṛ+|ᵃ÷Iᵃ×]t

Try it Online!

Port of Nick Kennedy's Jelly answer, sort of?

Jelly, 9 bytes

Uses the trick described by loopy walt in their excellent Python answer although the implementation is somewhat different.

IQ²÷¥@/+Ṫ

A monadic Link that accepts a triple of integers, [a, b, c], that is either arithmetic or geometric and yields the next integer in the series, d.

Try it online!

How?

IQ²÷¥@/+Ṫ - Link: list of integers, [a, b, c]
I         - forward differences -> [b-a, c-b]
 Q        - deduplicate
             -> if arithmetic: L=[b-a] (potentially [0])
                else:          L=[b-a, c-b] (with no zeros)
      /   - reduce L by:
     @    -   with swapped arguments:
    ¥     -     last two links as a dyad - f(x, y):
  ²       -       square -> x^2
   ÷      -       divide -> x^2/y
             -> if arithmetic: Z=b-a (only one item so reduce does not perform f)
                else:          Z=(c-b)^2/(b-a)
       +  - add {[a, b, c]} (vectorises) -> [a+Z,b+Z,c+Z]
        Ṫ - tail -> c+Z (= d)
             -> if arithmetic: d=c+(b-a)
                else:          d=c+(c-b)^2/(b-a)

R, 28 bytes

\(a,b,c)c+(c-b)^2/(b-a+!b-a)

Attempt This Online!

Port of @loopy walt's Python answer.

R, 33 bytes

\(a,b,c)`if`(c-b-b+a,c*c/b,c+c-b)

Attempt This Online!

Straightforward approach.

APL+WIN, 43 38 bytes

Prompts for integers

↑(=/¨a b)/(↑v×↑a←2÷/v),(↑v+↑b←2-/v←⌽⎕)

Try it online! Thanks to Dyalog Classic

Perl 5 -ap, 41 bytes

$_=2*($b=<>)-$_-($c=<>)?$c*$c/$b:$c+$b-$_

Try it online!

Python, 34 bytes

-1 thanks to @Arnauld

lambda a,b,c:c+(c-b)**2/(b-a or 1)

Attempt This Online!

How?

Uses the fact that the increments form a geometric sequence in either case. (This avoids the longish (in Python) ternary operator.)

Indeed: \$c+\frac{(c-b)^2}{b-a} = \begin{cases} c+(c-b)=2c-b & \text{if }c-b=b-a \\ c+(c-b)\times \frac c b = \frac{c^2} b & \text{if } \frac c b = \frac b a \end{cases}\$

JavaScript (ES6), 28 bytes

(a,b,c)=>c-2*b+a?c*c/b:2*c-b

Try it online!

Commented

(a, b, c) =>    // given the 3 integers,
c - 2 * b + a ? // if the sequence is not arithmetic:
  c * c / b     //   assume it's geometric and return the next geometric term
:               // else:
  2 * c - b     //   return the next arithmetic term

Funge-98, 46 bytes

<@.+wR-R:VI\OO-R:VIVI(4"FRTH"(4"STRN"
/\OR<@.*

Try it online!

Awk, 33 bytes

1,$0=$1-2*$2+$3?$2*$3/$1:$2+$3-$1

Try it online!

You could remove the 1, at the start if you assume that the output will not be 0, saving 2 bytes.

Raku (Perl 6), 13 bytes

A block taking a sequence of length 3 as a single argument. This task is a perfect match for Raku's sequence operator.

{($_...*)[3]}

Attempt This Online!

Google Sheets, 34 bytes

=IF(A3/A2=A2/A1,A3*A2/A1,A3+A2-A1) 

The formula assumes the sequence is in A1:A3.

Jelly, 11 bytes

I+×÷Ɲ$}E?UḢ

Try it online!

A monadic link taking a list of three integers and returning an integer.

Explanation

I           | Increments (differences between consecutive list members)
       E?   | If all equal then:
 +       U  | - Add the increments to the reversed input list
     $}  U  | Else, following as a monad applied to the reversed input list:
  ×         | - Multiply by:
   ÷Ɲ       |   - The result of dividing each pair of neighbouring list members
          Ḣ | Head