AWK, 25 bytes

1,$0=($1^3-$1)*($1/4+1/6)

Attempt This Online!

J-uby, 25 bytes

:+|:sum+:*%[~:**&2,~:-&1]

Attempt This Online!

Thunno 2, 6 bytes

RDḣ²Ø.

Attempt This Online!

Port of Jonathan Allan's Jelly answer.

Thunno 2, 8 bytes

RDS²s²S-

Attempt This Online!

Literal solution.

Explanation

RDḣ²Ø.  # Implicit input
R       # Push the range [1..input]
 D      # Duplicate it
  ḣ     # Remove the first item
   ²    # Square each number
    Ø.  # Dot product of the two lists
        # Implicit output

RDS²s²S-  # Implicit input
R         # Push the range [1..input]
 D        # Duplicate it
  S²      # Square the sum of the list
    s     # Swap so the range is back on top
     ²S   # Sum the squares of the numbers
       -  # Subtract
          # Implicit output

Vyxal, 7 bytes

ɾ:∑p²ƒ-

Try it Online!

A port of Fig which is a port of 05ab1e

Explained

ɾ:∑p²ƒ-
ɾ:∑p     # prepend the sum of the range [1, n] to the range [1, n]
   ²ƒ-  # square everything and reduce by subtraction

Julia 1.0, 20 bytes

Using the same formula as bigyihsuan:

~n=(n^3-n)*(3n+2)/12

Try it online!

~n=(a=[1:n...];sum(a)^2-sum(a.^2))

Try it online!

Husk, 11 8 bytes

-ṁ□ḣ¹□Σḣ

Try it online!

Explanation

-ṁ□ḣ¹□Σḣ
-              Subtract
 ṁ□ḣ¹          The sum of the first n squares
     □         From the square of
      Σḣ       The sum from 1 to n

First time code golfing so this solution probably can be improved.

Thanks to Dominic van Essen for the -3 bytes.

Pip, 15 13 11 bytes

SQ_*D_MS\,a

Try It Online!

Explanation

Adapted from one of the formulas in Engineer Toast's Excel answer:

$$ \sum_{i=1}^{n} i^3-i^2 $$

which is equivalent to

$$ \sum_{i=1}^{n} i^2(i-1) $$

SQ_*D_MS\,a
          a  Command-line argument
        \,   Inclusive range from 1
      MS     Map this function to each and sum the results:
SQ_            The argument squared
   *           Times
    D_         The argument decremented

Original 15-byte solution that implements the description directly:

\,:aSQ$+a-$+SQa

Try It Online!

Excel, 21 bytes

=(A1^3-A1)*(A1/4+1/6)

This is the heartless Excel version of the formula figured out by others. There is also the 33 byte solution that feels a little more Excel-y:

=LET(n,SEQUENCE(A1),SUM(n^3-n^2))

... and the 38 byte solution that looks the most like the original formulas in the question:

=LET(n,SEQUENCE(A1),SUM(n)^2-SUMSQ(n))

Fig, \$9\log_{256}(96)\approx\$ 7.408 bytes

a
R@-mQJS

Try it online!

Port of 05AB1E. Literal solution is 2 chars longer.

a
R@-mQJS
-------
a       # Range [1, n]
------- # Use ^ as the input to the next line:
     JS # Prepend the sum to the list
   mQ   # Square each
R@      # Reduce by
  -     # Subtraction

J, 15 bytes

(+/@:**:@>:)@i.

Uses the form \$\sum_{i=0}^{n-1}((i+1)^2\cdot i)\$

Attempt This Online!

(+/@:**:@>:)@i.
             i. NB. range 0..n-1
            @   NB. atop f (g x)
(          )    NB. monadic hook, y f (g y)
      *:@>:     NB. increment then square
 +/@:*          NB. multiply square with each i then sum the result
                NB. u@:v where v is dyadic and u is always monadic,
                NB. like atop except u will execute once on the entire result

Factor + math.unicode, 32 bytes

[ [1,b] 3 dupn Σ sq -rot v. - ]

Try it online!

word   | data stack                                     | comment
-------+------------------------------------------------+--------------
       | 10                                             | example input
[1,b]  | { 1 2 ... 10 }                                 |
3      | { 1 2 ... 10 } 3                               |
dupn   | { 1 2 ... 10 } { 1 2 ... 10 } { 1 2 ... 10 }   |
Σ      | { 1 2 ... 10 } { 1 2 ... 10 } 55               | sum
sq     | { 1 2 ... 10 } { 1 2 ... 10 } 3025             | square
-rot   | 3025 { 1 2 ... 10 } { 1 2 ... 10 }             |
v.     | 3025 385                                       | dot product
-      | 2640                                           | output

Vyxal, 8 bytes

ɾ∑²⁰ɾ²∑-

Try it Online!

literal solution, squares the sum and subtracts the sum of the squares.

C (gcc), 29 bytes

f(n){n=(3*n+2)*(n*n*n-n)/12;}

Try it online!

JavaScript (Node.js), 59 48 47 bytes

n=>(a=(x,b)=>x&&x*(b||x)+a(--x,b))(n,1)**2-a(n)

Try it online!

ARM (Thumb), 18 bytes

.section .text
.global func
.thumb
// int func(int dummy, int x)
// returns sum(i*i*(i-1) for i in 1..x)
// r1 = i (x->1), r2 = i*i*(i-1), r0 = sum
func:
    mov r0, #0       // 2000   int sum = 0;
    mov r2, #0       // 2200   int prod = 0;
.loop:               //        do {
    add r0, r0, r2   // 1880   sum += prod;
    mov r2, r1       // 1c0a   prod = x;
    mul r2, r1       // 434a   prod *= x;
    sub r1, #1       // 3901   x -= 1;
    mul r2, r1       // 434a   prod *= x;
    bne .loop        // d1f9   } while (prod != 0);
    bx lr            // 4770   return sum;

A function that takes the input integer via r1 and returns the value via r0.

Uses the iterative formula \$\sum_{i=1}^{x}{i^2(i-1)}\$, iterating backwards. The loop terminates when the value of \$i^2(i-1)\$ is 0.

For the purposes of writing test cases, the C-side function signature takes a dummy value for r0.

Test cases written in C:

#include <stdio.h>

int func(int dummy, int x);
int main(void) {
    int dummy;
    int inputs[] = {1, 2, 3, 10, 24, 100};
    for(int i = 0; i < 6; ++i) {
        int v = inputs[i];
        printf("input = %d -> ans = %d\n", v, func(dummy, v));
    }
    return 0;
}

Build, run, objdump commands:

arm-linux-gnueabihf-gcc main_c.c func_c.S -static -o main_c
qemu-arm -L /usr/arm-linux-gnueabihf ./main_c
arm-linux-gnueabihf-objdump -d main_c | awk -v RS= '/^[[:xdigit:]]+ <(func|.loop)>/'

(The objdump + awk trick comes from this SO answer.)

Labyrinth, 20 bytes

?(:):):_3*(***_12/!@

Try it online!

Straightforward closed-form formula of \$(x-1)x(x+1)(3x+2)/12\$.

?      Take input [x]
(:):)  Decrement, dup, increment, dup, increment [x-1 x x+1]
:_3*(  Dup, 3 times, decrement [x-1 x x+1 3x+2]
***    Multiply four values
_12/   Divide by 12
!@     Print it and halt

Labyrinth, 21 bytes

?
::(";;;!@
{  :
+**}

Try it online!

Labyrinth, 22 bytes

   ?
:(::
}  "
**+{;!@

Try it online!

Two versions using a loop to evaluate \$\sum_{i=1}^{x}{i^2(i-1)}\$. Both versions test the loop exit with i-1 == 0, discard the top values ; as necessary, print the sum and exit.

?     Take input [x]; enter loop [sum i] with initial values of sum=0, i=x
::(   Dup, dup, decrement [sum i i i-1]
:}    Dup and move i-1 to aux. store [sum i i i-1 | i-1]
**+   Multiply top three values and add to sum [sum' | i-1]
{     Move i-1 back to the main stack

      Loop exit test can be done anywhere in the last iteration
      (i==1 or i==0), since that iteration does not affect the sum.

05AB1E, 11 bytes

LDOnU€nOXs-

Try it online!

             # (implicit) push STDIN to stack
LD           # push [1, 2, 3, ..., top of stack] twice
  OnU        # store squared sum of all values in top of stack in variable X
     €       # for each in top of stack...
      n      # push top of stack squared
       O     # push sum of all values in top of stack
        Xs-  # push X - top of stack
             # (implicit) output top of stack to STDOUT

MATLAB/Octave, 21 bytes

@(x)(x^3-x)*(x/4+1/6)

Anonymous function. We could save another 4 bytes if we assume x can be given as variable in workspace.
Implements \$n(n−1)(n+1)(3n+2)/12\$

One of my approaches was @(x)-sum([sum(1:x)*i,1:x].^2) which is few bytes longer but I found it interesting because It used imaginary unit to change the sign.
I also found out I can generate sum of squares by multiplying normal and transposed vector with (1:x)*(1:x)'. Might be useful for future challenges!

Flurry -nii, 32 bytes

{}{{}[<><<>()>]<([])[][]>}[<>()]

Try it online!

Iterates through k=0 .. n-1 using the stack height, adding k(k+1)^2 each iteration.

How it works

main = n add-next-term 0

// Initial height is 0
// Increase the height at (push height), so subsequent `height` calls give k+1
add-next-term = \x. x succ <(push height) height height>
= \x. x + k * (k+1) * (k+1)

Assembly (MIPS, SPIM), 142 114 bytes, 6 * 12 10 = 72 60 assembled bytes

^{Saved 2 instructions (12 bytes) from @Bubbler's golf.}

main:li$v0,5
syscall
li$a0,0
l:mul$t0,$v0,$v0
sub$v0,$v0,1
mul$t0,$t0,$v0
add$a0,$a0,$t0
bgtz$v0,l
li$v0,1
syscall

Try it online!

Explanation

main:
    li $v0, 5              # Set syscall value 5
    syscall                # Syscall, v0 = input integer

    li $a0, 0              # a0 = 0

    loop:                  # Main loop (v0 is the counter):
        mul $t0, $v0, $v0  #     t0 = v0 * v0
        sub $v0, $v0, 1    #     v0 = v0 - 1
        mul $t0, $t0, $v0  #     t0 = t0 * v0
        add $a0, $a0, $t0  #     a0 = a0 + t0

        bgtz $v0, loop     #     if v0 > 0, jump to loop

    li $v0, 1              # Syscall value 1
    syscall                # Syscall (Output value of a0 as integer)

Pip, 21 bytes

($+\,a)**2-$+(\,a)**2

My first self written pip answer!

Try it online!

Haskell, 22 bytes

f n=(3*n+2)*(n^3-n)/12

Arn, 9 bytes

└V0¯„○aÌ$

Try it!

Explained

Unpacked: +{^3-^2}\~.

Performs \$\large\sum_{k=1}^{n}{(k^3-k^2)}\$ where n is the input

          \       Fold with
+                 Addition
  {               After mapping with this block
        _         Implied variable (implicit index of block).
      ^           To the power of
        3         Literal three
    -             Minus
        _
      ^
        2         Literal two
  }               End block
            ~     1-range to
              _   Initialized to STDIN; implied

Whispers v2, 70 bytes

> 2
> Input
>> L³
>> L²
>> ∑ 1 2 3
>> ∑ 1 2 4
>> 5-6
>> Output 7

Try it online!

How it works

Implements the formula

$$\sum^n_{i=2}i^3 - \sum^n_{i=2}i^2$$

As Whispers is difficult to deal with when it comes to compound statements inside of ∑, Each etc. statements, it's actually shorter to calculate two sums and take their difference than it is to calculate

$$\sum^n_{i=2}(i^3-i^2)$$

or any other single sum.

This has to be one of the easiest-to-read Whispers programs, aside from very trivial ones. We use the ∑ statement to calculate the two sums, which works by taking the starting bound as the first value, the end bound second and finally the function to sum through last. Both of the ∑ statements have the same bounds - \$2\$ and \$n\$ - so the only thing that differs is the function. For the first ∑ statement, we use the statement >> L³ (i.e. cube), and >> L² (square) for the second. This yields the sums \$\sum^n_{i=2}i^3\$ and \$\sum^n_{i=2}i^2\$. Finally, we take their difference and output it.

Whispers v3, 48 bytes

> 2
> Input
>>> L³-L²
>> ∑ 1 2 3
>> Output 4

With the introduction of pattern lines (starting with >>>) in version 3, compound statements in Whispers are much easier to deal with. Unfortunately, Whispers v3 is currently not available on TIO, so you'll have to use the interpreter in the Github repo.

This version takes advantage of pattern lines to shorten the code by only using one statement in the sum, rather than two separate sums. Other than that, it's fairly clear how it works, given the v2 program.

International Phonetic Esoteric Language, 21 bytes

ɪbwbbbffʍz3ʍf2sf{C}vo

Implements \$(n^3-n)(3n+2)/12\$.

Explanation:

ɪ       (input)
bw      (store n)
bbbffʍz (n^3-n)
3ʍf2s   (3n+2)
f       (multiply)
{C}v    (/12)
o       (print)

Testing against examples:

$ for i in 1 2 3 10 24 100; do echo {$i} | src/interpreter.py 'ɪbwbbbffʍz3ʍf2sf{C}vo'; done
0.0
4.0
22.0
2640.0
85100.0
25164150.0

Keg, 15 bytes

&⑻Ï⑷²⑸⅀⑻:⑨*½²$-

Try it online!

Calculates the sum of squares then the square of the sum.

PHP, 37 bytes

while($i<$argn)$s+=$i++*$i*$i;echo$s;

Try it online!

Standalone program input number via STDIN.

Perl 6, 22 bytes

{($_³-$_)*($_/4+⅙)}

Try it online!

PowerShell, 73 39 bytes

1.."$args"|%{$r+=$_;$s+=$_*$_}
$r*$r-$s

Try it online!

-34 bytes thanks to @mazzy and his genius PowerShell-foo

05AB1E, 6 bytes

LDOšnÆ

Try it online!

Explanation:

           # implicit input (example: 3)
L          # range ([1, 2, 3])
 DOš       # prepend the sum ([6, 1, 2, 3])
    n      # square each ([36, 1, 4, 9])
     Æ     # reduce by subtraction (22)
           # implicit output

Æ isn't useful often, but this is its time to shine. This beats the naïve LOnILnO- by two whole bytes.

cQuents, 17 15 bytes

b$)^2-c$
;$
;$$

Try it online!

Explanation

 b$)^2-c$     First line
:             Implicit (output nth term in sequence)
 b$)          Each term in the sequence equals the second line at the current index
    ^2        squared
      -c$     minus the third line at the current index

;$            Second line - sum of integers up to n
;$$           Third line - sum of squares up to n

C, C++, 46 40 37 bytes ( #define ), 50 47 46 bytes ( function )

-1 byte thanks to Zacharý

-11 bytes thanks to ceilingcat

Macro version :

#define F(n)n*n*~n*~n/4+n*~n*(n-~n)/6

Function version :

int f(int n){return~n*n*n*~n/4+n*~n*(n-~n)/6;}

Thoses lines are based on thoses 2 formulas :

Sum of numbers between 1 and n = n*(n+1)/2
Sum of squares between 1 and n = n*(n+1)*(2n+1)/6

So the formula to get the answer is simply (n*(n+1)/2) * (n*(n+1)/2) - n*(n+1)*(2n+1)/6

And now to "optimize" the byte count, we break parenthesis and move stuff around, while testing it always gives the same result

(n*(n+1)/2) * (n*(n+1)/2) - n*(n+1)*(2n+1)/6 => n*(n+1)/2*n*(n+1)/2 - n*(n+1)*(2n+1)/6 => n*(n+1)*n*(n+1)/4 - n*(n+1)*(2n+1)/6

Notice the pattern p = n*n+1 = n*n+n, so in the function, we declare another variable int p = n*n+n and it gives :

p*p/4 - p*(2n+1)/6

For p*(p/4-(2*n+1)/6) and so n*(n+1)*(n*(n+1)/4 - (2n+1)/6), it works half the time only, and I suspect integer division to be the cause ( f(3) giving 24 instead of 22, f(24) giving 85200 instead of 85100, so we can't factorize the macro's formula that way, even if mathematically it is the same.

Both the macro and function version are here because of macro substitution :

F(3) gives 3*3*(3+1)*(3+1)/4-3*(3+1)*(2*3+1)/6 = 22
F(5-2) gives 5-2*5-2*(5-2+1)*(5-2+1)/4-5-2*(5-2+1)*(2*5-2+1)/6 = -30

and mess up with the operator precedence. the function version does not have this problem

F# (Mono), 57 41 bytes

let f x=Seq.sumBy(fun y->y*y*(y-1))[1..x]

Try it online!

JAEL, 13 10 bytes

#&àĝ&oȦ

Try it online!

Explanation (generated automatically):

./jael --explain '#&àĝ&oȦ'
ORIGINAL CODE:  #&àĝ&oȦ

EXPANDING EXPLANATION:
à => `a
ĝ => ^g
Ȧ => .a!

EXPANDED CODE:  #&`a^g&o.a!

COMPLETED CODE: #&`a^g&o.a!,

#          ,            repeat (p1) times:
 &                              push number of iterations of this loop
  `                             push 1
   a                            push p1 + p2
    ^                           push 2
     g                          push p2 ^ p1
      &                         push number of iterations of this loop
       o                        push p1 * p2
        .                       push the value under the tape head
         a                      push p1 + p2
          !                     write p1 to the tapehead
            ␄           print machine state

Python 3, 72 bytes

x=[i+1for i in range(int(input()))]
print(sum(x)**2-sum(i**2for i in x))

Try it online!

Ruby, 24 bytes

->n{(n+n+3*n*=n)*~-n/12}

Try it online!

05AB1E, 8 bytes

ÝDOnsnO-

Explanation:

ÝDOnsnO-     //Full program
Ý            //Push [0..a] where a is implicit input
 D           //Duplicate top of stack
  On         //Push sum, then square it
    s        //Swap top two elements of stack
     nO      //Square each element, then push sum
       -     //Difference (implicitly printed)

Try it online!

QBASIC, 45 44 bytes

Going pure-math saves 1 byte!

INPUT n
?n^2*(n+1)*(n+1)/4-n*(n+1)*(2*n+1)/6

Try THAT online!

Previous, loop-based answer

INPUT n
FOR q=1TO n
a=a+q^2
b=b+q
NEXT
?b^2-a

Try it online!

Note that the REPL is a bit more expanded because the interpreter fails otherwise.

Brain-Flak, 74 72 68 64 bytes

((([{}])){({}())}{})([{({}())({})}{}]{(({}())){({})({}())}{}}{})

Try it online!

Pretty simple way of doing it with a couple of tricky shifts. Hopefully someone will find some more tricks to make this even shorter.

Clojure, 58 bytes

(fn[s](-(Math/pow(reduce + s)2)(reduce +(map #(* % %)s))))

Edit: I misunderstood the question

Clojure, 55, 35 bytes

#(* %(+ 1 %)(- % 1)(+(* 3 %)2)1/12)

Try it online!

MathGolf, 6 bytes

{î²ï*+

Try it online!

Calculates \$\sum_{k=1}^n (k^2(k-1))\$

Explanation:

{       Loop (implicit) input times
 î²     1-index of loop squared
    *   Multiplied by
   ï    The 0-index of the loop
     +  And add to the running total

Clojure, 91 bytes

(fn s[n](let[u #(apply + %)q #(Math/pow % 2)m(range 1(inc n))](-(q(u m))(u(map #(q %)m)))))

The naive, literal approach. See below:

(defn sum-sq-diff [n]
    (let [; Shortcut functions to save bytes
          sum #(apply + %)
          square #(Math/pow % 2)
          nums (range 1 (inc n))

          ss1 (sum (map #(square %) nums))
          ss2 (square (sum nums))]

      (- ss2 ss1)))

(mapv sum-sq-diff [1 2 3 10 24 100])
=> [0.0 4.0 22.0 2640.0 85100.0 2.516415E7]

Try it online!

Stax, 4 bytes

╡⌠(♠

Run and debug it

For all positive k integers up to the input, add k^2 * (k-1).

R, 28 bytes

x=1:scan();sum(x)^2-sum(x^2)

Try it online!

05AB1E, 6 bytes

LnDƶαO

Try it online!

Explanation

L         # push range [1 ... input]
 n        # square each
  D       # duplicate
   ƶ      # lift, multiply each by its 1-based index
    α     # element-wise absolute difference
     O    # sum

Some other versions at the same byte count:

L<ān*O
Ln.āPO
L¦nā*O

APL(NARS), 13 chars, 26 bytes

{+/⍵×⍵×⍵-1}∘⍳

use the formula Sum'w=1..n'(ww(w-1)) possible i wrote the same some other wrote + or - as "1⊥⍳×⍳×⍳-1"; test:

  g←{+/⍵×⍵×⍵-1}∘⍳
  g 0
0
  g 1
0
  g 2
4
  g 3
22
  g 10
2640

Axiom, 39 bytes

f(n)==reduce(+,[x^3-x^2 for x in 1..n])

test:

-> [[x,f x]for x in [1,2,3,10]]
     [[1,0],[2,4],[3,22],[10,2640]]

dc, 16 bytes

?dd3^r-r3*2+*C/p

Implements \$(n^3-n)(3n+2)/12\$

Try it online!

Java (JDK), 23 bytes

n->(3*n+2)*(n*n*n-n)/12

Try it online!

Mathematica, 21 17 bytes

^{-4 bytes thanks to alephalpha.}

(3#+2)(#^3-#)/12&

Pure function. Takes an integer as input and returns an integer as output. Just implements the polynomial, since Sums, Ranges, Trs, etc. take up a lot of bytes.

C#, 89 Bytes

int x=0,y=0;for(int i=1;i<=Int32.Parse(s[0]);i++){x+=i*i;y+=i;}Console.Write($"{y*y-x}");

ungolfed:

int x=0,y=0;
for(int i=1;i<=Int32.Parse(s[0]);i++){
x+=i*i;
y+=i;
}
Console.Write($"{y*y-x}");

Try it online!

TI-Basic (TI-83 series), 12 11 bytes

sum(Ans² nCr 2/{2,3Ans

Implements \$\binom{n^2}{2}(\frac12 + \frac1{3n})\$. Takes input in Ans: for example, run 10:prgmX to compute the result for input 10.

Pari/GP, 21 bytes

n->(3*n+2)*(n^3-n)/12

Try it online!

APL (Dyalog Unicode), 10 bytes

1⊥⍳×⍳×1-⍨⍳

Try it online!

How it works

1⊥⍳×⍳×1-⍨⍳
  ⍳×⍳×1-⍨⍳  Compute (x^3 - x^2) for 1..n
1⊥          Sum

Uses the fact that "square of sum" is equal to "sum of cubes".

Python 3,  28  27 bytes

-1 thanks to xnor

lambda n:(n**3-n)*(n/4+1/6)

Try it online!

Implements \$n(n-1)(n+1)(3n+2)/12\$

Python 2,  29  28 bytes: lambda n:(n**3-n)*(3*n+2)/12

SNOBOL4 (CSNOBOL4), 70 69 bytes

 N =INPUT
I X =X + N ^ 3 - N ^ 2
 N =GT(N) N - 1 :S(I)
 OUTPUT =X
END

Try it online!

Perl 6, 22 bytes

{sum (1..$_)>>²Z*^$_}

Try it online!

Uses the construction \$ \sum_{i=1}^n {(i^2(i-1))} \$

Pyth, 7 bytes

sm**hdh

Try it online here.

Uses the formula in Neil's answer.

sm**hdhddQ   Implicit: Q=eval(input())
             Trailing ddQ inferred
 m       Q   Map [0-Q) as d, using:
    hd         Increment d
   *  hd       Multiply the above with another copy
  *     d      Multiply the above by d
s            Sum, implicit print 

Charcoal, 12 10 bytes

ＩΣＥＮ×ιＸ⊕ι²

Try it online! Link is to verbose version of code. Explanation: \$ ( \sum_1^n x )^2 = \sum_1^n x^3 \$ so \$ ( \sum_1^n x )^2 - \sum_1^n x^2 = \sum_1^n (x^3 - x^2) = \sum_1^n (x - 1)x^2 = \sum_0^{n-1} x(x + 1)^2 \$.

   Ｎ        Input number
  Ｅ         Map over implicit range i.e. 0 .. n - 1
        ι   Current value
       ⊕    Incremented
         ²  Literal 2
      Ｘ     Power
     ι      Current value
    ×       Multiply
 Σ          Sum
Ｉ           Cast to string
            Implicitly print

APL (Dyalog), 16 bytes

((×⍨+/)-(+/×⍨))⍳

Try it online!

 (×⍨+/)            The square (× self) of the sum (+ fold)
       -           minus
        (+/×⍨)     the sum of the square
(             )⍳   of [1, 2, … input].

JavaScript, 20 bytes

f=n=>n&&n*n*--n+f(n)

Try it online

APL(Dyalog), 17 bytes

{+/(¯1↓⍵)×1↓×⍨⍵}⍳

(Much longer) Port of Jonathan Allan's Jelly answer.

Try it online!

Japt -x, 9 8 5 4 bytes

õ²í*

Try it

Explanation

õ        :Range [1,input]
 ²       :Square each
  í      :Interleave with 0-based indices
   *     :Reduce each pair by multiplication
         :Implicit output of the sum of the resulting array

Jelly,  5  4 bytes

Ḋ²ḋṖ

Try it online!

How?

Implements \$\sum_{i=2}^n{(i^2(i-1))}\$...

Ḋ²ḋṖ - Link: non-negative integer, n
Ḋ    - dequeue (implicit range)       [2,3,4,5,...,n]
 ²   - square (vectorises)            [4,9,16,25,...,n*n]
   Ṗ - pop (implicit range)           [1,2,3,4,...,n-1]
  ḋ  - dot product                    4*1+9*2+16*3+25*4+...+n*n*(n-1)

JavaScript (ES6), 22 bytes

n=>n*~-n*-~n*(n/4+1/6)

Try it online!