Janet, 41 bytes

|(distinct(seq[i :down[$ 0]](%(* i i)$)))                                                                                               

J-uby, 30 bytes

:*%[:& &~:%%(~:**&2),:*]|:uniq

Attempt This Online!

K (ngn/k), 11 10 bytes

{?x!*/&=x}

Try it online!

• &=x generate a list containing (0..x; 0..x)
• */ multiply the two items of the list together
• x! mod the result by x
• ? (implicitly) return the distinct last digits

J, 10 bytes

~.@:|2^~i.

Attempt This Online!

~.@:|2^~i.
        i.  NB. range [0..n)
     2^~    NB. square
  @:        NB. then
~.          NB. uniquify

Thunno 2, 4 bytes

L²ŒU

Try it online!

Explanation

L²ŒU  # Implicit input
L     # Push [0..input)
 ²    # Square each value
  Π  # Mod by input
   U  # Uniquify
      # Implicit output

JavaScript, 45 bytes

n=>(g=x=>x?g(g[x*x%n]=x-1):Object.keys(g))(n)

Try it online!

40 bytes

With output as an array that could contain multiple empty items.

n=>(g=x=>x?g(x-1,a[y=x*x%n]=y):a)(a=[n])

Try it online!

Thunno D, \$ 6 \log_{256}(96) \approx \$ 4.94 bytes

R2^$ZU

Attempt This Online!

Thunno, \$ 7 \log_{256}(96) \approx \$ 5.76 bytes

DR2^$ZU

Attempt This Online!

Explanation

DR2^$ZU  # Implicit input
DR       # Duplicate and get range(input)
  2^     # Square each number
    $    # Mod by the input
     ZU  # Uniquify
         # Implicit output

Nibbles, 5 bytes

`$.,$%^$~@

Attempt This Online!

`$ Deduplicate
.   map
,    range
$     input
%    mod
^     power
$      it
~      2
@     input

Pyt, 7 bytes

Đř²⇹%Ụş

Try it online!

Đ            implicit input (n); Đuplicate on stack
 ř           řangify (push [1,2,...,n])
  ²          square
   ⇹%        modulo n
     Ụş      get Ụnique elements and şort in ascending order; implicit print

Vyxal, 5 bytes

ɾ²$%U

Try it Online!

ɾ     # 1...n
 ²    # Each squared
   %  # Modulo...
  $   # input
    U # Uniquify the result

Prolog (SWI), 49 bytes

N+X:-setof(K,A^(between(0,N,A),K is A^2mod N),X).

Try it online!

Creates a set of values from K=(A^2)%N, where 0<=A<=N.

AWK, 44 43 40 bytes

{for(;i++<$0;)if(!a[j=i*i%$0]++)print j}

I managed to cut off 3 bytes after revisiting this old answer.

Step by step:

{
for(;i++<$0;)        For all numbers less of equal to the input...
            
if(!a[j=i*i%$0]++)   Increments the element j of the array a.
                     j is the last digit of each i*i.
                     If the element a[j] is being incremented
                     for the first time (a[j]++ returns 0,
                     so negating it (!a[j]++) returns 1)...

print j              Prints j.
}

Try it online!

Former answer (43 bytes)

{for(;i++<$0;)a[i*i%$0];for(j in a)print j}

Edit: Incrementing the i variable during the for condition saved one more byte.

Step by step:

{                        
for(;               # starts the loop with no previous statement.
    i++<$0;)        # loops while _i_ is less than the input ($0).
                    # also increments 1 to the _i_ variable after
                    # it is evaluated.

    a[i*i%$0];      # fiat the _i*i%$0_ element of the _a_ array!
                    # by only stating _array[element]_, the element exists.
                    # i*i%$0 means: i squared (mod $0), i.e., the last digit.

for(j in a)         # for every element _j_ existing in the _a_ array,
    print j         # prints the element _j_
}

Try it online!

Perl 5 -MList::Util=uniq -na, 32 bytes

say for uniq map$_**2%"@F",0..$_

Try it online!

APL (Dyalog Unicode), 6 bytes

∪⊢|⍳×⍳

Try it online!

Explanation:

∪⊢|⍳×⍳
   ⍳×⍳    ⍝ square every number in range from 1 to ⍵
 ⊢|       ⍝ modulo ⍵
∪         ⍝ unique elements

Whispers v2, 71 bytes

> Input
>> [1)
>> L²
>> L%1
>> Each 3 2
>> Each 4 5
>> {6}
>> Output 7

Try it online!

The TIO Footer simply sorts the output, remove it to see the unsorted set.

Pari/GP, 25 bytes

n->Set([x^2%n|x<-[1..n]])

Try it online!

Java (OpenJDK 8), 86 77 bytes

n->java.util.stream.IntStream.range(1,n+1).map(a->a*a%n).distinct().toArray()

Try it online!

Google Sheets, 52 51 47 bytes

=ArrayFormula(Join(",",Unique(Mod(Row(A:A)^2,A1

_{^{Saved 4 bytes thanks to Taylor Scott}}

Sheets will automatically add 4 closing parentheses to the end of the formula.

It doesn't return the results in ascending order but it does return the correct results.

8th, 138 131 bytes

Code

[] swap dup >r ( 2 ^ r@ n:mod a:push ) 1 rot loop rdrop ' n:cmp a:sort ' n:cmp >r -1 a:@ swap ( tuck r@ w:exec ) a:filter rdrop nip

Explanation

[] - Create output array

swap dup >r - Save input for later use

( 2 ^ r@ n:mod a:push ) 1 rot loop - Compute square end

rdrop - Clean r-stack

' n:cmp a:sort - Sort output array

' n:cmp >r -1 a:@ swap ( tuck r@ w:exec ) a:filter rdrop nip - Get rid of consecutive duplicates from array

SED (Stack Effect Diagram) is: a -- a

Usage and example

: f [] swap dup >r ( 2 n:^ r@ n:mod a:push ) 1 rot loop rdrop ' n:cmp a:sort ' n:cmp >r -1 a:@ swap ( tuck r@ w:exec ) a:filter rdrop nip ;

ok> 120 f .
[0,1,4,9,16,24,25,36,40,49,60,64,76,81,84,96,100,105]

Swift, 47 35 32* bytes

* -3 thanks to @Alexander.

Possibly the first time in history Swift ties beats Python?

{m in Set((0..<m).map{$0*$0%m})}

Try it online!

Explanation

• (0..<m).map{} - Iterates through the range [0...m) and map the following results:

• $0*$0%m - The square of each integer modulo the base m.

• Set(...) - Removes the duplicates.

• m in - Assigns the base to a variable m

PHP, 53 bytes

for(;$i<$t=$argv[1];)$a[$z=$i++**2%$t]++?:print"$z
";

Loop from 0 to the input number, using the n^2 mod base formula to mark numbers that have been used. It goes to that position in an array, checking if it's been incremented and outputting it if it hasn't. It then increments it afterwards so duplicate values don't get printed.

Try it online!

Ruby, 31 30 bytes

->m{(0..m).map{|n|n*n%m}.uniq}

Try it online!

MATL, 6 5 bytes

-1 byte thanks to @LuisMendo

:UG\u

Try it online!

Japt, 7 6 bytes

Dz%UÃâ

Test it

1 byte saved thanks to Oliver

Explanation

Implicit input of integer U.

Ç   Ã

Create an array of integers from 0 to U-1, inclusive and pass each though a function.

²

Square.

%U

Modulo U.

â

Get all unique elements in the array and implicitly output the result.

R, 28 bytes

unique((1:(n<-scan()))^2%%n)

Try it online!

PowerShell, 35 bytes

0..($n="$args")|%{$_*$_%$n}|sort -u

Try it online!

Brachylog, 10 9 bytes

>ℕ^₂;?%≜ᶠ

Try it online!

Explanation

       ≜ᶠ       Find all numbers satisfying those constraints:
    ;?%           It must be the result of X mod Input where X…
  ^₂              …is a square…
>ℕ                …of an integer in [0, …, Input - 1]

Python 2, 59 bytes

t=int(input())
print list(set([(i*i)%t for i in range(t)]))

Try it online!

Pyth, 6 bytes

{%RQ*R

Try it online

How it works

{%RQ*RdQ    implicit variables
       Q    autoinitialized to eval(input())
    *R      over [0, …, Q-1], map d ↦ d times
      d         d
 %R         map d ↦ d modulo
   Q            Q
{           deduplicate

Pyth, 13 bytes

VQ aY.^N2Q){Y

Try online.

Lame attempt at explaining:

VQ               for N in [0 .. input-1]
                   blank space to supress implicit print inside the loop
     .^N2Q         N ^ 2 % input
   aY              append that to Y, which is initially an empty list
          )      end for
           {Y    deduplicate and implicit print

To sort the output, insert an S on any side of the {

I think there should be a shorter way...

Perl 6, 19 bytes

{set (^$_)»²X%$_}

Test it

Expanded:

{ # bare block lambda with implicit param ｢$_｣

  set        # turn the following into a Set (shorter than ｢unique｣)

      (
        ^$_  # a Range upto (and excluding) ｢$_｣
      )»²    # square each of them (possibly in parallel)

    X%       # cross modulus the squared values by

      $_     # the input
}

Haskell, 45 bytes

import Data.List
f m=nub[n^2`mod`m|n<-[0..m]]

-4 bytes from Anders Kaseorg

Try it online!

Haskell, 44 bytes

f m=[k|k<-[0..m],or[mod(n^2)m==k|n<-[0..m]]]

Try it online!

JavaScript (ES6), 52 bytes

f=(m,k=m,x={})=>k?f(x[k*k%m]=m,k-1,x):Object.keys(x)

Test cases

f=(m,k=m,x={})=>k?f(x[k*k%m]=m,k-1,x):Object.keys(x)

;[1, 2, 10, 16, 31, 120]
.map(m => console.log(m + ' -> ' + f(m)))

Non-recursive version, 60 58 bytes

Saved 2 bytes thanks to @ThePirateBay

m=>(a=[...Array(m).keys()]).filter(v=>a.some(n=>n*n%m==v))

Test cases

let f =

m=>(a=[...Array(m).keys()]).filter(v=>a.some(n=>n*n%m==v))

;[1, 2, 10, 16, 31, 120]
.map(m => console.log(m + ' -> ' + f(m)))

Clojure, 40 bytes

#(set(map(fn[x](mod(* x x)%))(range %)))

Retina, 70 bytes

.+
$*

;$`¶$`
1(?=.*;(.*))|;1*
$1
(1+)(?=((.*¶)+\1)?$)

D`1*¶
^|1+
$.&

Try it online! Warning: Slow for large inputs. Slightly faster 72-byte version:

.+
$*

$'¶$';
1(?=.*;(.*))|;1*
$1
+s`^((1+)¶.*)\2
$1
^1+

D`1*¶
^|1+
$.&

Try it online!

Scala, 32 30 bytes

Simple use of the easy tip from OP.

(0 to n-1).map(x=>x*x%n).toSet

Try it online!

-2 bytes thanks to @MrXcoder, with priorities (no need for () around * operation)

Wondering: is this possible to implicitly tell the compiler to understand things like (0 to n-1)map(x=>x*x%n)toSet (without having to import scala.language.postfixOps)?

JavaScript (ES6), 48 bytes

f=
n=>[...new Set([...Array(n)].map((_,i)=>i*i%n))]

<input type=number min=0 oninput=o.textContent=f(+this.value)><pre id=o>

43 bytes if returning a Set instead of an array is acceptable.

Mathematica, 30 bytes

Union@Table[Mod[i^2,#],{i,#}]&

Try it online!

Octave, 27 bytes

@(n)unique(mod((1:n).^2,n))

Try it online!

CJam, 12 bytes

{_,_.*\f%_&}

Anonymous block accepting a number and returning a list.

Try it online!

Explanation

_,          Copy n and get the range [0 .. n-1]
  _.*       Multiply each element by itself (square each)
     \f%    Mod each by n
        _&  Deduplicate

05AB1E, 5 bytes

Lns%ê

Try it online! or as a Test Suite

L     # Range 1 .. input
 n    # Square each
  s%  # Mod by input
    ê # Uniquify (also sorts as a bonus)

C#, 63 bytes

using System.Linq;m=>new int[m].Select((_,n)=>n*n%m).Distinct()

Try it online!

Actually, 11 bytes

;╗r⌠²╜@%⌡M╔

Try it online!

Explanation:

;╗r⌠²╜@%⌡M╔
;╗           store a copy of m in register 0
  r          range(m)
   ⌠²╜@%⌡M   for n in range:
    ²          n**2
     ╜@%       mod m
          ╔  remove duplicates

Python 3, 40 39 37 bytes

-1 byte thanks to Mr. Xcoder. -2 bytes thanks to Business Cat.

lambda m:[*{n*n%m for n in range(m)}]

Try it online!

Jelly, 5 bytes

R²%³Q

Try it online!

Explanation

R²%³Q   Main link, argument: n

R       Range from 1 to n
 ²      Square each
  %³    Mod each by n
    Q   Deduplicate