Ruby, 53 bytes

->l{l.map{|x|(5-x*5.0*~-l.size/(l.sum-x))**2>1?-x:x}}

Try it online!

Ported from Arnauld's Javascript answer, using Luis Mendo's formula.

R, 46 42 41 bytes

-4 bytes thanks to pajonk

\(L,`+`=sum)L-2*L*(5*abs(L*+L^0-+L)>+L-L)

Attempt This Online!

14 19 bytes shorter than the previous R answer. This is equivalent to the more readable

\(L,n=length(L),S=sum(L))L*(-1)^(5*abs(L*n-S)>S-L)

How it works

Element \$X_i\$ needs to be replaced by \$-X_i\$ if the following condition is met, with \$n\$ the length of the vector and \$S\$ the sum of all elements:

$$ 5\left|nX_i - S\right|>S-X_i. $$

J, 20 bytes

*_1+2*0.2>1|@-]%+/%#

Try it online!

Essentially a port of @mkst q/kdb+ solution

JavaScript (ES7), 61 bytes

This is using Luis Mendo's formula, which doesn't require an extra variable.

a=>a.map(v=>(~-a.length*v/(eval(a.join`+`)-v)-1)**2>.04?-v:v)

Try it online!

JavaScript (ES6), 64 bytes

a=>a.map(v=>(q=(eval(a.join`+`)-v)/~-a.length)*.96/v+v/q>2?-v:v)

Try it online!

Explanation

Given the current value \$v\$ and the mean \$q\$ of all other values, the straightforward test is:

$$|q-v|>q/5$$

which can also be done as:

$$(q-v)^2>(q/5)^2$$

which gives:

$$q^2+v^2-2qv>q^2/25$$ $$24/25\times q^2+v^2>2qv$$

and for \$q\neq0, v\neq0\$:

$$24/25\times q/v+v/q>2$$

Using IEEE 754 JS numbers, this test actually works for \$v=0,q\neq0\$ and \$v=0,q=0\$ as well.

APL(NARS), 30 chars

{⍵ׯ1*0.2<∣1-⍵÷(⍵-⍨+/⍵)÷¯1+≢⍵}

If I have an array ⍵ and one element of that array y, the mean without that element that we call m will be in APL notation

(y-⍨+/⍵)÷¯1+≢⍵ 

So the question it seems to me want in math notation

m-0.2*m≤y≤m+0.2*m
     

or equivalentement |y-m|≤0.2*m or because m>0 |(y/m)-1|≤0.2 or |1-(y/m)|≤0.2 or in APL notation

0.2≥∣1-y÷(y-⍨+/⍵)÷¯1+≢⍵

So if a value has 0.2<∣1-y÷(y-⍨+/⍵)÷¯1+≢⍵ I multiply that value for -1. Test:

  f←{⍵ׯ1*0.2<∣1-⍵÷(⍵-⍨+/⍵)÷¯1+≢⍵}
  f 0 0 0 0
0 0 0 0 
  f 1 1 1 1
1 1 1 1 
  f 2 2 2 1
2 2 2 ¯1 
  f 2 2 2 3
2 2 2 ¯3 
  f 11 11 8
11 11 ¯8 
  f 10 9 6 9 9 10 10 9 9
10 9 ¯6 9 9 10 10 9 9 

Octave, 38 bytes

@(x)x-(abs(5*sum(x-x'))>sum(x)-x)*2.*x

Try it online!

Port of my Python+NumPy solution.

APL(Dyalog Unicode), 22 bytes ^SBCS

⊢ׯ1*(+/-⊢)<5×∘|+/-≢×⊢

Try it on APLgolf!

Explanation

                     ⊢   the values
                    ×    times
                   ≢     tally, the number of values
                  -      
                 /       reduce
                +/       the sum of the values
               |         absolute value
              ∘          preprocess with
             ×           times
            5×∘|         5 multiplied by absolute value of...
           <             
     (+/-⊢)             the sum of values minus these values
     (+/-⊢)<5×∘|+/-≢×⊢  the outlie condition
    *                    power                  
  ¯1                    -1                   
⊢ׯ1*                   negate only those values that fails a condition...
⊢ׯ1*(+/-⊢)<5×∘|+/-≢×⊢ solution

The outlie condition is essentially the same as presented by Robin Ryder:

$$S - X_i < 5\left|S - nX_i\right|$$

q/kdb+, 25 bytes

Solution:

{x*-1 1@.2>abs 1-x%avg x}

Tests:

f:{x*-1 1@.2>abs 1-x%avg x}
f each (0 0 0 0; 1 1 1 1; 2 2 2 1; 2 2 2 3; 11 11 8; 10 9 6 9 9 10 10 9 9)
0 0 0 0
1 1 1 1
2 2 2 -1
2 2 2 -3
11 11 -8
10 9 -6 9 9 10 10 9 9

Explanation:

{x*-1 1@.2>abs 1-x%avg x} / the solution
{                       } / lambda taking implicit x
                   avg x  / calculate the mean
                 x%       / divide input by mean
               1-         / subtract from 1
           abs            / take the absolute value
        .2>               / is less than 0.2
       @                  / index into
   -1 1                   / the list (-1 ; 1)
 x*                       / multiply x by this

Python, 60 bytes

-1 thanks to @tata.

lambda l:[[x,-x][5*abs(sum(l)-len(l)*x)>sum(l)-x]for x in l]

Attempt This Online!

Old Python, 61 bytes

lambda l:[[-x,x][5*abs(sum(l)-len(l)*x)<=sum(l)-x]for x in l]

Attempt This Online!

Old Python, 63 bytes

lambda l:[[-x,x][sum(4*l)+x<=len(5*l)*x<=sum(6*l)-x]for x in l]

Attempt This Online!

Older Python, 65 bytes

lambda l:[[-x,x][-~(n:=len(5*l))*x/6<=sum(l)<=~-n*x/4]for x in l]

Attempt This Online!

Python + NumPy, 48 bytes

lambda l:l-2*l*(5*abs(sum(l-l[1>0].T))>sum(l)-l)

Attempt This Online!

Old Python + NumPy, 53 bytes

-1 thanks to @tata.

lambda l:l-2*l*(abs(5*(l-l[:,1>0]).sum(0))>l.sum()-l)

Attempt This Online!

Old Python + NumPy, 54 bytes

lambda l:2*l*(abs(5*(l-l[:,1>0]).sum(0))<=l.sum()-l)-l

Attempt This Online!

These all assume that outliers are determined against the mean of all other numbers in the list.

Charcoal, 22 bytes

ＩＥθ⎇›×⁵↔⁻×ＬθιΣθ⁻Σθι±ιι

Try it online! Link is to verbose version of code. Explanation: Uses the same formula as @RobinRyder, but I didn't get around to writing my answer until now.

  θ                     Input array
 Ｅ                      Map over elements
           θ            Input array
          Ｌ             Take the length
         ×              Multiplied by
            ι           Current element
       ↔⁻               Absolute difference with
              θ         Input array
             Σ          Take the sum
     ×                  Multiplied by
      ⁵                 Literal integer `5`
    ›                   Is greater than
                 θ      Input array
                Σ       Take the sum
               ⁻        Subtract
                  ι     Current element
   ⎇                    If true then
                    ι   Current element
                   ±    Negated
                     ι  Else current element
Ｉ                       Cast to string
                        Implicitly print

Jelly,  16  13 bytes

-3 taking inspiration from Kevin Cruijssen's 05AB1E answer

L×,ạS÷/>.2N×o

A monadic Link that accepts a list of non-negative integers and yields a list of integers with (all) outliers negated.

Try it online! Or see the test-suite.

How?

L×,ạS÷/>.2N×o - Link: list of non-negative integers, V
L             - length of {V}
 ×            - multiply {that} by {V}
  ,           - pair {that} with {V}
    S         - sum {V}
   ạ          - {[V, V×Length(V)]} absolute difference {Sum(V)}
     ÷/       - reduce {that} by division
       >.2    - greater than 0.2?
          N   - negate
           ×  - multiply {that} by {V}
            o - logical OR {that} with {V}

05AB1E, 13 bytes

g*‚IOα`5*‹ÅÏ(

Try it online or verify all test cases.

Explanation:

With \$I\$ being the input-list and \$v\$ being each value within that list, I use the following formula check for which values \$v\$ need to be negated in the output:

$$\left\vert v-\sum(I)\right\vert < \left\vert len(I)\times v-\sum(I)\right\vert\times 5$$

g             # Get the length of the (implicit) input-list
 *            # Multiply it to each value in the (implicit) input-list
  ‚           # Pair this list with the (implicit) input-list
   IO         # Push the sum of the input-list
     α        # Get its absolute difference with each value in the pair of lists
      `       # Pop and push both lists separated to the stack
       5*     # Multiply all values in the top one by 5
         ‹    # Then check if the values in the first list are smaller than
              # those in the second
          ÅÏ  # For every position this is truthy in the (implicit) input-list:
            ( #  Negate that value
              # (after which the result is output implicitly)

Python, 117 Bytes (but more robust)

==============================

lambda l:(lambda a,y:[[-x,x][b != max(a) or b<=0] for b,x in zip(a,y)])([5*abs(sum(l)-len(l)*x)-sum(l) for x in l],l)

Attempt This Online!

So the test case I think people missed is: [10,9,1200,9,10,10,10,9,9]

Try the other code with that and you get all negative numbers. The solution I am posting builds off of Albert lang's solution but only checks the most significant outlier. I am 100% sure there are shorter ways to do this. Happy hunting :)

MATL, 19 18 bytes

nqGsG/q/q|.2>_EQG*

Try it online! Or verify all test cases.

How it works

An element \$x\$ of the input needs to be negated if and only if the following condition is satisfied, where \$n\$ is the size of the input and \$S\$ is the sum of the input:

$$ \left| \frac{n-1}{S/x-1} - 1 \right| > 0.2. $$

n    % Implicit input: number of elements
q    % Minus 1
G    % Push input again
s    % Sum
G    % Push input again
/    % Divide
q    % Minus 1
/    % Divide
q    % Minus 1
|    % Absolute value
.2   % Push 0.2
>    % Greater than? Gives 1 if true, 0 if false
_    % Negate: 1 becomes -1
E    % Times 2: -1 becomes -2
Q    % Minus 1: 0,-2 become 1,-1
G    % Push input again
*    % Multiply. Implicit display

R, 60 bytes

\(L)L*(-1)^sapply(seq(L),\(i,m=mean(L[-i]))abs(L[i]-m)>.2*m)

Attempt This Online!

Iterating on indexes of the input vector.

R, 61 bytes

\(L)Map(\(x,m=sum(L,-x)/sum(L|1,-1))x*(-1)^(abs(x-m)>.2*m),L)

Attempt This Online!

Iterating on elements of the input vector. Returns a list.

APL+WIN, 32 67 bytes

Prompts for vector of integers. Compares each element to the mean of the remainder.

v×(0=r)+r←¯1×.2<|¯1+(,(n,1)↑r)÷(+/0 1↓r←(⍳n)⌽m←((2⍴n)⍴v))÷¯1+n←⍴v←⎕

Try it online! Thanks to Dyalog Classic

CASIO BASIC (CASIO fx-9750GIII), 68 bytes

thanks arnauld for pointing out a mistake

?→List1
For 1→I To Dim List1
List1[I→E
Abs (Sum Abs List1-E)÷(Dim List1-1
.2Ans<Abs (Ans-E⇒-E→List1[I
Next
List1

try it online* at basic.crevola.org

* due to limitations in the interpreter, the code is different, but it should work the same. You have to type the list into the code in the interpreter.

cool!!