Zsh, 100 bytes

for i ({1..$1})echo&&for j ({1..$1})echo -n ${${(#)$(((i==j)|(j+i==$1+1)?RANDOM%93+33:32))}/[Yy-]/:}

• for i prints $1 lines. for j iterates the chars.
• If we're on the X, get a random number, otherwise 32 [space].
• Convert ascii-value to text. Change any of Yy- to colon :, and print.

Try it online!   _{115b 122b 130b}

CASIO BASIC (CASIO fx-9750GIII), 216 bytes

"!#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXZ[\\]^_`abcdefghijklmnopqrstuvwxz{|}~"→Str 1
?→A
A-(A Rmdr 2→E
1→D
For 0→B To E
RanInt#(1,91
Text 6B+1,4D+1,StrMid(Str 1,Ans,1)
RanInt#(1,91
B≠.5E⟹Text 6B+1,4D+8+4(E-2D),StrMid(Str 1,Ans,1)
D-1+2(B<.5E→D
Next

lol

Pip -l, 25 bytes

{a?RCPADC" Yy"s}MMQPEYHUa

Try It Online!

Explanation

{a?RCPADC" Yy"s}MMQPEYHUa
                        a ; Command-line argument
                      HU  ; Halve and round up
                    EY    ; Identity matrix of that size
                  QP      ; Quad-palindromize
                          ; This gives an X of 1s against a background of 0s
{              }MM        ; Map this function to elements of elements:
 a?                       ;  If the argument is truthy (1):
   RC                     ;   Randomly choose from
     PA                   ;   printable ASCII
       DC" Yy"            ;   with those characters deleted
              s           ;  Otherwise, space
                          ; Autoprint, each row concatenated on a separate
                          ; line (-l flag)

Original (slightly incorrect) solution, 32 bytes

I missed the fact that the X is also not supposed to contain any spaces. That could be fixed for +1 byte by changing RM-`y` to DC" Yy" like in the updated answer.

Strangely enough, the code begins with Y and ends with y...

Ya{$=a|$+a=y-1?RCPARM-`y`s}MMCGy

Takes input as a command-line argument. Try it online!

Explanation

Constructs a grid of the appropriate size; replaces elements on the diagonals with a random non-y character, and all other elements with space.

                                  a is 1st cmdline arg; PA is printable ASCII characters;
                                  s is space (implicit)
Ya                                Yank a into y (global var, accessible within functions)
                             CGy  y by y coordinate grid
  {                       }MM     To each coordinate pair, map this function:
   $=a                             Fold on equality (true if both elements are equal)
      |                            Logical OR
       $+a                         Fold on +
          =y-1                     and test if equal to size - 1
              ?                    If the preceding expression is true, then:
                 PARM               From printable ASCII chars, remove         
                     -`y`           regex matching y, case-insensitive
               RC                   Take a random choice from the resulting string
                         s         Else, space
                                  The whole expression returns a nested list, which is
                                  autoprinted as lines of concatenated items (-l flag)

Uiua, 33 31 bytes

+@ ×+1++∩>86,55.⌊×92∵⋅⚂.↥⇌.⊞=.⇡

Try it

Look ma, no y!

See also RomanPro100's solution which does the same thing but better

Uiua 0.10.0, 29 bytes

+@ ++∩≥57,88.⌈×92×∵⋅⚂.↥⇌.⊞=.⇡

Try it yourself

Vyxal (canvas), 121 bits^v2, 15.125 bytes

½⌈?dkP‛YyFƈṅ\xø∧

Try it Online!

Bitstring:

0001001010010001111011001101111101001110110100101000000011001100001010010000110110000101011011000011110111101111101100110

Vyxal 2.6, 22 bytes

½⌈:ʁ\%꘍↲∞v∞⁋?dkP‛yYFƈ%

Try it Online!

½⌈:                    # Push ⌈x/2⌉ twice
   ʁ                   # For each 0...^-1
    \%꘍                # Prepend that many spaces to a %
½⌈     ↲               # Pad to length ⌈x/2⌉ with spaces
        ∞v∞            # Quad palindromise
           ⁋           # Join by newlines
                    ƈ  # Choose...
            ?d         # Input * 2
                    ƈ  # Random items from...
              kP       # Printable ASCII
                ‛yYF   # Without Y
                     % # Replace % in the first bit with the random chars

Japt -R, 25 22 bytes

A much messier approach that somehow worked out shorter than the original. The need for the j1 is annoying me but this is the closest I can get otherwise, without tanking my score (the middle line is wrong).

;Æ2ÆEÅkÕöÃqSp´UaX¹j1Ãû

Try it

;Æ2ÆEÅkÕöÃqSp´UaX¹j1Ãû     :Implicit input of integer U
 Æ                         :Map each X in the range [0,U)
  2Æ                       :  Map the range [0,2)
;   E                      :    ASCII
     Å                     :    Slice off the first character (space)
      k                    :    Case insensitively remove
       Õ                   :      "y"
        ö                  :    Random character
         Ã                 :  End map
          q                :  Join with
           S               :    Space
            p              :    Repeat
             ´U            :      Prefix decrement U
               aX          :      Absolute difference with X
                 ¹         :  End join
                  j1       :  Remove character at 0-based index 1
                    Ã      :End map
                     û     :Centre pad each element with spaces to the length of the longest
                           :Implicit output joined with newlines

Original (w/o flag), 25 bytes

;z ôÈç iÕêÃÔê û ·ry@EÅkÕö

Try it

;z ôÈç iÕêÃÔê û ·ry@EÅkÕö     :Implicit input of integer
 z                            :Floor divide by 2
   ô                          :Range [0,result]
    È                         :Map each X
     ç                        :  X spaces
       i                      :  Prepend
        Õ                     :    "y"
         ê                    :  Palindromise
          Ã                   :End map
           Ô                  :Reverse
            ê                 :Palindromise
              û               :Centre pad each element with spaces to the length of the longest
                ·             :Join with newlines
                 ry           :Replace "y"s
                   @          :  Pass each match through the following function
;                   EÅkÕö     :    As above

Vyxal mM, 34 bytes

‹½ƛ\%꘍⁰↲øm;øm⁋?‹4*›ƛ‛yY95ɾ31+CF℅;%

Try it Online!

‹½ƛ       ;                        # Map 0...(n-1)/2...
   \%꘍                             # That many spaces before a %
      ⁰↲                           # Left-padded to correct width
        øm                         # Palindromise
           øm                      # Palindromise
             ⁋                     # Join by newlines
              ?‹4*›                # 4(n-1)+1
                   ƛ            ;  # Map 0...n...
                       95ɾ31+C     # Printable ASCII
                    ‛yY            # Push ys
                              F    # Remove ys
                               ℅   # Choose a random element from this
                                 % # Format the cross by this

Pyth, 28 27 26 25 bytes

jmuXGHO-rF"!~""Yy"{,d-tQd*;Q
VQuXGHO-rF"!~""Yy"{,N-tQN*d
VQuXGHO-r\!\~"Yy"{,N-tQN*d
VQuXGHO-r\!\~"Yy",N-tQN*d
VQuXGHO-r\!\[DEL];"Yy",N-tQN*d

Try it online!

PHP, 100 bytes

for(;($x%=$n=$argv[1])?:$y++<$n&print"\n";)echo strtr(chr($y+$x++-$n&&$x-$y?32:rand(33,126)),yY,zZ);

takes input from command line argument; run with -nr.

combined loop prints characters depending on position

breakdown

for(;
    ($x%=$n=$argv[1])       // inner loop
        ?
        :$y++<$n&print"\n"  // outer loop; print newline
;)
    echo strtr(chr(             // 2. replace Y with Z; print
        $y+$x++-$n&&$x-$y       // 1: if position is not on diagonals
            ?32                 // then space
            :rand(33,126)       // else random printable
    ),yY,zZ);

SmileBASIC, 97 bytes

INPUT S
FOR X=1TO S
FOR Y=1TO S
Q=RND(93)+33?CHR$((Q+!(Q-121&&Q-89))*(X==Y||X+Y==S+1));
NEXT?NEXT

Instead of having to calculate the number of spaces between each character or something, I decided to just print in all locations where X==Y or X+Y==Size+1.
The random character generator just adds 1 if it generates y or Y, so z and Z are slightly more common than usual.

Python, 142 139 135 bytes

This is a straight forward implementation create the square character by character. If the character is on a diagonal: use a random char, else: use a space. This also uses a regex substitution and random int to generate non-Y characters:

import re,random
lambda x:''.join('\n'*(i%x<1)+re.sub("y|Y","t",chr(random.randint(33,126))+' ')[i%x!=i/x!=x-i%x-1]for i in range(x*x))

Explanation [ Old ]

"\n".join( ... for i in range(x)) # Create 'x' lines 
''.join( ... for j in range(x))   # Create 'x' chars on each line
(...)[j!=i!=x-j-1]                # Not on diagonals? 2nd char in "? "; Else, choose the 1st
j!=i                              # Not on downward diagonal
i!=x-j-1                          # Not on upward diagonal
re.sub("y|Y","t", ... )           # Replace y or Y for t
chr(random.randint(33,126))+' '   # Random char + a space

Update

• -4 [16-07-30] Shortened newline conditional
• -3 [16-07-30] Changed to single for-loop
• -6 [16-07-29] Exchanged if statement for ternary op. Thanks to @RootTwo
• -11 [16-07-27] Removed extra brackets/spaces and flipped if statement
• -49 [16-07-27] Absorded @squeamishossifrage's method by creating the square step-by-step, Thanks!
• -10 [16-07-27] Shorten random char lambda + math stuff from @ConorO'Brien
• -22 [16-07-26] Squeaze in a lambda + misc golfing
• -6 [16-07-26] import* - Thanks to @KevinLau

x86 machine code, 70 bytes

60 89 d7 31 db 43 88 ce b2 fe 49 d1 e1 87 da 0f
c7 f0 24 7f 3c 22 72 f7 48 3c 79 74 f2 3c 59 74
ee aa 49 7c 1c 00 df 79 06 86 f7 42 43 eb f6 f6
c3 01 74 03 b0 0a aa 51 88 f9 b0 20 f3 aa 59 eb
cc c6 07 00 61 c3

My executable code, disassembled:

0000003d <myheh>:                                       
  3d:   60                      pusha                   
  3e:   89 d7                   mov    %edx,%edi        
  40:   31 db                   xor    %ebx,%ebx        
  42:   43                      inc    %ebx             
  43:   88 ce                   mov    %cl,%dh          
  45:   b2 fe                   mov    $0xfe,%dl        
  47:   49                      dec    %ecx             
  48:   d1 e1                   shl    %ecx             

0000004a <myloop>:                                      
  4a:   87 da                   xchg   %ebx,%edx        

0000004c <myrand>:                                      
  4c:   0f c7 f0                rdrand %eax             
  4f:   24 7f                   and    $0x7f,%al        
  51:   3c 22                   cmp    $0x22,%al        
  53:   72 f7                   jb     4c <myrand>      
  55:   48                      dec    %eax             
  56:   3c 79                   cmp    $0x79,%al        
  58:   74 f2                   je     4c <myrand>      
  5a:   3c 59                   cmp    $0x59,%al        
  5c:   74 ee                   je     4c <myrand>      
  5e:   aa                      stos   %al,%es:(%edi)   
  5f:   49                      dec    %ecx             
  60:   7c 1c                   jl     7e <mydone>      

00000062 <mylab>:                                       
  62:   00 df                   add    %bl,%bh          
  64:   79 06                   jns    6c <myprint>     
  66:   86 f7                   xchg   %dh,%bh          
  68:   42                      inc    %edx             
  69:   43                      inc    %ebx             
  6a:   eb f6                   jmp    62 <mylab>       

0000006c <myprint>:                                     
  6c:   f6 c3 01                test   $0x1,%bl         
  6f:   74 03                   je     74 <myprint1>    
  71:   b0 0a                   mov    $0xa,%al         
  73:   aa                      stos   %al,%es:(%edi)   

00000074 <myprint1>:                                    
  74:   51                      push   %ecx             
  75:   88 f9                   mov    %bh,%cl          
  77:   b0 20                   mov    $0x20,%al        
  79:   f3 aa                   rep stos %al,%es:(%edi) 
  7b:   59                      pop    %ecx             
  7c:   eb cc                   jmp    4a <myloop>      

0000007e <mydone>:                                      
  7e:   c6 07 00                movb   $0x0,(%edi)      
  81:   61                      popa                    
  82:   c3                      ret                     

It is a function that receives the size of the X in ecx, and a pointer to the output buffer in edx.

It fills the output buffer sequentially with bytes. There are 2 * n - 1 iterations (equal to the number of non-space characters to output). At each iteration, it does the following:

• Generate a random number
• Fiddle with the number to fit it into range; if it's bad, go back and generate anew
• Print the random character
• Print a newline (every other iteration)
• Print the proper number of spaces

Conversion from a random number to a random character is not remarkable:

myrand:
    rdrand eax;
    and al, 7fh;
    cmp al, 22h;
    jb myrand;
    dec eax;
    cmp al, 'y';
    je myrand;
    cmp al, 'Y';
    je myrand;

The interesting part is the calculation of the number of spaces. It must generate the following numbers (example for N=9):

7    1
5    2
3    3
1    4

     3
1    2
3    1
5    0
7

The numbers are taken alternatingly from two arithmetic progressions. The first one goes down with step -2, and the second one goes up with step 1. When the first progression arrives at -1 (in the middle of the X), there is a glitch (-1 is removed), and then the progressions change direction.

The progressions are stored in registers ebx and edx - the high parts bh and dh store the current number, and the low parts bl and dl store the step. To alternate between the progressions, the code swaps the registers with xchg.

When the progression arrives at -1 (around the mylab label), it increases both registers, switching the steps from -2, 1 to -1, 2. This also changes the roles of the registers, so then it swaps the high parts of the registers.

At the end of the function, it stores a zero byte to indicate an end of string.

C, 154 bytes (or 119 without the boiler-plate)

o(w,c){c=rand()%94+33;printf("%*c",w,w?c+!(c&95^89):10);}main(h){scanf("%d",&h);srand(time(0));for(int n=h,p;n--;)p=abs(h/2-n),o(h/2-p+1),p&&o(p*2),o(0);}

Or 119 bytes as a function X(h) with srand(time(0)) taken care of elsewhere:

o(w,c){c=rand()%94+33;printf("%*c",w,w?c+!(c&95^89):10);}X(h,n,p){for(n=h;n--;)p=abs(h/2-n),o(h/2-p+1),p&&o(p*2),o(0);}

Breakdown:

o(w,c){                         // "Output" function, for all printing
    c=rand()%94+33;             // Generate random char, whether we need it or not
    printf("%*c",               // Print a char with some number of leading spaces
           w,                   // Use "w" (width) - 1 leading spaces
           w?                   // Either print the random char...
             c+!(c&95^89)       // (exclude "y" and "Y" by incrementing to "z"/"Z")
                         :10    // ...or print a newline if called with w = 0
    );
}
main(h){                        // Main function; repurpose argc to store grid size
    scanf("%d",&h);             // Get grid size from stdin
    srand(time(0));             // Boiler-plate for random number seeding
    for(int n=h,p;n--;)         // Loop over all lines (count down to save chars)
        p=abs(h/2-n),           // Calculate half-distance between "X" bars
        o(h/2-p+1),             // Output the first half of the "X" (">")
        p&&                     // If we are not in the centre:
           o(p*2),              //   output the second half of the "X" ("<")
        o(0);                   // Output a newline
}

MATLAB, 86 bytes

a=@(n)(char(changem(randi(92,n),33+[0:55 57:87 89:93],1:92).*(eye(n)|fliplr(eye(n)))))

Some examples:

>> a(1)

ans =

i


>> a(3)

ans =

~ {
 Z 
* ^


>>a(5)

ans =

k   E
 | M 
  }  
 ] s 
b   t


>> a(10)

ans =

Q        k
 +      a 
  j    w  
   X  [   
    rO    
    %3    
   P  d   
  K    q  
 r      & 
?        v

Python 2, 204 191 183 bytes

Okay, Python competition here is getting fierce. Here's my attempt on shaving as many bytes as possible. By now I'm stuck (Ok, stuck again).

Credits to @NonlinearFruit for the way random characters are selected.

183 byte version:

import re,random
s=i=input();t=lambda:re.sub("y|Y","t",chr(random.randint(33,126)))
while i>-s:i-=2;u=abs(i);z=(s-u)/2-1;print('',' '*-~z+t()+'\n')[-1==i]+(' '*z+t()+' '*u+t())*(i>-s)

Try It Online! (Ideone)

Main change is to rewrite the conditional

(" "*(z+1)+t()+"\n"if -1==i else"") 

as

(""," "*-~z+t()+'\n')[-1==i]

which saves 7 bytes.

191 byte version:

import re,random
s=i=input();t=lambda:re.sub("y|Y","t",chr(random.randint(33,126)))
while i>-s:
 i-=2;u=abs(i);z=(s-u)/2-1;print(' '*(z+1)+t()+'\n'if -1==i else'')+(' '*z+t()+' '*u+t())*(i>-s)

Try It Online! (Ideone)

Main changes are the way random characters are selected and some code rearrangement such as s=input();i=s; becoming s=i=input();, removing the r=range assignment as it is no longer needed and calling abs directly as it results in less bytes of code.

Beating the previous shortest answer in Python by 1 byte! @R. Kap 's approach is used for generating the random characters. Each iteration of the while loop a row of the ex is printed.

204 byte version:

from random import*
s=input();i=s;a=abs;r=range;t=lambda:chr(choice(r(33,89)+r(90,121)+r(122,128)))
while i>-s:
 i-=2;z=(s-a(i))/2-1;print(' '*(z+1)+t()+'\n'if -1==i else'')+(' '*z+t()+' '*a(i)+t())*(i>-s)

Try It Online! (Ideone)

Ungolfed version to get an idea of how it works:

from random import *
random_character = lambda : chr(choice(range(33,89)+range(90,121)+range(122,128)))

size = input()
current_row = size

while current_row > -size:
    current_row-=2
    n_leading_spaces = (size-abs(current_row)/2)-1 
    row_to_print = ''
    if current_row == -1:
        row_to_print = ' ' * (n_leading_spaces+1) + random_chr() + '\n'
    if current_row > -size:
        row_to_print += ' ' * n_leading_spaces + random_chr()+' '*abs(current_row)+random_chr()
    print row_to_print

It was hard to handle the 1-character case!

Matricks, 79 bytes (non-competing)

Matricks excels as the beginning of making the x and all the random values, but flops when it comes to conditionals...

I marked this as noncompeting because I had to fix a few bugs and get all the new features working after this challenge was posted.

m:n;:1;mr=c:L:L;k({}|{X;})*{m_?33:126;;:L:l;miC<121,89>:gr:c;;:49:gr:c;;:L:l;};

Run with python matricks.py x.txt [[]] <input> --asciiprint

Explanation:

m:n;:1;mr=c:L:L;                   #Initialize matrix to be a square with
                                   #a diagonal of 1s
k...;                              #Set the output to...
({}|{X;})*                         #The boolean x matrix multiplied by...
{m_?33:126;;:L:l;                  #A bunch of random characters
miC<121,89>:gr:c;;:49:gr:c;;:L:l;} #But make sure they are not y or Y

This also supports even numbers.

Python 2, 171 bytes

from random import*
def x(n):
 w=range(-n/2+1,n/2+1)
 for i in w:
  o=''
  for j in w:c=randint(33,124);c+=(c>88)+(c>119);c=[c,32][bool(i^j and i^-j)];o+=chr(c)
  print o

Guaranteed to choose random characters with uniform probability.

Try it out here: ideone link

EDIT: Thanks to Morgan Thrapp for the corrections.

C, 268 bytes

V(c){for(c=89;c==89||c==121;c=rand()%95+33);return c;}p(n,s){n^1?s-n?printf("%*.c",s-n,32):0,printf("%c%*.c%c\n",V(),n*2-3,32,V()),p(n-1,s),s-n?printf("%*.c",s-n,32):0,printf("%c%*.c%c\n",V(),2*n-3,32,V()):printf("%*.c%c\n",s-n,32,V());}f(n){srand(time(NULL));p(n,n);}

Call f() with the size of the x to draw.

PowerShell v2+, 112 bytes

Param($i)function f{Random(33..126-ne121-ne89|%{[char]$_})};1..$i|%{$a=,' '*$i;$a[$_-1]=f;$a[$i-$_]=f;$a-join''}

Reads input off the command line.

For each line, an array of spaces is created, the correct indices filled in with characters pulled from function f, then the char array is joined to output as one line.

Actually, 62 bytes

"!⌂"♂┘ix♂c"Yy"@-╗½≈u;r2@∙`i=`M╪k`;dXR@+`M;dXR@+`"╜J' aI"£MΣ`Mi

This is one of the longest Actually programs I've ever written.

Try it online!

Explanation:

Part 1: setting up the character list

"!⌂"♂┘ix♂c"Yy"@-
"!⌂"              push the string "!⌂"
    ♂┘            CP437 ordinal of each character ([21, 127])
      ix          range(21, 127)
        ♂c        character at each ordinal (list of printable ASCII characters)
          "Yy"@-  set difference with ["Y", "y"] (printable ASCII except "Y" and "y")

Try it online!

Part 2: constructing the boolean array for an X

½≈u;r2@∙`i=`M╪k`;dXR@+`M;dXR@+
½≈u;                            two copies of int(input/2)+1
    r                           range
     2@∙                        Cartesian product with itself
        `i=`M                   for each sublist: push 1 if both elements are equal, else 0
             ╪k                 split into int(input/2)+1-length chunks
                                (at this point, we have one quarter of the X)
               `;dXR@+`M        mirror each sublist (one half of the X)
                        ;dXR@+  mirror the entire list (the whole X)

Try it online!

Part 3: picking random characters

`"╜J' aI"£MΣ`Mi
`"╜J' aI"£MΣ`M   for each row:
 "╜J' aI"£M        for each column:
  ╜J                 push a random value from the character list
    '                push a space
      a              invert the stack
       I             take the character if the value is 1, else take the space
           Σ       concatenate the strings
              i  flatten the list and let implicit output take care of the rest

Try it online!

Dyalog APL, 35 bytes

⎕UCS 32+(⊢+∊∘57 89)⌊?95×(⊢∨⌽)∘.=⍨⍳⎕

⎕ prompt for number
⍳ 1 through that number
∘.=⍨ equality table (i.e. the diagonal has 1s)
(⊢∨⌽) itself OR its mirror image (gives both diagonals)
95× multiply by 95
? rand int between 1 and 95 for the diagonals, rand float between 0 and 1 for the rest
⌊ floor to get rid of the floats
(⊢+∊∘57 89) add one to the elements that are a member of {57,89} (Yy – 32)
32+ add 32 to make the 0s into spaces, and other numbers into the proper range
⎕UCS convert to text

TryAPL!

JavaScript (ES6), 137 131 125 bytes

n=>[...Array(n)].map((_,i,a)=>String.fromCharCode(...a.map((r=Math.random()*94,j)=>i-j&&i+j+1-n?32:(r+72&95&&r)+33))).join`\n`

Where \n represents the literal newline character. Edit: Saved 1 byte by moving the ' ' inside the String.fromCharCode expression. Saved 5 bytes by making my random character generation non-uniform; the expression r+72&95 is zero for the values that map to Y and y and an ! is generated in their place. Saved 4 bytes when I realised that spreading over String.fromCharCode avoids having to join. Saved 2 bytes by stealing a trick from @edc65.

Lua, 277 Bytes

Well... Lua is sooooo good at manipulating strings :D. First time I had to use local in a statement! I could save some bytes by using Lua 5.1 instead of 5.3 because they moved the global function unpack into the object table at Lua 5.2. But I prefer to stick with the latest version I have :).

Defines a function that should be called with a single parameter (the second one is used for recursion purpose) and returns a string.

function f(n,N)N=N or n e=" "p="\n"r=math.random C=e.char
R={}for i=1,4 do x=r(33,126)R[i]=x~=89 and x~=121 and x or r(33,88)end
local s,S,a,b,c,d=e:rep((N-n)/2),e:rep(n-2),table.unpack(R)return
n<2 and s..C(a)..p or s..C(a)..S..C(b)..s..p..f(n-2,N)..s..C(c)..S..C(d)..s..p
end

Ungolfed

function f(n,N)                       
  N=N or n                          -- N is equal to the n we had on the first call
  e=" "                             -- shorthand for the space
  p="\n"                            -- shorthand for the newline
  r=math.random                     -- shorthand for math.random
  C=e.char                          -- uses the string e to obtain the function string.char
  R={}                              -- define an array for our random values
  for i=1,4                         -- iterate 4 times (for the random characters)
  do
    x=r(33,126)                     -- random between ASCII "!" and "~"
    R[i]=x~=89 and x~=121           -- if we didn't pick y or Y
           and x                    -- keep this number
         or r(33,88)                -- or roll for a character between "!" and "X"
  end
  local s,S                         -- these variables have to be local
          ,a,b,c,d                  -- or the recursion would change them
         =e:rep((N-n)/2),e:rep(n-2) -- s and S are the number of spaces for the X
           ,table.unpack(R)         -- a,b,c and d are the 4 random characters
  return n<2                        -- if n==1 
           and s..C(a)..p           -- we're at the center of the X, time to end recursion
         or                         -- else
           s..C(a)..S..C(b)..s..p   -- concatenate the topmost line for n
           ..f(n-2,N)               -- with the inner X
           ..s..C(c)..S..C(d)..s..p -- and the bottom line
end

JavaScript (ES6), 128 131

Edit 3 bytes saved thx @Neil

So bulky, probably not the best approach. Bonus - it works with odd or even input.

n=>[...Array(n)].map((_,i,z)=>String.fromCharCode(...z.map((r=1+Math.random()*94,j)=>32+(j==i|j==n+~i&&(r+7&31?r:25))))).join`
`

F=n=>[...Array(n)].map((_,i,z)=>String.fromCharCode(...z.map((r=1+Math.random()*94,j)=>32+(j==i|j==n+~i&&(r+7&31?r:25))))).join`\n`

Z=_=>{
    o=F(+S.value),O.textContent=o,/y/i.test(o)||setTimeout(Z,100)
}
setTimeout(Z,100)

<input id=S value=15 type=number>
<pre id=O></pre>

Emacs Lisp, 269 bytes

(defalias'n'number-sequence)(set'c(mapcar'string(delq 89(delq 121(n 33 126)))))(defun c()(nth(random(length c))c))(defun s(x)" ")(defun x(l)(let((s(mapcar's(n 1 l))))(dotimes(i l)(set'x(copy-seq s))(setf(nth i x)(c)(nth(-(length x)i 1)x)(c))(message(apply'concat x)))))

Ungolfed and slightly modified:

(defvar c (mapcar 'string (delq 89 (delq 121 (number-sequence 33 126)))))
(defun c() (nth (random (length c)) c))
(defun s(x)" ")
(defun x(l)
  (let ((s(mapcar's(n 1 l)))
        x)
    (dotimes (i l)
      (set 'x (copy-seq s))
      (setf (nth i x) (c)
            (nth (- (length x) i 1) x) (c))
      (message (apply 'concat x)))))

php, 135 bytes

<?php for(;$i<$n=$argv[1];){$s=str_pad('',$n);$s[$i?:0]=chr(rand(33,126));$s[$n-++$i]=chr(rand(33,126));echo str_ireplace(Y,X,"$s
");}

Fairly straightforward approach uses str_pad to make a string of spaces of the required length, replaces the necessary characters with random ones then replaces any Ys (case insensitive) with Xs and echos the line.
Generates 2n+3 notices but, as usual, that's fine.

MATL, 28 bytes

6Y2'Yy 'X-iZr1MZrXdwXdP2$X>c

Try it online!

All the allowed characters have the same probability of appearing. Works for even input too.

6Y2     % Predefined literal of ASCII chars from 32 to 126
'Yy '   % Not allowed chars
X-      % Set difference. Produces the set of allowed chars
i       % Input number, n
Zr      % Random sample without replacement. Gives a string with n chars taken from 
        % the allowed set
1MZr    % Do the same
Xd      % Diagonal matrix. Zeros will be displayed as spaces
wXd     % Diagonal matrix with the other string
P       % Flip vertically
2$X>    % Maximum of the two matrices
c       % Convert to char. Implicitly display

Python 2.7, 205 bytes:

from random import*;C=input()/2;S=' ';R=range;Z=lambda:chr(choice(R(33,89)+R(90,121)+R(122,128)));T=lambda*G:''.join([S*i+Z()+S*(2*(~-C-i)+1)+Z()+S*i+'\n'for i in R(*G)]);print T(C)+S*C+Z()+'\n'+T(~-C,-1,-1)

Try It Online! (Ideone)

Ruby, 102 bytes

Array#sample doesn't do repetitions for sampling from the character set, but that's OK because the character distribution don't have to be perfectly uniform! Recursive function, returns an array of lines.

Try it online!

f=->l{w,x,y,z=([*?!..?~]-%w"y Y").sample 4
l<2?[w]:[w+(s=' '*(l-2))+x,*f[l-2].map{|e|" #{e} "},y+s+z]}

Mathematica, 146 bytes

a:=RandomChoice[33~CharacterRange~126~Complement~{"Y","y"}];StringRiffle[Normal@SparseArray[{{b_, b_}:>a,{b_,c_}/;c-1==#-b:>a},{#,#}," "],"
",""]&

Anonymous function. Takes a number as input, and returns a string as output.