05AB1E, 31 bytes

ª€gàLãʒvøy"üÿ"δ.V}εε˜ágΘ}}à}€Pà

Input as a matrix of characters. Outputs an empty string instead of 0 if the input lacks any X.

Try it online or verify (almost) all test cases (the largest test case is omitted, since it times out).

Explanation:

ª           # Append the (implicit) input-matrix to the (implicit) input-matrix
 €g         # Get the length of each inner list (aka all rows, and the matrix itself)
   à        # Pop and push the maximum
            # (which is either the amount of rows or amount of columns, whichever is larger)
    L       # Pop and push a list in the range [1, maxDimension]
     ã      # Pop and get all possible pairs of this list
            # (these are all possible rectangle dimensions)
ʒ           # Filter this list of pairs by:
 v          #  Pop and loop over the current pair:
  ø         #   Zip/transpose; swapping rows/columns
            #   (which will use the implicit input-matrix in the first iteration)
   y        #   Push the current dimension
    "üÿ"    #   Push string "üÿ", where `ÿ` is automatically replaced with the integer
        δ   #   Map over each inner row:
         .V #    Evaluate and execute the `üy` as 05AB1E code,
            #    pushing an overlapping lists of size `y`
 }          #  After the inner loop, we have all rectangles of the current dimensions
  εε        #  Nested map over each rectangle of values:
    ˜       #    Flatten it to a single list
     á      #    Only keep all letters (aka, the "X"s)
      g     #    Pop and push the length
       Θ    #    Check that this length equals exactly 1
  }}à       #  After the nested inner map: check if any is truthy
 }€P        # After the filter: get the product of each remaining pair†
    à       # Pop and keep the maximum (or "" if the list is empty)
            # (which is output implicitly as result)

† Note: the € before the P is for inputs that lack any X. For those inputs, the list is empty after the filter, causing the Pà to incorrectly become []→1→1 instead of []→[]→"" when using €Pà.

Jelly, 17 bytes

ZẆ$⁺€Ẏ⁺€ċ”X⁼ɗƇ1ZL

A monadic Link that accepts a list of lists of . and X characters and yields an integer - the maximal rectangular area containing exactly one X where zero means no such area exists.

Try it online!

How?

ZẆ$⁺€Ẏ⁺€ċ”X⁼ɗƇ1ZL - Link: list of lists of characters, the "Board"
   ⁺€             - apply to {Board} and then for each element of the result apply:
  $               -   last two links as a monad:
Z                 -     transpose
 Ẇ                -     all non-empty, contiguous sublists
      ⁺€          - apply to {that} and then for each element of the result apply:
     Ẏ            -   tighten
             Ƈ    - keep those for which:
            ɗ 1   -   last three links as a dyad - f(String, 1):
         ”X       -     literal 'X'
        ċ         -     count occurrences of {'X'} in {String}
           ⁼      -     {that} equals {1}?
               Z  - transpose
                L - length

K (ngn/k), 35 34 33 bytes

|/(#(1-/88=)_,/)'4(|',/,'\'$:)/,:

Try it online!

Returns 0 if the input has no X.

                               ,:   enlist
                 4(          )/     through a full rotation:
                   |'  ,'/'$:         rotate
                     ,/  \                  d prefixes
|/(            )'                   max of:
    (1-/88=)_                         (drop if ≠1 X)
   #         ,/                       area

Enlisting the characters with $ ensures that ,'/ transposes correctly. Try it online!

Dyalog APL, 51 45 39 chars

{⌈/∊⍵∘{{(×/⍴⍺↓⍵)×1=+/∊'X'=⍵}⌺⍵⊢⍺}¨,⍳⍴⍵}

Edit: I realized not only do I not need to explicitly filter out the edge cases, but its more correct to include them.

Explanation, separated into helper functions for clarity:

windows←,∘⍳∘⍴  ⍝ Generates all possible window sizes
               ⍝     inside the rectangular input  
        ,      ⍝ Ravel (turn a table into an array)
          ⍳    ⍝ the indices of
            ⍴  ⍝ the shape of the input

validSize←{(×/⍴⍺↓⍵)×1=+/∊'X'=⍵}  ⍝ This is the stencil function
                                 ⍝ It determines if the rectangle is valid (only one X)
                                 ⍝ The left argument is how many cells are overhanging, would be 0,0 if no overhang
                                 ⍝ The right argument in the windowed input matrix
                    1=+/∊'X'=⍵   ⍝ Check if elements are equal to 'X', flatten, sum it, ensure it's equal to 1
           (×/⍴⍺↓⍵)              ⍝ Multiply the elements of the shape to find its area, after dropping padding (the answer we're looking for)
                   ×             ⍝ Multiply these things together, so that if the rectangle is invalid, the result becomes zero;
                                 ⍝     otherwise it is the area of the rectangle

{⌈/∊⍵∘{validSize⌺⍵⊢⍺}¨windows ⍵}
                     ¨windows ⍵    ⍝ For each window size
      {validSize⌺⍵⊢⍺}              ⍝     run the stencil over the original input
    ⍵∘                             ⍝ Bind the original input as the left arg (⍺ in the above function)
 ⌈/∊                               ⍝ Flatten and find the max

JavaScript (Node.js), 121 bytes

f=S=>!!S.replace(/./g,c=>X+=c>[++L],L=X=0)<X?Math.max(...[/.*\n?/,/\n?.*$/,/^./mg,/.$/mg].map(r=>f(S.replace(r,'')))):X*L

Try it online!

-2 bytes by l4m2.

Expect a string with ., X, and \n.

Python, 90 bytes

f=lambda A:max(map(f,[A[1:],A[:-1],A[:,1:],A[:,:-1]]))if(T:=(A=="X").sum())>1else A.size*T

Attempt This Online!

Expect a np.array with "." and "X".

Uiua, 27 bytes

/↥×≡(/↥=₁♭⊃⧈(⧻⊚)/×)♭₂⍜¯⇡⊃△¤

Tests

The basic idea is to generate every possible rectangle, find its area, and then filter that list by whether the rectangle has only one X

Brachylog, 24 bytes

{ṇ₂ẹs\scLọ∋["X",1]∧Ll}ᶠ⌉

Try it online!

Expects input as a string with ., X, and \n.

Explanation

{                    }ᶠ⌉    Find max solution of:
 ṇ₂                           Input split by lines,
   ẹ                          each line to list of chars.
    s\s                       Get a transposed submatrix,
       cL                     concatenate it; set it to L.
         ọ                    Get counts of each letter in L.
          ∋["X",1]            Assert it has ["X", 1].
                  ∧Ll         Output length of L.

The s\s\ trick for getting submatrices is stolen from another question. We don't need the actual submatrix though, so we just concatenate.

Maple, 171 bytes

proc(M)map(n->combinat:-choose([$n,$n],2),[op(1,M)]);max(0,map(z->`if`(numboccur((y:=[seq(M[z[]])]),"X")=1,nops(y),NULL),[seq(seq(map(`..`@op,[i,j]),i=%[1]),j=%[2])]))end;

proc(M)                      # accepts an n x m Matrix of "X" and "."
map(n->combinat:-choose([$n,$n],2), # all different pairs from
                             # [1..n,1..n] so it includes [i,i]                                     
      [op(1,M)]);            # map for each dimension
# Now have: [ [[1,1],[,2],..,[n,n]], [[1,1],[1,2],..,[m,m]] ]
max(0,                       # default value if no "X"
   map(z->                   # for each z = [i..j,k..l]
                             # use as indexing to extract submatrix
                             # if one "X" then return num. elements
      `if`(numboccur((y:=[seq(M[z[]])]),"X")=1,nops(y),NULL),
                             # map works on ranges made from above
       [seq(seq(map(`..`@op,[i,j]),i=%[1]),j=%[2])]
      )
   )
end;

MATL, 30 25 bytes

&fh!"1G88=@1$l&2Y+m@p*vX>

Try at MATL online! Or verify all test cases.

Explanation

The code first turns characters . and X into 0 and 1 respectively.

It then computes 2D convolution (the key to success) with rectangular masks of all sizes (m,n) up to the input size, filled with the value 1. If the convolution output contains 1, meaning that some of the shifted versions of the mask covered exactly one X, then m·n is a potential result.

The code outputs the maximum (computed cumulatively) of 0 and all those potential results.

        % Implicit input: M×N character array containing 'X' and '.'
&f      % Two-output find: gives row and column indices of each character in
        % the input, as column vectors      
h       % Concatenate horizontally. Produces an (M*N)×2 matrix
!       % Transpose. Columns are all [m; n] with m = 1,...,M and n = 1,...,N
"       % For each column, say [m; n]
  1     %   Push 1 (*)
  G     %   Push input again
  88=   %   Compare each entry with 88. Gives 1 for 'X', 0 otherwise
  @     %   Push [m; n]
  1$l   %   Matrix of ones with size m×n
  &2Y+  %   2D convolution, keeping only valid part, without edge effects (**)
  m     %   Ismember? Gives true if 1 (*) is present in (**), false otherwise
  @     %   Push [m; n]
  p     %   Product: gives m*n
  *     %   Multiply. Gives m*n if 1 was present in (**), 0 otherwise
  v     %   Concatenate vertically with accumulated result so far, if any
  X>    %   Maximum
        % Implicit end
        % Implicit display

Uiua, 41 chars

G←/◇⊂⍚⧈□⊙¤⍜¯⇡⊸⧻
/↥⍚(/↥≡(×⊃⧻(=₁/+)◇♭)G⍉)G

Pad Link

Thanks to Ruben Verg for helping define G

R, 125 119 bytes

\(x)max(apply(expand.grid(rep(list(1:nrow(x),1:ncol(x)),2)),1,\(s,r=x[s[1]:s[3],s[2]:s[4]])(sum(r=="X")==1)*length(r)))

Attempt This Online!

JavaScript (Node.js), 121 bytes

f=(x,l,u=y=0,r,d,z=~(r-l)*~(d-u),s=1)=>y=x.map((R,i)=>R.map((c,j)=>1/r?s-=!(1+c|j<l|j>r|i<u|i>d):f(x,j,i,l,u)))|s|y>z?y:z

Try it online!

Python 3, 146 bytes

lambda x:max(W*H*(1==sum(x[D-j][R-i]>'W'for j in r(H)for i in r(W)))for R in r(len(x[0]))for W in r(2+R)for D in r(len(x))for H in r(2+D))
r=range

Try it online!

Charcoal, 42 bytes

ＷＳ⊞υιＩ⌈Ｅυ⌈Ｅ⊕κ⌈Ｅι⌈ＥＥ⊕ξ⭆✂υλ⊕κ¹✂ςπ⊕ξ¹∧⁼¹№πXＬπ

Try it online! Link is to verbose version of code. Takes input as a list of newline-terminated strings. Explanation: Lots of brute force but still manages to complete the large test case on TIO.

ＷＳ⊞υι

Read in the rectangle.

Ｉ⌈Ｅυ⌈Ｅ⊕κ⌈Ｅι⌈ＥＥ⊕ξ⭆✂υλ⊕κ¹✂ςπ⊕ξ¹∧⁼¹№πXＬπ

Generate all of the subrectangles, then for each one concatenate their rows into a single string, then take the length of those strings containing exactly one X and print the longest length.