Uiua, 35-5+1=31 points

⤸:↯⊙@O⊟⟜÷⊢⊏⍏⌵:°⊟⍉▽¬±⊃◿(⍉⊟⟜-⟜÷)+1⇡..

Try it in the pad!

Outputs a matrix of "O"s, the second argument is whether to transpose or not.

AWK, 100 bytes

{for(;i++<sqrt($1);)$1/i==int($1/i)&&x=i;for(j=$1;j>0;j--&&s=s"\n")for(k=$1/x;k;k--&&j--)s=s"X"}$0=s

Try it online!

{for(;i++<sqrt($1);)           # test up to sqrt
$1/i==int($1/i)&&x=i;          # test if it's a whole divisor
for(j=$1;j>0;j--&&s=s"\n")     # columns
for(k=$1/x;k;k--&&j--)s=s"X"}  # row string
$0=s                           # set output

Vyxal, 14 - 10 - 5 = -1 byte

KIht‛OX¹Ẏẇ?ß∩⁋

Try it Online!

        Ẏ      # Extend
    ‛OX        # "OX"
       ¹       # To length of input
         ẇ     # Cut into chunks of length
 Iht           # Middle element of
K              # divisors
          ?ß   # If the input (to rotate) is truthy
            ∩  # Transpose
             ⁋ # And finally join by newlines

Charcoal, 33 bytes - 10 - 5 = 18

Ｎθ≔⌊Φ⊕θ¬∨‹×ιιθ﹪θιηＥ÷θη⭆η§XO⁺ιλ¿Ｎ⟲

Try it online! Link is to verbose version of code. Explanation:

Ｎθ

Input N.

≔⌊Φ⊕θ¬∨‹×ιιθ﹪θιη

Take the range 0..N, keep only the numbers whose squares are not less than N and divide N, and take the minimum of those numbers.

Ｅ÷θη⭆η§XO⁺ιλ

Use the discovered factor to output a rectangle of the appropriate width and height using a chequerboard pattern. (This should be ＵＯη÷θηXO¶OX for a 1-byte saving but that's buggy right now.)

¿Ｎ⟲

If the second input is nonzero then rotate the output. (If requiring the second input to be 0 or 2 is acceptable, then this could be ⟲Ｎ for a 1-byte saving.)

Retina 0.8.2, 66 bytes + 1 byte penalty = 67

.+
$*X
((^|\3)(X(?(3)\3)))+(\3)*$
$3 $3$#4$*X
X(?=X* (X+))| X+
$1¶

Try it online! Explanation:

.+
$*X

Convert the input to a string of Xs.

((^|\3)(X(?(3)\3)))+(\3)*$

The first pass of the outer capture matches the start of the string while on subsequent passes the previous value of the inner capture is matched. The inner capture is then incremented and matched. The upshot of this is that the amount of string consumed by the outer capture is the square of the inner capture, which therefore cannot exceed the square root of the input. Meanwhile the subsequent repetition ensures that the inner capture is a factor of the length of the string.

$3 $3$#4$*X

Save the discovered factor and calculate the other divisor by adding on the number of subsequent repetitions.

X(?=X* (X+))| X+
$1¶

Rearrange the factors into a rectangle.

05AB1E (legacy), score: 7 (22 bytes - 15 bonus)

„OXI∍¹tï[D¹sÖ#<}äIiø}»

Try it online or verify some more test cases.

Takes the inputs N first, then the boolean (0/1) whether it should rotate or not.

Uses the Python legacy version of 05AB1E since zip with a string-list implicitly flattens and joins the characters, unlike the newer Elixir rewrite version of 05AB1E.

Explanation:

„OX         # Push string "OX"
   I∍       # Extend it to a size equal to the first input
            #  i.e. 9 → "OXOXOXOXO"
            #  i.e. 10 → "OXOXOXOXOX"
¹t          # Take the first input again, and square-root it
            #  i.e. 9 → 3.0
            #  i.e. 10 → 3.1622776601683795
  ï         # Then cast it to an integer, removing any decimal digits
            #  i.e. 3.0 → 3
            #  i.e. 3.1622776601683795 → 3
   [        # Start an infinite loop:
    D       #  Duplicate the integer
     ¹sÖ    #  Check if the first input is evenly divisible by that integer
            #   i.e. 9 and 3 → 1 (truthy)
            #   i.e. 10 and 3 → 0 (falsey)
        #   #  And if it is: stop the infinite loop
    <       #  If not: decrease the integer by 1
            #   i.e. 3 → 2
   }        # After the infinite loop:
ä           # Divide the string into that amount of equal sized parts
            #  i.e. "OXOXOXOXO" and 3 → ["OXO","XOX","OXO"]
            #  i.e. "OXOXOXOXOX" and 2 → ["OXOXO","XOXOX"]
 Ii }       # If the second input is truthy:
   ø        #  Zip/transpose; swapping rows/columns of the strings
            #   i.e. ["OXOXO","XOXOX"] → ["OX","XO","OX","XO","OX"]
»           # And finally join the strings in the array by newlines
            #  i.e. ["OXO","XOX","OXO"] → "OXO\nXOX\nOXO"
            #  i.e. ["OX","XO","OX","XO","OX"] → "OX\nXO\nOX\nXO\nOX"
            # (and output the result implicitly)

J, 32 bytes - 15 = 17 bytes

'XO'$~[|.](%,])i.@]{~0 i:~i.@]|]

Try it online!

The rotation is controlled by a 0/1 flag taken as the left argument

APL (Dyalog Unicode), 30 - 15 = 15 bytes^SBCS

Anonymous infix lambda. Takes N as right argument and param as left argument. Rectangles will either have stripes of X and O or be chequered.

{⍉⍣⍺⍴∘'XO'⊃∘c⌈.5×≢c←⍸⍵=∘.×⍨⍳⍵}

Try it online!

{…} "dfn"; ⍺ is left argument (param), ⍵ is right argument (N):

 ⍳⍵ ɩndices 1…N

 ∘.×⍨ multiplication table of that

 ⍵= mask where N is equal to that

 ⍸ ɩndices of true values in the mask

 c← store that in c (for candidates)

 ≢ tally the candidates

 .5× one half multiplied by that

 ⌈ ceiling (round up)

 ⊃∘c pick that element from c

 ⍴∘'XO' use that to cyclically reshape "XO"

 ⍉⍣⍺ transpose if param

Petit Computer BASIC, 72 bytes

INPUT N,S$FOR I=1TO SQR(N)IF N%I<1THEN M=I
NEXT
?(S$*M+" "*(32-M))*(N/M)

Ruby, 74

f=->n{w=(1..n).min_by{|z|n%z>0?n:(n/z-n/(n/z))**2};$><<("X"*w+"\n")*(n/w)}

Explanation

• Input is taken as arguments to a lambda. It expects an Integer.
• Check if n (the input) is divisible by every integer from 1 to n.
  • If it is, calculate the difference between the length and width.
  • If it is not, return a large number (n).
• Take the smallest of length-width differences to best resemble a square.
• Use (the overly concise) String#* method to "draw" the square.

Mathematica, 71 chars

f@n_:=#<>"\n"&/@Array["O"&,{#,n/#}&[#[[⌊Length@#/2⌋]]&@Divisors@n]]<>""

GolfScript 26 (41 - 10 - 5)

:x),1>{x\%!},.,2/=.x\/@{\}*'X'*n+*1>'O'\+

Expects two parameters to be on the stack:

• 0 for normal or 1 for transposed
• the n value

The pattern is that the board is full of Xs and the top left corner is an O. Needless to say, this pattern is maintained when transposing the board.

Demo: regular, transposed

CJam, 16 (31 - 10 - 5)

This takes two integers are input, first one being 0 or 1 for direction and second one being the number of O or X in the grid.

It prints an alternate O and X.

:X"OX"*X<\Xmqi){(_X\%}g_X\/?/N*

This is just the function body, to try it out add l~ in front of the code like:

l~:X"OX"*X<\Xmqi){(_X\%}g_X\/?/N*

and give input like

0 10

to get output like

OXOXO
XOXOX

or input like

1 10

for

OX
OX
OX
OX
OX

Try it online here

How it works:

l~                                 "Put the two input integers to stack";
  :X                               "Assign the number of cells to X";
    "OX"*                          "Take string "OX" and repeat it X times";
         X<                        "Slice it to take only first X characters";
           \                       "Swap top two stack elements, now string is at bottom";
            Xmqi)                  "Take square root of X, ceil it and put on stack";
                 {(_X\%}g          "Keep decrementing until it is perfectly divisible by X";
                         _X\/      "Copy it, divide X by that and put it on stack";
                             ?     "Based on first input integer, take either of numbers";
                              /    "Divide the XOXO string that many times";
                               N*  "Join the string parts with a new line";

Example run:

l~ed:X"OX"*edX<ed\edXmqi)ed{(_X\%}ged_edXed\ed/ed?ed/edN*ed

#INPUT:
1 10

#OUTPUT:
Stack: [1 10]

Stack: [1 "OXOXOXOXOXOXOXOXOXOX"]

Stack: [1 "OXOXOXOXOX"]

Stack: ["OXOXOXOXOX" 1]

Stack: ["OXOXOXOXOX" 1 4]

Stack: ["OXOXOXOXOX" 1 2]

Stack: ["OXOXOXOXOX" 1 2 2]

Stack: ["OXOXOXOXOX" 1 2 2 10]

Stack: ["OXOXOXOXOX" 1 2 10 2]

Stack: ["OXOXOXOXOX" 1 2 5]

Stack: ["OXOXOXOXOX" 2]

Stack: [["OX" "OX" "OX" "OX" "OX"]]

Stack: ["OX
OX
OX
OX
OX"]

OX
OX
OX
OX
OX

Python, 79 75 (no bonuses)

The bonuses seem tricky, so here is a pretty simple Python function:

def f(N):c=max(x*((x*x<=N)>N%x)for x in range(1,N+1));print(N/c*'O'+'\n')*c

JavaScript (E6) 84 (83+1) or 101 (116-10-5)

Pattern + rotation (parameter f, 0 or 1) - bonus 15

F=(n,f)=>{
  for(r=x=0;y=n/++x|0,x<=y;)x*y-n?0:z=f?x:y;
  for(o='';n;)o+=(n--%z?'':(r^=1,c='\n'))+'OX'[r^(c^=1)];
  alert(o)
}

No pattern, no rotation - penalty 1

F=n=>{
  for(x=0;y=n/++x|0,x<=y;)x*y-n?0:z=y;
  alert(('O'.repeat(z)+'\n').repeat(n/z));
}

Test In FireFox/FireBug console

F(30,0)

OXOXOX
XOXOXO
OXOXOX
XOXOXO
OXOXOX

F(30,1)

OXOXO
XOXOX
OXOXO
XOXOX
OXOXO
XOXOX

APL (36 - 5 - 10 = 21)

{'OX'⍴⍨⍺⌽⊃∆/⍨⍵=×/¨∆←∆[⍋|-/¨∆←,⍳2/⍵]}

The left argument is rotation, the right argument is the size. It also uses a simple pattern (it just alternates 'X' and 'O').

      0{'OX'⍴⍨⍺⌽⊃∆/⍨⍵=×/¨∆←∆[⍋|-/¨∆←,⍳2/⍵]}¨4 5 8 9
 OX  OXOXO  OXOX  OXO 
 OX         OXOX  XOX 
                  OXO 
      1{'OX'⍴⍨⍺⌽⊃∆/⍨⍵=×/¨∆←∆[⍋|-/¨∆←,⍳2/⍵]}¨4 5 8 9
 OX  O  OX  OXO 
 OX  X  OX  XOX 
     O  OX  OXO 
     X  OX      
     O       

Explanation:

• ∆←,⍳2/⍵: generate all possible pairs of numbers from 1 to ⍵ and store in ∆.
• ∆←∆[⍋|-/¨∆...]: sort ∆ ascending in the absolute difference of the two numbers in each pair, and store the result back in ∆.
• ⊃∆/⍨⍵=×/¨∆: for each pair, multiply the numbers together. Select only those pairs that multiply to ⍵, and take the first one that matches (which is the 'most square' because of the sort).
• ⍺⌽: rotate the list of lengths (which has 2 elements) by ⍺.
• 'OX'⍴⍨: create a matrix of that size, and fill it with alternating O and X.

CJam, 25 22 21 (31 - 10)

This is a function body. If you want a complete program, add riri to the front. If you want to use it as a code block, surround it in {}. Test it on cjam.aditsu.net.

It takes input as two integer arguments: the switch for whether the rectangle is vertical (any non-zero value) or horizontal (zero), and the number of Os to use.

:Xmqi){(_X\%}g_X\/@{\}{}?'O*N+*

Explanation

:X "Assign the top item on the stack (the second input) to variable X";
mq "Take its square root";
i  "Convert to integer (round)";
)  "Increment it";

{  "Start code block";
  (  "Decrement";
  _X "Duplicate top item on stack; push X to the stack";
  \% "Swap top 2 items and take division remainder";
}g "Loop until top item on stack is 0; pop condition after checking it";

_X "Duplicate top item on stack; push X to the stack";
\/ "Swap top 2 items and divide";

"OMIT THIS BIT TO GET A 25-CHAR FUNCTION WITHOUT THE 10PT BONUS";
 @  "Rotate top 3 items on stack";
 {\}"Code block 1: swap top two items";
 {} "Code block 2: do nothing";
 ?  "If top item of stack is 0, run code block 1, otherwise run code block 2";

'O "Push the character O to the stack";
*  "Repeat it N times, where N is the second item from the top of the stack (O is first)";
N+ "Push a new line and concatenate it with the string on the top of the stack";
*  "Repeat the string N times";

Haskell, 59 characters

r=replicate
f n=[r x$r y '0'|x<-[1..n],y<-[1..x],x*y==n]!!0