Nim, 328 bytes

This program solves the magic square of squares problem if you have enough memory and time:

include prelude,random
var
 n=[1,2,3,4,5,6,7,8,9]
 i,m=8
template f(a,b,c):int=n[a]^2+n[b]^2+n[c]^2
while on:(i=8;while i>=0 and n[i]>i-9+m:i-=1
if i<0:(m+=1;for j in 0..7:n[j]=j+1)else:(n[i]+=1;while(if allit([f(0,1,2),f(3,4,5),f(6,7,8),f(0,4,8),f(2,4,6),f(0,3,6),f(1,4,7)],it==f(3,5,8)):quit $n;n.nextpermutation):m=m)
n.sort)

It works by generating all combinations of 9 numbers in lexicographical order and checking if any permutation is satisfying the equation:

Ungolfed:

include prelude,random # import math, algorithm, sequtils
var
 n=[1,2,3,4,5,6,7,8,9] # square roots in flat array
 i,maxVal=8
template f(a,b,c): int = n[a]^2 + n[b]^2 + n[c]^2
while on: # on == true
  # find right-most value we can increment
  i=8; while i>=0 and n[i] > i - 9 + maxVal: i-=1
  if i<0:
    maxVal += 1
    for j in 0..7: n[j] = j+1
  else:
    n[i]+=1
    while (
      if allit([f(0,1,2),f(3,4,5),f(6,7,8),f(0,4,8),
                f(2,4,6),f(0,3,6),f(1,4,7)], it==f(3,5,8)): quit $n
      nextPermutation(n)): # mutates array and returns true on success
        discard            # nop
    sort(n)                # undo permutations

Vyxal, 7 6 Bytes

While it functions differently, i was recommended to edit my post.

Here is my new solution:

{ǎ:1o|

It starts from N = 0, Assigns N to the N'th prime, And if N in decimal is all ones, Quits.

Explanation

{ Open While loop 
ǎ Get the Nth prime. at first it uses the input, 
  but since there is no input this will be interpreted as 0
: Duplicate
1 Push 1
o Remove all 1's from the new prime. Output is all digits that are not 1 as an array
| If the empty array, exit. Otherwise, repeat

Old attempt

₁{TDd⋏|

Tries to find a \$n\$ where \$n > 0\$ such that \$100 \cdot 3^n\$ such that it in binary does not contain '11'

Explanation

₁ Push 100 to the Stack
{ Open While loop 
T Triple the number
D Push three copies to the stack
d Double (Bit shift left)
⋏ Logical and of v and v * 2
| If not zero, then 11 was found. repeat.

inspired by racheline's answer

Python, 34 bytes

a,b=8,3
while b:a+=a>>1;b+=2-a%2*3

This is a codegolfed version of the open Antihydra problem by @mxdys.

It can be expanded to

def antihydra():
    # The iterated value
    a = 8
    # Keeps track of how many odd and even numbers have been encountered
    b = 1
    # If b <= 0 there have been more than twice as many odd than even 
    # numbers in the sequence of a's, and the program halts
    while b != 0:
        if a % 2 == 0:
            b += 2
        else:
            b -= 1
        # Add the number divided by two (integer division) to itself
        a += a//2        

Try it online!

Python, 27 21 20 bytes

n=7
while n&n*2:n*=7

Try it online!

This one terminates iff there's a power of 7 (other than 1) which has no "11" in its binary representation. The problem is similar to this conjecture by Erdős.

For completeness, here's my first answer (27 bytes):

n=m=3
while m:n^=m;m+=n%5-1

Try it online!

This was inspired by the open problems that are common in small undecided Turing machines. I started from the Hydra, which can be done in 32 bytes:

n=m=3
while m:n+=n>>1;m+=n%2*3-1

This is nice, but the n%2*3 is a bit long, and replacing it with n%4 gives a problem which is just as unpredictable in 30 bytes. And n+=n>>1 is a bit long as well, but turns out we can get chaotic behavior with XOR if we also make it depend on m, though specifically with m+=n%4-1 it isn't chaotic, so we need to change that 4 to at least 5.

Vyxal, 8 bytes

λEE›æ;5ȯ

Try it Online!

Finds the fifth Fermat Prime.

λ    ;5ȯ # First 5 numbers where
 EE›     # 2^2^n + 1
   æ     # Is prime

Pyth, 13 bytes

.u@,/G2h*5GG7

Like Steven H.'s answer, but saving a few bytes by using the 5n+1 problem instead

Do the iterates of $$f(n)=\begin{cases}5n+1&\mbox{if }n\mbox{ is odd}\\\frac{n}{2}&\mbox{otherwise}\end{cases}$$ starting with 7, cycle?

While it seems unlikely, the answer is unknown!

Pushy, 13 bytes

7$h&fh&r2e=?i

Try it online!

This program attempts to find a solution to Brocard's Problem - it will keep running until it finds a \$n \ge 8\$ such that \$ n! +1=m^2, m \in \mathbb{N} \$

Brachylog (v2), 3 bytes in Brachylog's encoding

⟦cṗ

Try it online! (will time out without doing anything visible, for obvious reasons)

Full program; if run with no input, searches for the first Smarandache prime, and outputs true. if and when it finds one. It's an open question as to whether any Smarandache primes exist. (Note that Brachylog's prime-testing algorithm, although it works in theory on arbitrarily large numbers, tends to run slowly on them; thus, if you're interested in finding Smarandache primes yourself, I recommend using a different language.)

Explanation

⟦cṗ
⟦     Form an increasing range from 0 to {the smallest number with no assertion failure} 
 c    Concatenate all the numbers that make up that range, in decimal
  ṗ   Assert that the result is prime

Brachylog operates on the decimal digits of a number whenever you try to treat it like a list, so "range" followed by "concatenate" is a very terse way to generate the sequence of Smarandache numbers (and then we filter that by primality; Brachylog's default full program behaviour will then force the first element of the resulting generator). The range has a leading zero, but fortunately, with this flow pattern, Brachylog removes the zero rather than failing.

Here's an example that finds the first Smarandache number that's equal to 6 (mod 11), as a demonstration of a similar program that terminates within 60 seconds rather than having unknown halting status:

⟦c{-₆~×₁₁&}

Try it online!

This would print true. as a full program, but I threw in the Z command line argument to actually print the number in question, giving a better demonstration that this general approach works.

Jelly, 7 bytes

*+3Ẓµ4#

Try it online! (prints two elements, not 4, so that you can actually see it halt)

Prints the first four \$n\$ such that \$n^n+3\$ is prime. It is not known whether four+ numbers with this property exist. The first three are 2, 1036, and 2770.

Explanation

*+3Ẓµ4#
     4#  Find the first four numbers with the following property:
    µ      (bracketing/grouping: place everything to the left inside the loop)
*          {The number} to the power of {itself}
 +3        plus 3
   Ẓ       is prime

Pyt, 14 bytes

27*²0`ŕĐ₫+Đ₽¬ł

Port of mbomb007's answer.

Jelly, 11 9 bytes

ÆẸ⁺‘ÆPµ6#

This will terminate once the sixth Fermat prime is found.

Try it online!

How it works

ÆẸ⁺‘ÆPµ6#  Main link. No arguments. Implicit argument: 0

      µ6#  Convert the links to the left into a monadic chain and call it with
           arguments k = 0, 1, 2, ... until 6 of them return 1.
ÆẸ         Convert [k] to the integer with that prime exponent factorization, i.e.,
           into 2 ** k.
  ⁺        Repeat.
   ‘       Increment.
           We've now calculated 2 ** 2 ** k + 1.
    ÆP     Test the result for primality.

Clojure, 154 bytes

(loop[x 82001](if(= 0(reduce +(map{true 1 false 0}(for[y(range 3 6)](true?(for[z(str(range 2 y))](.indexOf z(Integer/toString x y))))))))x(recur(inc x))))

Checks if there's a number above 82,000 that only contains 0's and 1's for base 2 all the way to base 5. In other words, it checks if there's another number in this sequence.

In that special group, there are only 3 numbers: 0, 1 and 82,000. There are no more numbers that follow that rule that are less than approximately 3*10^19723.

Python, 68 bytes

n=2
while"".join(str((i+2)**n)[0]for i in range(8))!="23456789":n+=1

Try it online

Tries to answer one of Gelfand's Questions.

2. Will the row "23456789" ever appear for n>1? None does for n<=10^5. ...

Python 3, 101 bytes

I know it's longer than many others, but I spent a lot of time seeing how short I could golf this.

This attempts to disprove Euler's Sum of Powers Conjecture for k=6 (there exists no positive integer solution to the Diophantine equation A^6+B^6+C^6+D^6+E^6==F^6), for which no counterexample has been found.

R=[1]
while 1:R+=[R[-1]+1];eval(("[1/("+"+%s**6"*5+"!=%%s**6)%s]"%("for %s in R "*6))%(*"ABCDEF"*2,))

In Python 2 (104 bytes):

R=[1]
while 1:R+=[R[-1]+1];eval(("[1/("+"+%s**6"*5+"!=%%s**6)%s]"%("for %s in R "*6))%tuple("ABCDEF"*2))

Less golfed:

x=2
while 1:
    R=range(1,x)
    [1/(A**6+B**6+C**6+D**6+E**6!=F**6)for F in R for E in R for D in R for C in R for B in R for A in R]
    x+=1

Mathy version with no eval:

R=range
x=2
while 1:
    for i in R(x**6):1/(sum(map(lambda x:x**6,[1+(i%x**-~j/x**j)for j in R(6)]))-i%x-1)
    x+=1

Alternate reference: Euler's Sum of Powers Conjecture - MathWorld

Python 2, 123 98 92 bytes

p=lambda n,k=2:n<=k or n%k*p(n,k+1)
g=lambda n:[p(b)*p(n-b)for b in range(n)]and g(n+2)
g(4)

This code will terminate if the Goldbach conjecture does not hold for all even numbers (i.e. if all even numbers can be expressed as the sum of two primes). It has currently been tested for numbers up to 4*10^18.

A huge amount of thanks to @Pietu1998 for shortening my code by a lot!

EDIT: Thanks to @JonathanAllan for shaving 6 bytes off my code!

Jelly, 7 bytes

!‘Ʋµ4#

Try it online!

Background

This will terminate once it finds a fourth solution to Brocard's problem, i.e. a solution n! + 1 = m² with (n,m) ≠ (4, 5), (5, 11), (7, 71) over the positive integers. The implementation doesn't use floating point arithmetic, so it will only terminate if it does find a fourth solution or if n! can no longer be represented in memory.

Brocard's problem was first used in this answer by @xnor.

How it works

!‘Ʋµ4#  Main link. No arguments. Implicit argument: 0

    µ4#  Convert the links to the left into a monadic chain and call it with
         arguments k = 0, 1, 2, ... until 4 of them return 1.
!        Factorial; yield k!.
 ‘       Increment; yield k! + 1.
  Ʋ     Squareness; return 1 if k! + 1 is a perfect square, 0 if not.

R, 30 bytes, arguable whether it is deterministic

while(any(sample(2,654,T)>1))1

R's default random number generator has equidistribution in 653 consecutive dimensions but it is not known to in 654 dimensions. Thus there may or may not be a sequence of pseudorandom numbers that sample the lowest element from a given vector 654 times in a row (here the vector 1:2).

Since R's RNG is periodic (albeit with very long period), I claim that this is deterministic since it will eventually loop round to the start. Your opinions may differ, of course.

Python 2, 88 bytes

p=lambda n:all(n%x for x in range(2,n))
s=lambda n:0if p((10223*2**n)+1)else s(n+1)
s(0)

This code will terminate if 10223 is a Sierpiński number. 10223 is currently the smallest candidate that may or may not be a Sierpiński number, as of December 2013.

A Sierpiński number is a number k in which all numbers of the form (k * 2^n) + 1 are composite.

Python, 30 28 bytes

n=2
while 2**~-n%n**3-1:n+=1

This program will halt if and only if there is an integer n bigger than 1 such that 2^(n-1)-1 is divisible by n^3. To my knowledge it is not known whether any number with this property exists (if a number satisfying this property is prime, it is called a Wieferich prime of order 3 to base 2, and it is open whether any such prime exists).

Python, 80 bytes

Terminates if the Collatz Conjecture is proven false. See this question.

n=2
while 1:
 L=[];x=n
 while~-n:1/(n not in L);L+=[n];n=(n/2,n*3+1)[n%2]
 n=x+1

Python 2, 64 bytes

A Lychrel number is a natural number that cannot form a palindrome through the iterative process of repeatedly reversing its digits and adding the resulting numbers.

No Lychrel numbers have been proven to exist in base ten. 196 is the smallest base ten Lychrel number candidate. It has been shown that if a palindrome exists (making 196 not a Lychrel number), it would have at least a billion (10^9) digits, because people have run the algorithm that long.

n=196
while 1:
    x=str(n);r=x[::-1]
    if x!=r:n=n+int(r)
    else:1/0

Haskell, 46 bytes

[n|m<-[1..],n<-[1..m],product[1..n]+1==m^2]!!3

Terminates if it finds the 4th solution to brocard's problem.

JavaScript (ES6), 104 101 bytes

for(n=[6,9,p=1];!p;n=n.map((x,i)=>(q=n[n.length+~i],p|=x^q,c=q+x+c/10|0)%10).concat(c/10|0||[]))c=p=0

Uses the same method as the Perl answer: sets n to 196, then repeatedly adds n to its base 10 reverse until it's a palindrome in base 10. This would be shorter if JS supported arbitrary-precision numbers, but oh well.

Jelly, 9 8 bytes

-1 byte thanks to @Dennis! (use exponentiation instead of multiplication to avoid Æṣ(0))

*ḂÆṣ=µ2#

Will return a list of zero and the smallest odd perfect number, if any exist.

How?

*ḂÆṣ=µ2# - Main link: no arguments
     µ   - monadic chain separation
      2# - count up from implicit `n=0` and return the first 2 truthy results of
 Ḃ       -     mod 2        -> n%2
*        -     exponentiate -> n**(n%2)  (1 when n is even, n when n is odd)
  Æṣ     -     sum of proper divisors of n**(n%2)
    =    -     equals n?    -> 1 if n is zero or both perfect and odd, else 0

Actually, 16 bytes

1`;;pY)▒@D÷íu*`╓

Try it online!

This code terminates iff there is some composite number n such that totient(n) divides n-1 (Lehmer's totient problem).

Explanation:

1`;;pY)▒@D÷íu*`╓
1`            `╓  first integer, starting with 0, where the following function leaves a truthy value on top of the stack:
    pY       *      composite (not prime) and
   ;  )▒            totient(n)
  ;     @D֒u       is in the list of divisors of n-1

Haskell, 47 bytes

[n|n<-[1..],2*n==sum[d|d<-[2..n],n`mod`d<1]]!!0

Searching the first quasiperfect number, which is a number n whose sum of divisors is 2*n+1. Instead of adding 1, I exclude 1 from the list of divisors.

MATL, 11 bytes

`@QEtZq&+=z

Terminates iff the Goldbach conjecture is false. That is, the program stops if it finds an even number greater than 2 that cannot be expressed as the sum of two primes.

`        % Do...while
  @      %   Push iteration index k. Gives 1, 2, 3, ...
  QE     %   Add 1 and multiply by 2. Gives 4, 6, 8, ...
  tZq    %   Duplicate. Push all primes up to current k
  &+     %   Matrix with all pairwise additions of those primes
  =z     %   Number of entries of that matrix that equal k. This is used as loop
         %   condition. That is, the loop continues if this number is nonzero
         % Implicit end

05AB1E, 8 bytes

Will terminate when the 6th Fermat prime is found.

5µNoo>p½

Explanation

5µ          # loop over increasing N (starting at 1) until counter reaches 5
  Noo       # 2^2^N
     >      # + 1
      p½    # if prime, increase counter

Python, 92 bytes

This isn't winning any code golf competitions, and it requires infinite memory and recursion depth, but this is an almost perfect opportunity to plug an interesting problem I asked on math stackexchange two years ago, that no Fibonacci number greater than 8 is the sum of two positive cubes. Funnily enough, it started as an code golf challenge idea, so I guess I've come full circle.

def f(i,j):
 r=range(i)
 for a in r:
  for b in r:
   if a**3+b**3==i:1/0
 f(j,i+j)
f(13,21)

Brain-Flak, 212 208 204 bytes

This program uses a multiplication algorithm written by MegaTom and a non-square checker written by 1000000000

Try It Online

(((()()()()){})){{}((({}()))<{(({})[()])}{}>[()]){({}<({}<>)({<({}[()])><>({})<>}{}<><{}>)>[()])}{}(({}())){(({}[()]<>)<>)(({({})({}[()])}{}[({})]<>)){{}{}({}<>)(<([()])>)}{}({}()){(((<{}{}<>{}>)))}{}}{}}

This program starts at 8 and tests each number to see if n!+1 is a square number. It exits when it finds one. This is known as Brocard's Problem and it is an open problem in mathematics.

Perl, 50 38 36 34 33 bytes

$_=196;$_+=$%while($%=reverse)-$_

Explanation: 196 is a possible Lychrel number - a number that does not form a palindrome by repeatedly adding its reverse to itself. The loop continues until $n is equal to its reverse, which is as yet unknown for initial value 196.

25 + 52 = 77

which is not valid.

96 + 69 = 165
165 + 561 = 726
726 + 627 = 1353
1353 + 3531 = 4884

so none of the numbers in this sequence are valid.

Edit: Golfed it down by using an until loop instead of a for loop (somehow). Also, I had fewer bytes than I thought (I should probably look at my bytecount more carefully in the future).

Edit: Replaced $n with $_ to save 2 bytes for the implied argument in reverse. I think this is as golfed as this implementation is going to get.

Edit: I was wrong. Instead of using until($%=reverse)==$_ I can go while the difference is nonzero (i.e. true): while($%=reverse)-$_.

Python, 36 bytes

k=n=1
while(n+1)**.5%1+7/k:k+=1;n*=k

Uses Brocard's problem:

Is n!+1 a perfect square for any n≥8?

Computes successive factorials and checks whether they are squares and have k>7. Thanks to Dennis for 2 bytes!

This assumes Python continues to have accurate arithmetic for arbitrarily large numbers. In actual implementation, it terminates.

Pyth, 10 bytes

fP_h^2^2T5

Uses the conjecture that all Fermat numbers 2^(2^n)+1 are composite for n>4.

f        5   Find the first number T>=5 for which
   h^2^2T    2^(2^T)+1
 P_          is prime                   

Pyth, 16 bytes

f!}1.u@,/G2h*3GG

Returns the first value for which the Collatz conjecture does not hold. As it is unknown whether the conjecture holds for all numbers, it is unknown whether this code will terminate.