Python, 128 bytes

Thanks @xnor for -10.

def f(a,b):
 X=eval(a)+eval(b);z=0
 while any(z for x in X[:-1]if X[z:=z+1]==x):X[z-1:z+1]=[];z=0
 return[]<X
g=lambda b,a:a+b+a

Attempt This Online!

Previously:

Python, 138 bytes

def f(a,b):
 X=eval(f"[*{a},*{b}]");z=0
 while any(z for x in X[:-1]if X[z:=z+1]==x):X[z-1:z+1]=[];z=0
 return[]<X
g=lambda b,a:[*a,*b,*a]

Attempt This Online!

Takes function-prefixed input just like my quandle answer but uses lists instead of strings. Returns False for equal and True for not equal.

How?

Recklessly based on my unproven conjecture that we can replace the free product of infinite cyclic groups in the quandle task with the free product of two-element groups and leave everything else in place. This becomes quite a bit simpler because reflections are "involutions", i.e. they satisfy \$r=r^{-1}\$, IOW: say goodbye to explicit inverses. Conjugation becomes \$aba\$. Note how conjugation unlike composition preserves the property of being an involution. As a consequence, we needn't reverse the order of composite inverses. The cancellation rule becomes: Remove any even number of consecutive repeats.

Wolfram Language (Mathematica), 60 bytes

SameQ@@(#//.{a_^a_:>a,(a_^b_)^b_:>a,a_^b_^c_:>((a^c)^b)^c})&

Try it online!

Takes input as a list of two expressions, where atoms are represented as symbols (undefined variables) and the kei operation is represented as ^, e.g., {(a ^ c) ^ (b ^ c), (a ^ b) ^ c}.

It can be shortened to 56 bytes if we are allowed to swap the two arguments of the kei operation, because ^ is right-associative in Mathematica:

SameQ@@(#//.{a_^a_:>a,b_^b_^a_:>a,(c_^b_)^a_:>c^b^c^a})&

Try it online!

JavaScript, 95 87 81 bytes

-8 thanks to emanresu A

a=>b=>(p=([x,y],r=[])=>y?p(y,p(x,p(y,r))):(v=r.pop())^x?[...r,v,x]:r)(a)+""==p(b)

Attempt This Online!

Takes input in the form

type Kei =
  | [number] // positive integer representing atom
  | [Kei, Kei] // reflection of x in y

Outputs true if they are equal, and false otherwise.

Given expressions p and q, it normalizes n > p and n > q (where n is a unique atom) each into the form ((((n > a) > b) > c) ...) (with no repeated atoms, and represented as the array [a, b, c]), and compares their equality.

Charcoal, 53 bytes

Ｆ²«≔ηθＦ⁻Ｓ(⊞υ⎇⁼κ)⁺⁺§υ±¹Φ⮌⊟υμ⊟υκ≔⁺⌈υΦ⮌⊟υληＦη≔⁻η×κ²η»⁼θη

Try it online! Link is to verbose version of code. Takes two space- or newline-separated fully parenthesised inputs without spaces or operators. Explanation: Uses @tjjfvi's approach of normalising everything to an expression of the form (..(p > q)..> r), represented as r...qp.

Ｆ²«

Loop through each input.

≔ηθ

Save the previous pass, if any.

Ｆ⁻Ｓ(

Loop though the next input but ignoring (s as the operator is binary.

⊞υ⎇⁼κ)⁺⁺§υ±¹Φ⮌⊟υμ⊟υκ

If the next character is a ) then mirror the RHS and append the LHS, otherwise just push the character.

≔⁺⌈υΦ⮌⊟υλη

Mirror the final expression.

Ｆη≔⁻η×κ²η

Remove any pairs of characters.

»⁼θη

Compare the normalised values.

Retina 0.8.2, 82 bytes

{({\w+)}
$1
{(\w+){(\w+)(\w)}
{$1$3{$2}$3
{(\w)}
$1
}r`(({\3)|\3)(\w)
$2
^(.+)¶\1$

Try it online! Link is to test suite that mostly converts from the given format to what the program wants, which is expressions separated by newlines, {}s instead of ()s and no spaces or >s. Explanation: Uses @tjjfvi's approach of normalising everything to an expression of the form (..(p > q)..> r), represented as {pq..r}.

{({\w+)}
$1

{{p..q}r} becomes {p..qr}, where r can be any expression.

{(\w+){(\w+)(\w)}
{$1$3{$2}$3

{p..q{r..st}u} becomes {p..qt{r..s}tu}, where u can be any expression.

{(\w)}
$1

{p} becomes p.

r`(({\3)|\3)(\w)
$2

{pp becomes {p but other pp pairs are deleted. Note that the regex is evaluated right-to-left (as if it were a lookbehind), so that (\w) is matched first, and then the (({\3}|\3) is looked for behind it (but remains part of the match).

}`

Repeat until both expressions have been normalised.

^(.+)¶\1$

Compare the two expressions.

Python, 322 bytes

lambda a,b:{str(a),str(K(a))}&{str(b),str(K(b))}!=set()
K=lambda a:[u:=f(a),u[0]][len(u)==1]
I=lambda a:'['!=str(a)[0]
def f(a):
 if I(a):return a
 match a:
  case[[x,y],z]if y==z:a=x
  case[[x,y],z]if I(z):a=[[x,z],[y,z]]
  case[[x,y],[z,w]]if y==w:a=[[x,z],y]
  case[x,y]if x==y:a=x
 return a if I(a)else[f(i)for i in a]

Attempt This Online!