JavaScript (Node.js), 100 ^{108 99} bytes

+9 bytes because 0 was considered like NaN
New tests added to handle special cases provided with Shaggy and l4m2 's help
-8 bytes thanks to Shaggy

Outputs NaN if the expression is invalid.
In this answer, we'll consider MR followed by M+ (or M-) invalid

s=>s.replace(/\d+|M[-+R]?/g,x=>v=(y=v!=v|v)/x?x*y:x[1]?(x>"MQ"?M*=0:eval(x+"="+v))-s:M*y,M=0,v=1)&&M

Try it online!

• Uses the same regex as Conor O'Brien 's JS answer
• The result of replace is never used. It's the shortest function allowing to use a regex and an arrow function on a string
• The regex separates the string into the elements [numbers] M M+ M- and MR
• The result of the function will be returned via the M variable with the &&M at the end. The variable is named exactly M because we use an eval at some point
• x is the current element parsed in the string parameter, and v contains the current computation of [numbers] and M
• (...)/x? checks if x is a number
• x[1]? checks if the second character of x exists (to differentiate between M and M+ M- MR)
• x>"MQ"? is used to differentiate between MR and M+ M- (using alphabetical order)
• v will contain NaN after each operation (M+ M- or MR). It serves as some kind of "poison" that will go into M if M+ or M- are used at an inappropriate moment
• if v contains NaN, any [number] or M can "cure" it with v!=v|v (stored in y). This code is also necessary to differentiate between NaN and 0
• if M contains NaN, it will contain it forever (to detect and remember if the expression is invalid. Which is why we have to do M*=0 instead of M=0 so that MR can't erase the memory of an invalid expression). NaN is particularly useful here because NaN remains NaN if used in any computation.
• eval(x+"="+v) uses M+ and concatenates it to construct M+=[number in v] (same with M-)
• -s is used to generate NaN in 2 bytes, and some parenthesis are used to put this NaN in v after any of M+ M- MR
• v is initialized with 1 because it's the neutral element in multiplication

CASIO BASIC (CASIO fx-9750GIII), 280 bytes

CASIO function on a CASIO calculator that doesn't have the function, lol.

I'm the only person that's coding in CASIO BASIC here now, and it's sad :(

?→Str 1
For 1→I To StrLen(Str 1
1→P
StrMid(Str 1,I,1→Str 2
StrMid(Str 1,I,2→Str 8
If StrSrc("0123456789",Str 2
Then For 1→K To StrLen(Str 1
Not StrCmp("M",StrMid(Str 1,I+K,1))⟹Break
Next
Exp(StrMid(Str 1,I,K))(O+Not O→O
I+K-2→I
0→P
IfEnd
O≠0
If Not StrCmp("M+",Str 8
Then O+MAns→M
0→O~P
IfEnd
If Not StrCmp("M-",Str 8
Then MAns-O→M
0→O~P
IfEnd
Not StrCmp("MR",Str 8)⟹0→M~P
I+Not P→I
PNot StrCmp("M",Str 2)⟹MO+MNot O→O
Next
M

outputs 0 if the string is invalid.

JavaScript (Node.js), 146 bytes

s=>eval(s.replace(/\d+|M[-+R]?/g,b=>`_${c?"*":""}=${b<":"?(d=c++,b):d&&b[1]?"m=``//":`m${(d=b[1])||";a"}=_;${c=!d,b>"M-"&&"m=0"}`};`,m=c=d=0)+"m")

Try it online!

Creates a program based on the input and then evaluates it. Returns the empty string for invalid inputs. My favorite aspect of this solution is using b<":" to separate numeric strings from the other kinds of strings that can appear here.

JavaScript (Node.js), 138 bytes

s=>eval(s.replace(/\d+|M[-+R]?/g,b=>`_${c?"*":""}=${b<":"?(d=c++,b):d&&b[1]||`m${(d=b[1])||";a"}=_;${c=!d,b>"M-"&&"m=0"}`};`,m=c=d=0)+"m")

Try it online!

Similar to @Mukundan314's reasoning, this version throws an error instead of returning a specific value.

Python, ^{172 179 174 169 164 156} 153 bytes

def f(x,M=0):
 try:exec(s("M(?=\d|M)","M*",s("(\d|M)M","\\1*M",s("R=","*=0M",s("(.*?)(M[-+R])",r"\2=\1;",x))))+"\nprint(M)")
 except:0
import re
s=re.sub

Attempt This Online!

-3 bytes thanks to @The Thonnu

Python, ^{148 155 150 145 140} 137 bytes

lambda x,M=0:exec(s("M(?=\d|M)","M*",s("(\d|M)M","\\1*M",s("R=","*=0M",s("(.*?)(M[-+R])",r"\2=\1;",x))))+"\nprint(M)")
import re
s=re.sub

Attempt This Online!

Shorter version which throws an error for invalid expressions.

+7 bytes to both versions due to bug in code.

Charcoal, 78 72 bytes

≔¹ηＦ⪪⁺RθM«¿⊙ι№-+Rκ«≡§ι⁰-¿υ≧⁻Πυζ≔⁰η+¿υ≧⁺Πυζ≔⁰η≔⁰ζ≔⟦⟧υ≔Φιλι»⊞υζ¿ι⊞υＩι»¿ηＩζ

Try it online! Link is to verbose version of code. Explanation:

≔¹η

Start by assuming the expression is valid.

Ｆ⪪⁺RθM«

Split the input on M and loop over the parts, however prefix an R to the first part to get a free memory reset.

¿⊙ι№-+Rκ«

If this part contains a -, + or R, then:

≡§ι⁰

Switch over the first character in the part.

-¿υ≧⁻Πυζ≔⁰η

If it's a - then subtract the product of the stack from the memory, but if the stack is empty then mark the expression as invalid.

+¿υ≧⁺Πυζ≔⁰η

Similarly for +.

≔⁰ζ

Otherwise, this is an R, so clear the memory.

≔⟦⟧υ

Clear the stack.

≔Φιλι

Remove the leading character from the part.

»⊞υζ

Otherwise, push the memory to the stack.

¿ι⊞υＩι

If the part isn't empty then push its value to the stack.

»¿ηＩζ

If the expression was valid then output the memory.

Retina, 248 bytes

\d+|M
¶$&
\d+
*
{`^(.*)¶(_*¶)?MR|^.*¶M[-+](¶.*)*

+`^(.*)(¶-?_*¶?)M(?!.)
$1$2$1
^(.*¶-?)(_*)¶(-?)(_*)(?!.)
$1$3$.2*$4
--

^(.*)¶(.*)¶M([-+])
$1$3$2
\+-
-
--
+
^(_*)\+|^(-_*)-|^-(_*)\+\3|^(-_*?)(_*)\+\5
$1$2$4
)`(_+)-\1(\B)?
$#2*-
L$`^(-?)(_*)
$1$.2

Try it online! Link includes test cases. Explanation:

\d+|M
¶$&

Insert newlines before each number or memory operation. The newline before the very first operation serves to set the initial memory to zero.

\d+
*

Convert all the numbers to unary.

{`
)`

Repeat until all of the numbers and memory operations have been processed.

^(.*)¶(_*¶)?MR|^.*¶M[-+](¶.*)*

MR will reset just the memory and M- or M+ in an invalid place will clear everything resulting in an output of zero.

+`^(.*)(¶-?_*¶?)M(?!.)
$1$2$1

M either immediately after an MR, M- or M+, or with a number in between, will read the memory.

^(.*¶-?)(_*)¶(-?)(_*)(?!.)
$1$3$.2*$4
--

Two values (at least one of which will have been the result of a memory read) will multiply together.

^(.*)¶(.*)¶M([-+])
$1$3$2
\+-
-
--
+
^(_*)\+|^(-_*)-|^-(_*)\+\3|^(-_*?)(_*)\+\5
$1$2$4
(_+)-\1(\B)?
$#2*-

M+ and M- after the value will add it to or subtract it from the memory.

L$`^(-?)(_*)
$1$.2

Convert the final memory to decimal, ignoring any pending value.

Vyxal D, 40 bytes

⁽±ḊƛǍ[f;f(nw‛MR‛¥⁼½dĿ÷Ė‟n±ß¥£!3=ß*)¥D⌊=*

Test suite. Returns blank in the case of an invalid expression.

Explanation:

⁽±Ḋ                                      # group digits and non-digits together, preserving order
   ƛǍ[f;                                 # split into list of chars for non-digit items
        f                                # flattened
         (                        )      # for each item:
          nw                             #  item, wrapped in a single list
            ‛MR                          #  literal "MR"
               ‛¥⁼½d                     #  list ["¥¥","⁼⁼"] (D flag: do not decompress string)
                    Ŀ                    #  transliterate
                     ÷Ė                  #  unwrap and execute the item:
                                         #    M: push register twice
                                         #    R: always 0, also makes stack length to 1
                                         #    + & -: add and subtract respectively
                                         #      concatenates with input if stack is insufficient
                                         #    <number>: push <number> as int
                       ‟                 #  rotate the item to the back of stack
                        n±ß¥             #  if the current item is numeric, push register
                                         #    (preventing the number from being stored)
                            £            #  store to register
                             !3=ß*       #  if stack length == 3, multiply
                                   ¥     # push register
                                    D⌊=* # multiply depending on being an integer
                                         # implicit output

05AB1E, 52 47 bytes

0U…+-RS©D¶«.:¶¡ε®'Mì„X+…Xs-‚…)\0ª'U«:'M„XP:.V}X

Try it online or verify all test cases.

Explanation:"

0U            # Set variable `X` to 0 (it's 1 by default)
…+-RSD¶«.:¶¡  # Split the (implicit) input after each "+"/"-"/"R":
…+-RS         #  Push triplet ["+","-","R"]
     ©        #  Store it in variable `®` (without popping)
      D       #  Duplicate it
       ¶«     #  Append a newline to the values in the pair
         .:   #  Replace all "+"/"-"/"R" with "+\n"/"-\n"/"R\n" respectively in the
              #  (implicit) input
           ¶¡ #  Then split on newlines
ε             # Map over each substring:
              # (used as a foreach, where all will have their own separated stack):
 ®            #  Push triplet ["+","-","R"] from variable `®`
  'Mì        '#  Prepend an "M" in front of each
 „X+          #  Push string "X+"
     ‚        #  Pair it with
 …Xs-         #  String "Xs-"
     ª        #  And also append
 …)\0         #  String ")\0"
      'U«    '#  Append an "U" after all three: ["X+U","Xs-U",")\0U"]
 :            #  Replace all "M+"/"M-"/"MR" with "X+U"/"Xs-U"/")\0U" respectively
 'M„XP:      '#  Then replace all remaining "M" with "XP"
 .V           #  Execute it as 05AB1E code (see below)
}X            # After the map: push value `X`
              # (which is output implicitly as result)

)\0U          # MR:
)             #  Wrap the entire stack into a list
 \            #  Discard it
  0U          #  Reset `X` to 0

X+U           # M+:
X+            #  Add `X` to the top value
  U           #  Pop and store it as new value `X`

Xs-U          # M-:
X             #  Push `X`
 s            #  Wrap the top two values on the stack
  -           #  Subtract the value from `X`
   U          #  Pop and store it as new value `X`

XP            # Remaining M:
X             #  Push value `X`
 P            #  Get the product of all values on the stack (for the current group)

Python3, 287 bytes:

import re
R=re.findall
V=lambda m,x,k=1:k if[]==x else V(m,x[1:],k*(int(x[0])if x[0].isdigit()else m))
E=lambda x,M=0:M if[]==x else E(x[1:],M+[-1,1][x[0][1][-1]!='-']*V(M,R('\d+|M',x[0][0]))*(x[0][1]!='MR'))
f=lambda x:''.join(map(''.join,k:=R('([M\d]+)(M(?:\-|\+)|MR)',x)))==x and E(k)

Try it online!