JavaScript (ES9), 105 bytes

s=>s.replace(/[()]((?<=\()\w{4}|\d?.)/g,(_,s)=>s[1]?String.fromCodePoint(parseInt(s,36)):s.toUpperCase())

Try it online!

Method

Regular expression:

[()]           // a parenthesis followed by either:
(              //
  (?<=\()\w{4} //   if it's an opening parenthesis: 4 alphanumeric characters
               //   matches: (xxxx
  |            //   or
  \d?.         //   an optional digit followed by any character
)              //   matches: ((, )), )a, )", and )xx

where the a in )a is a lower case letter and xx / xxxx are base-36 strings.

NB: Nothing else will be match provided that the input string does not contain invalid tokens, as specified in the challenge.

We don't really care about the escape character and only retrieve the string s that follows. It boils down to two cases:

• If s is at least 2 characters long, proceed with base-36 decoding:

  String.fromCodePoint(parseInt(s,36))
  

• Otherwise, return s in upper case (leaving (, ) and " unchanged):

  s.toUpperCase()
  

Japt, 35 bytes

Port of Arnauld's JS solution.

r"%)%d?.|%(%w\{0,3}."Ȥ?XÅn36 d:XÌu

Try it (includes all test cases)

r"..."Ȥ?XÅn36 d:XÌu     :Implicit input of string
r                        :Replace
 "..."                   :  RegEx /\)\d?.|\(\w{0,3}./g
      È                  :  Pass each match, X, through the following fuction
       ¤                 :    Slice off the first 2 characters
        ?                :    If truthy (non-empty string)
         XÅ              :     Slice off the first character
           n36           :     Convert from base 36
               d         :     Get character at that codepoint
                :        :   Else
                 XÌ      :     Last character
                   u     :     Uppercase

Perl 5 -MMath::Base36=:all -p, 108 bytes

s/(?<!\))\)(\pL)/\U$1/g;s/(?<!\))\)(\d.)|(?<!\()\((\w{4})/chr decode_base36$1.$2/ge;s/\)([")])|\((\()/$1$2/g

Try it online!

05AB1E, 43 bytes

¶ì„)(vJy©¡NUεDõQi®ë¬diX>·ôćAžhìÅβçšJëćuì]J¦

Try it online.

Explanation:

¶ì                           # Prepend a newline before the (implicit) input-string
  „)(                        # Push string ")("
     v                       # Loop over its characters `y`:
      J                      #  (First iteration: no-op)
                             #  Second iteration: Join the list of the previous iteration back together to a string
      y                      #  Push the current parenthesis-character
       ©                     #  Store it in variable `®` (without popping)
        ¡                    #  First iteration: split the implicit input by this character)
                             #  Second iteration: split the current string by this character
      NU                     #  And also store the current index in variable `X`
         ε                   #  Map over each part:
          DõQi               #   If the current part is empty:
              ®              #    Push the current parenthesis-character `®` instead
          ë¬di               #   Else-if the current part starts with a digit:
              X              #    Push index `X`
               >·            #    Increment and double (0 becomes 2 and 1 becomes 4)
                 ô           #    Split the string into parts of that size
                  ć          #    Extract head; push first item and remainder-list separately
                   AžhìÅβ    #    Convert it from custom base "0-9a-z" to a base-10 integer
                         ç   #    Convert that from a codepoint-integer to a character
                          š  #    Prepend it back to the list
                           J #    Join the list back together
          ë                  #   Else: it doesn't start with a digit
           ć                 #    Extract head
            u                #    Uppercase it (no-op for '"')
             ì               #    Prepend it back to the remainder-string
     ]                       # Close the if-else statements; map; and loop
      J                      # Join the list back together to a string
       ¦                     # Remove the leading newline again
                             # (after which the result is output implicitly)

Minor note: the double J (one right after v and one after ]) is shorter than a single join after closing both if-else statements and the map at the end of every loop-iteration:

      ↓                                 ↓↓
¶ì„)(vJy©¡NUεDõQi®ë¬diX>·ôćAžhìÅβçšJëćuì]J¦
¶ì„)(vy©¡NUεDõQi®ë¬diX>·ôćAžhìÅβçšJëćuì}}}J}¦
                                       ↑↑↑↑↑

Charcoal, 62 bytes

ＦＳ≡⪫υωω¿№()ι⊞υιι(≡ι⊟υι⊞υι)¿⁼ιＩΣι≔⟦ωωι⟧υ∧⊟υ↥ι¿⁼Ｌ⊞Ｏυι⁴«℅⍘υ³⁶≔⟦⟧υ

Try it online! Link is to verbose version of code. Explanation:

ＦＳ

Loop over each character of the input string.

≡⪫υω

Check which state the program is in. The program state is kept as a list as this makes it easier to reset the program state by popping the character (saving 6 bytes), but switch only works on hashable types, so the list is joined here, although the only interesting cases are the empty list and the list containing a parenthesis.

ω

If the program is in the starting state:

¿№()ι

If the current character is a parenthesis, then...

⊞υι

... set the state to that character, otherwise...

ι

... output the character.

(

If the program is in the open parenthesis state:

≡ι⊟υ

If the current character is also an open parenthesis, resetting the program state, then...

ι

... output the current character, otherwise...

⊞υι

... add this base 36 digit to the program state.

)

If the program is in the close parenthesis state:

¿⁼ιＩΣι

If the current character is a digit, then...

≔⟦ωωι⟧υ

... set the program state to waiting for the final base 36 digit, otherwise...

∧⊟υ↥ι

... reset the program state and output the current character in upper case.

¿⁼Ｌ⊞Ｏυι⁴«

Otherwise, if this is the last base 36 digit to collect, then:

℅⍘υ³⁶

Convert the state from base 36 and output that Unicode character.

≔⟦⟧υ

Reset the program state.