Google Sheets, 91 bytes

=sort(join(,iferror(char(code(mid(A2,row(A:A),1))*code(mid(rept(A3,999),row(A:A),1))^A1))))

Put the input in cells A1:A3 and the formula in B1.

Uses 1 for encryption and and -1 for decryption. Assumes that the text to process is 1000 characters or less, and that the key is 32 bytes or less. To ease those limits (100 bytes):

=sort(let(_,lambda(v,code(mid(v,mod(sequence(len(A2))-1,len(v))+1,1))),join(,char(_(A2)*_(A3)^A1))))

...or, to go another route using dproduct() (121 bytes):

=sort(let(r,row(A:A),c,torow(r),join(,iferror(char(dproduct({c;code(mid({A2;rept(A3,999)},c,1))^{1;A1}},r,vstack(,)))))))

PowerShell Core, 72 70 bytes

param($m,$k)$args|%{[char](""+ +$_+"*/"[$m]+ +$k[$i++%$k.Length]|iex)}

Try it online!

Takes the parameters as:

• 0 or 1, 0 to cypher, 1 to decypher
• The key as string
• The word as a list of characters using splatting

And returns an array of characters.

Builds a string using the ascii codes of the characters:

• \$char\times keychar\$ to encode
• \$char\div keychar\$ to decode

and executes it, then converts the result to a char

x64 Assembly, 172 chars

My local copy of as recognises mulb and divd which would save 10 chars, but Clang on godbolt says no, so I've submitted this as the opcodes I can showcase. This gets assembled down to 35 bytes, which is what I was really going for.

push rbx
mov rbx,rdx
a:push rbx
push rcx
b:lodsd
neg r9
jne c
mul byte ptr[rbx]
je d
c:cdq
div eax,[rbx]
d:stosd
add rbx,4
dec r8
loopne b
pop rcx
pop rbx
jne a
pop rbx
ret

I'm taking advantage of some unspecified constraints:

• The text encoding of the input and output isn't specified - I'm assuming UTF-32 because that allows me to use lodsd and stosd instead of lodsw and stosw as I would if UTF-16 was specified, which would cost an extra byte for the size override prefix
• The order of arguments isn't specified, so I've swizzled them around to have everything be in a useful position when entering the function (using the system v ABI, which also handily specifies the state of df when entering the function)
• The behavior in the case message length == 0 isn't specified, so I'm just assuming it won't happen and not handling it
• Same as above for key length == 0

The C-style interface to this function (using some C++ types) would be:

void Mulenere(char32_t* output, const char32_t* input, const char32_t* key, size_t key_length, size_t input_length, bool decode)

Thankfully that's the exact amount of arguments needed, because any more and they'd start spilling onto the stack, which would make this implementation far more painful.

Human readable version with comments:

//save rbx, use it to store key pointer.
//r10 is available as a scratch register,
//but I want one of the lower 8 registers
//to avoid a REX prefix on the mul and div
    push rbx
    mov rbx, rdx
save_key:
//push the original values of the key length
//and key pointer to the stack, so we can restore
//them later if we run out key
    push rbx
    push rcx
main_loop:
//load the character into eax, increment input pointer
    lodsd
//check r9 for zero/nonzero. Neg is the shortest
//named opcode that accomplishes this
    neg r9
    jne decode
encode:
    mulb [rbx]
    je store
decode:
    cdq //zero rdx, input must be at most 0x3FFF so sign bit of eax is zero
    divd [rbx]
store:
//store character in eax, increment output pointer
    stosd
//increment key pointer
    add rbx,4
//decrement input length
    dec r8
//decrement key length and check if either that or the input length is zero
    loopne main_loop
//restore the original values of the key pointer and length
    pop rcx
    pop rbx
//if the input length is not zero, continue encoding/decoding after re-saving the key
    jne save_key
//restore rbx from the start and return
    pop rbx
    ret

R, 42 bytes

\(x,y,b,u=utf8ToInt)intToUtf8(u(x)*u(y)^b)

Attempt This Online!

Dyalog APL, 39 chars

{d t k←⍵⋄⎕UCS⊃{⍵×(÷⍣d)⍺}/⎕UCS¨(k⍴⍨⍴t)t}

Explanation

d t k←⍵                                ⍝ Set vars for the input: decode_flag, text, key
                             (     )    
                              k⍴⍨      ⍝ Reshape (backwards) the key
                                 ⍴t        to the length of the cipher text
                        ⎕UCS¨(     )t  ⍝ Convert that and the text to codepoint
             {        }/               ⍝ Apply this function between the two sets of codepoints
                                       ⍝       this would be on two vectors,
                                       ⍝       but the internals are all pervasive
              ⍵×     ⍺                 ⍝   Multiply them
                (÷⍣d)                  ⍝   But first, find the reciprocal of ⍺ 0 or 1 times depending on the decrypt flag
           ⊃                           ⍝   Pick the first (and only) element of the boxed value given by /
       ⎕UCS                            ⍝ Convert back to characters

Example:

    {d t k←⍵⋄⎕UCS⊃{⍵×(÷⍣d)⍺}/⎕UCS¨(k⍴⍨⍴t)t}0 'Hello' 'Golf'
ᏸ⯋ⶐ⬈ỉ
    {d t k←⍵⋄⎕UCS⊃{⍵×(÷⍣d)⍺}/⎕UCS¨(k⍴⍨⍴t)t}1 'ᏸ⯋ⶐ⬈ỉ' 'Golf'
Hello

Standard Pascal, 226 bytes

function g(f,i,k:string):string;var u,t:word;y:string;begin y:='';while length(k)<length(i)do k:=k+k;for u:=1to length(i)do begin if f='d' then t:=ord(i[u])div ord(k[u])else t:=ord(i[u])*ord(k[u]);y:=y+char(t);end;exit(y)end;

f could be d or e based on action (Decode or Encode), and i is input and k is key.

Does not work if key or input is over 65 thousand characters.

Readable Version

function g(f, i, k: string): string;
var
  u: word;
  y: string;
  t: word;
begin
  y := '';
  while length(k) < length(i) do
    k := k + k;

  for u := 1 to length(i) do
  begin
    if f = 'd' then
      t := ord(i[u]) div ord(k[u])
    else
      t := ord(i[u]) * ord(k[u]);

    y := y + char(t);
  end;
  Result := y;
end;
```

Ruby, 44 bytes

This is a port of Lucenaposition's Python 3 solution; give them an upvote!

Takes two arrays of codepoints and either 1 for encode or -1 for decode. Returns an array of codepoints.

->x,y,z{x.zip(y.cycle).map{(_1*_2**z).to_i}}

Attempt This Online!

Technically I could take out the (...).to_i which would make the output (when decoding) an array of Rationals but that seemed like a bridge too far.

Ruby, 66 bytes

If we want strings for input and output it costs another 22 bytes.

->x,y,z{x.chars.zip(y.chars.cycle).map{_1.ord*_2.ord**z}.pack"U*"}

Attempt This Online!

Vyxal 3, 8 bytes

⎂O⤻M?*×O

Vyxal It Online!

⎂O⤻M?*×O­⁡​‎‎⁡⁠⁡‏⁠‎⁡⁠⁢‏‏​⁡⁠⁡‌⁢​‎‎⁡⁠⁣‏⁠‎⁡⁠⁤‏‏​⁡⁠⁡‌⁣​‎‎⁡⁠⁢⁡‏⁠‎⁡⁠⁢⁢‏‏​⁡⁠⁡‌⁤​‎‎⁡⁠⁢⁣‏‏​⁡⁠⁡‌⁢⁡​‎‎⁡⁠⁢⁤‏‏​⁡⁠⁡‌­
⎂O        # ‎⁡convert first 2 inputs to codepoints
  ⤻M      # ‎⁢mold key to the shape of the text
    ?*    # ‎⁣key^3rd input
      ×   # ‎⁤multiply the key (potentially inverted) by the text
       O  # ‎⁢⁡convert back to unicode
💎

Created with the help of Luminespire.

<script type="vyxal3">
⎂O⤻M?*×O
</script>
<script>
    args=[["Golf","Hello","1"],["Golf","ᏸ⯋ⶐ⬈ỉ","-1"]]
</script>
<script src="https://themoonisacheese.github.io/snippeterpreter/snippet.js" type="module"/>

Google Sheets, 95 bytes

Expects encode/decode in A1 (1 for encode, -1 for decode), input in A2 and key in A3

=SORT(LET(l,LEN(A2),i,SEQUENCE(l),JOIN(,CHAR(CODE(MID(A2,i,1))*CODE(MID(REPT(A3,l),i,1))^A1))))

Another 95

=SORT(JOIN(,CHAR(EXP(MMULT(LN(CODE(MID({A2,REPT(A3,LEN(A2))},SEQUENCE(LEN(A2)),1))),{1;A1})))))

05AB1E, 10 bytes

DŠ∍‚Ç`Im*ç

Inputs in the order \$input,key,flag\$, where the \$flag\$ is either 1 for encoding or -1 for decoding.
(Minor note: key should be a list of characters instead of string if it only contains digits or can be otherwise interpret as a number - otherwise ∍ will extend/shorten it to that value instead of the length.)

Try it online (encoding input) or try it online (decoding input).

Explanation:

D          # Duplicate the first (implicit) input
 Š         # Triple-swap, so the order is input, (implicit) input-key,input
  ∍        # Extend/shorten the input-key to a length equal of the regular input
   ‚       # Pair the two strings together
    Ç      # Convert both to a list of codepoint integers
     `     # Pop and push these lists to the stack again
      I    # Push the third input-integer
       m   # Take the codepoints to the power this flag
        *  # Multiply the integers at the same positions in the lists
         ç # Convert from codepoint-integers to unicode characters
           # (after which the result is output implicitly)

Python 3, 76 bytes

lambda x,y,z:''.join(chr(round(ord(a)*ord(b)**z))for a,b in zip(x,y*len(x)))

Try it online!

Uses 1 for encode and -1 for decode.

Due to floating point error, I cannot use int so I must use round.

Jelly,  12  8 bytes

The exponentiation idea was lifted from doubleunary's Google Sheets answer - I should have thought of it :(

ṁO*⁵×ỌɓO

A full program that accepts the key, the plaintext/ciphertext, and an integer, 1 = encode or -1 = decode, and prints the result to stdout.

Encode online!

Decode online!

How?

ṁO*⁵×ỌɓO - Main Link: Key; Plaintext/Ciphertext
      ɓ  - dyadic chain with swapped arguments - f(Plaintext/Ciphertext, Key):
       O -   convert {Plaintext/Ciphertext} to ordinals -> PlainOrdinals/CipherOrdinals
ṁ        - mould {Key} like {PlainOrdinals/CipherOrdinals}
 O       - convert {that} to ordinals -> MouldedKeyOrdinals
   ⁵     - program's third argument -> Encode/Decode = +/- 1
  *      - {MouldedKeyOrdinals} exponentiate {Encode/Decode}
    ×    - mulitpliy {that} by {PlainOrdinals/CipherOrdinals}
     Ọ   - convert {that} from ordinals to characters
         - implicit print

Japt, 9 bytes

Takes the string to be processed as first input, the key as the second, and 1 or -1 as the third to indicate encode or decode, respectively.

cÈ*VcY pW

Try it: Encode | Decode

cÈ*VcY pW     :Implicit input of strings U=text & V=key and integer W
c             :Map the charcodes of U by
 È            :Passing each at index Y through the following function
  *           :  Multiply by
   VcY        :    Get the charcode in V at modular index Y
       pW     :  Raised to the power of W

Charcoal, 12 bytes

⭆η℅×℅ιＸ℅§ζκθ

Try it online! Link is to verbose version of code. First input is 1 to encode, -1 to decode. Explanation:

 η              Second input
⭆               Map over characters and join
     ι          Current character
    ℅           Take the ordinal
   ×            Multiplied by
         ζ      Third input
        §       Cyclically indexed by
          κ     Current index
       ℅        Take the ordinal
      Ｘ         Raised to power
           θ    First input
  ℅             Convert to Unicode character
                Implicitly print