JavaScript (Node.js), score: 124 - 95 = 29

console['l\157g'](`Hello, Wo${"#%&'*+-/0124789:;<=>?@ABCDEFGIJKLMNOPQRTUVXYZ^_abfjkmpquvwxyz|~".length.toString(36)[1]}ld!`)

Try it online!

Zsh, score \$ 116 - 108 = 8 \$

1=`wc -m<<<'␀␁␂␃␄␅␆␇␈␎␏ABCDEFGIJKMNOPQRSTUVXYZabfghjkqsuvxyz234567890_+[];:~./>?\|"%^&*'`
print   He${(L#)@}lo, World!

Attempt This Online!

␀␁␂␃␄␅␆␇␈␎␏ should be the unprintable characters 0x00 to 0x08 and 0x0E to 0x0F, but these don't render in SE markdown.

BASIC, 129 bytes

FOR i=8-2 TO 79 STEP 3+3:PRINT MID$("9*./!H065<>e?@ABClGJKLQlUVYZ[o\]^_`,abcf  ghjksWmnpqxotuvwyrz4{|}l~#%&'d",MOD(i,73),1);:NEXT

Try it online!

Updated to a valid version thanks to Dominic van Essen.

Husk, 256 bytes, 256 characters, score=0

m←`C₁†ȯ→*2c¨abqnpsyhkzjwf
¢"H½↕↑↓↔∟¦¡¿‼…‰‡√≤≥±∂∫∞≈≠≡⌐¬÷×τ►#$%&'()+Φ-./013456789:;<=>?@ABDEFG¤IJKLMNOPQRSTUVṄXYZ[\]^_⁵ı§χituvxg¹{|}~·₀₂₃₄₅₆₇₈₉⌈⌉⌊⌋ΓΔΘΛΞΠΣ€ΨΩαβγδεζηθλμξπρςσ▲φƒψω⁰r²³⁴d⁶⁷⁸⁹£,¥o´ ▼!◄lȦḂĊḊĖḞĠḢİĿṀWȮṖṘṠṪẆẊẎŻȧḃċḋėḟġḣeȷŀṁṅṗṙṡṫẇẋẏżÄËÏÖÜŸØäëïöüÿø◊□¶«»

Try it online!

I think this is irreducible. It certainly took a bit of effort to build, but striving for a score of zero is inviting pretty close scrutiny...

Husk has 256 characters on its codepage, so this pangram needs to use all of them. Aiming for a zero score also means that (a) the 'program' code cannot easily include any of the letters of 'Hello, World!', and (b) we need to find a way to 'recycle' the letters 'l' and 'o', which each occur more-than-once in the output.
Unfortunately, Husk uses '!' as its indexing function, which rules-out the most straightforward approach to extracting elements from a long string, so we need to find a different way to do this.

Here's my approach:

line 2: the encoding string (228 characters, + 2 characters ¢")

¢"H½↕↑↓↔∟¦¡¿ ... öüÿø◊□¶«»      # A 228-character string, repeated forever.  
                                # Since one of the characters is a '\', though,
                                # this is read by Husk as a 227-character array.
                                # The characters of 'Hello, World!' are inserted
                                # at specific positions in the infinite string, 
                                # (so they occur out-of-order in the single copy).

line 1: the indexing program (26 characters)

     †ȯ   c¨abqnpsyhkzjwf       # For all the codepoints of the index string "abqnpsyhkzjwf":
        *2                      # double them
       →                        # and add 1.
  `C₁                           # Now divide the encoding string into substrings with these lengths
m←                              # and get the first letter of each.

The indexing string "abqnpsyhkzjwf" is constructed to 'hit' the letters 'o' and 'l' 2 and 3 times, respectively, by 'wrapping around' in the infinite list, so that those letters only need to occur once in the encoding string. The largest index needs to be the same as the length of the encoding string, in order to 'hit' the same target ('l') twice in a row: this is achieved with 'q' (2x codepoint 113 + 1 = 227).

JavaScript (Node.js), ^{118 117} 116 - 95 = 21

shoutout to tsh for spotting 2 issues

alert(`Hello, Wo${('\71'+";#%&*/8025:<=>[]?@ABCDEFGIJKLMNOPQRTUVXYZ|_bcfjkpmqsuvwxyz".length^-~94).toString(36)}d!`)

Try it online!

Backhand, 100 - 95 = 5

M#$%&"()*+H-./0e1234l5678l9:;<o=>?@,ABCD FGIJWKLNOoPQRSrTUVXlYZ[\c^_`b"dfgh]ikmn'pqst!uvwxryz{}a|~jE

Try it online!

Note that the version on TIO is missing a couple of instructions (notably '), so I've copied the interpreter here. This program works because Backhand executes every fifth instruction thanks to the first M (it's actually a little more complex than this), so the actual code is more like:

M    "    H    e    l    l    o    ,         W    o    r    l    c    "    ]    '    !    r    a   j 

However, since you can't remove the spaces inbetween without breaking the flow of the program, you can replace them with whatever you want. The final check is that the j jumps to the 10th character (H) to halt and output,, which is less feasible if you modify the program. It's possible you might be able to form a valid program from this mishmash, but it is unlikely, and I'm sure that there's another permutation that fixes that.

Explanation:

M                      Increase the step count from 3 to 5
 "Hello, Worlc"        Push the string to the stack
               ]       Increment c to d
                '!     Push !
                  r    Reverse the stack
                   aj  Jump back to the 10th character
  H                    Halt and output the stack

C (gcc), 111 109 bytes, score = 14

main(){printf("H%cllo, Wo\x72ld!",39+sizeof"14568:<=>?@ABCDEFGIJKLMNOPQRSTUVXYZ[]`bghjkquvwy0|~#$&'*-./^_");}

Try it online!

Edit: bugfix thanks to Noodle9

Edit 2: changed order of numbers and operators in the string to fix a problem found by G B, replaced "Hello, World!" with "Hello, Wo\x72ld!"

shorter solution based on Sheik Yerboutis Code (107 bytes, score = 12)

main(){printf("H%cllo, Wo\x72ld!","_:?=<#$&*-+^|./4138596@ABCDEFGIJKLMNOPQRSTUVXYZ`bghjkquvwysz0~e"['>']);}

Try it online!

JavaScript (ES6), score 30 124 - 95 = 29

alert(`Hell`+String.fromCharCode(057^"#$%&*-/1234689:;<=>?@ABDEFGIJKLMNOPQRTUVXYZ][_bcjkpqsuvwxyz{|}~\\".length)+', World!')

The os of fromCharCode cannot be put inside quotes. Its char code is 111, which can't be uses directly as an argument to String.fromCharCode. You can't XOR to get it because you need 64, 86, 104 or 106, which are also impossible, I think.

Python 3, 101 bytes, 101 - 95 = 6

print('o#$%&*./0W368;<=>? @ABCDEFG,IJKLMNOPoQRSTUVXY\154Z^_`abcflghjkmqsuevwyz{|}~H'[::-9]+"\x72ld!")

Try it online!

Ruby, 119 112 bytes / score 17

puts ["%x"%(("!\#&+,-01245:;<>?@ABDEFIJKLNOQRUVWXYZ`dflqnry{|}"=~/$/)^"s8j9bePwGp7SgMTCvznH".to_i(36))].pack'h*'

Try it online!

C (gcc), score 114 - 95 = 19

Saved 2 bytes and removed 3 points thanks to Sheik Yerbouti!!!

j;main(){for(;j<74;j+=6)putchar("H#$%&'e*-./0l12358l9:>?@oABCDE,FGIJK LMNOPWQRSTUoVXYZ\\r^_`bglkqsvwdxyz|~!"[j]);}

Try it online!

All characters outside of the double-quotes are needed in order for the program to compile and run properly. Taking any of the characters out from the double-quoted string will mess-up Hello, World! from being printed properly. That string is exactly Hello, World! with 5 characters in between each character.

R, 119 117 bytes, score=24 22

Thanks to Dominic van Essen for spotting a bug.

cat('Hello, Wo',intToUtf8(nchar("#$%&*-./012479:;<>?@ABDEFGIJKLMNOPQRSVXYZ[]^_`bgjkmquvwyz{|}~")+53),"\x6Cd!",sep="")

Try it online!

Uses the hex code trick to encode the last l as \x6C. This saves 2 bytes: 1 by avoiding the reuse of the l (none of the other characters are used anywhere else), and 1 by avoiding having to escape the \ (as \\) in the string.

The long string in the middle is 61 characters long. They are all necessary to get the character r (ASCII code 114), which is produced by the intToUtf8(nchar("...")+53) part.