JavaScript (Node.js), 147 bytes

f=(X,d=-8,t=[w=0,n=(x=X+0+0).length/3|0,n*2].map(b=>+(y=x.substr(b,n).slice(d).replace(/[g-z]/gi,0),w+=y[0]<1,'0x0'+y.slice(0,2))))=>w>2?f(X,d+1):t

Try it online!

Seems only one other JS answer compute and it's 200+ bytes

Fix that 0e01 is also treated as 0

Also the two branch in description is unnecessary

AWK, 237 bytes

{$0=toupper($0);for(gsub(/[^0-9A-F]/,0);(k=length)%3||!k;)$0=$0 0
for(z=k/3;l<k;l+=z)a[++x]=(z<2?0:X)substr($0,l+1,z)
for(;z>8||z>2&&a[1]~/^0/&&a[2]~/^0/&&a[3]~/^0/;z--)for(n=0;n++<3;)sub(/^./,X,a[n])
for(;m++<3;)printf substr(a[m],1,2)}

Attempt This Online!

This went through a ton of iterations to properly handle all cases. Might try a different approach another time.

{$0=toupper($0);         # uppercase input
for(gsub(/[^0-9A-F]/,0); # replace invalid chars
(k=length)%3||!k;)       # check input
$0=$0 0                  # append 0s if nec
for(z=k/3;               # grab length of 'word'
l<k;l+=z)                # iterate through input
a[++x]                   # insert in array [1,2,3]
=(z<2?0:X)               # prepend 0 if nec
substr($0,l+1,z)         # then add our word
for(;z>8||               # if word too long or
z>2&&                    # if short and 
a[1]~/^0/&&a[2]~/^0/&&a[3]~/^0/ 
                         # all words start w/ 0
;z--)                    # shrink length of words
for(n=0;n++<3;)          # for each array member
sub(/^./,X,a[n])         # remove first char
for(;m++<3;)             # for each member of array
printf substr(a[m],1,2)} # print first two chars

C (clang), 273 bytes

int p(char*c){unsigned int l=(strlen(c)+2)/3,i,j,r,g,b,v;for(j=0;j<3;j++){v=0;for(i=0;i<l;i++)v=v<<4|(*c?*c>47&&*c<58?*c++-48:*c>64&&*c<71?*c++-55:*c>96&&*c<103?*c++-87:*c++&0:0);!j?r=v:j==1?g=v:(b=v);}while(r>255||g>255||b>255){r/=16;g/=16;b/=16;}return (r<<16)|(g<<8)|b;}

Try it online!

Ungolfed

int parse(char *c) {
    unsigned int length;
    unsigned int i, j;
    unsigned int r, g, b;
    unsigned int value;

    /* Round up to multiple of 3, then divide by 3 to get the component length */
    length = (strlen(c) + 2) / 3;

    /* For each component (r,g,b) */
    for (j = 0; j < 3; j++) {
        value = 0; /* Start at 0 */

        for (i = 0; i < l; i++) {
            int char_value;

            if (*c != '\0') {
                /* Parse a hex digit */
                if (*c > '0' && *c < '9') {
                    char_value = *c - '0';
                } else if (*c > 'A' && *c <= 'F') {
                    char_value = *c - 'A' + 10;
                } else if (*c >= 'a' && *c <= 'f') {
                    char_value = *c - 'a' + 10;
                } else {
                    /* Invalid digits are 0 */
                    char_value = 0;
                }

                /* Next character */
                *c++;
            } else {
                /* At the end of the string. Pretend string is right padded with '0' */
                char_value = 0;
            }

            /* Now append. If there are more than 8 digits, the leftmost ones will be lost to overflow */
            value = value << 4 | char_value;
        }

        /* Set the correct component value */
        if (j == 0) {
            r = value;
        } else if (j == 1) {
            g = value;
        } else {
            b = value;
        }
    }

    /* Remove the last few digits until all are within the range 0-255 */
    while (r > 255 || g > 255 || b > 255) {
        r /= 16;
        g /= 16;
        b /= 16;
    }

    /* Return the colour as an integer in the form 0x00RRGGBB */
    return (r << 16) | (g << 8) | b;
}

Explanation

This algorithm is different from the one given, so why does it work? Well when I was reading the rant I noticed the part where if the component length was greater than 8 digits, it would remove the leftmost digits until there were 8. The number 8 stuck out to me as 8 hex digits fit neatly inside a 32-bit integer. And the whole removing leading '0' thing also fits with the whole integer idea, as you can't tell a leading '0' from no leading '0' anyway in an integer. Also the steps for lengths <= 3 seemed to be superfluous as they could be explained using just the general case.

Going through each of the steps given:

1. "Replace chars not matching [0-9a-fA-F] with '0'". This is handled by invalid hex digits being treated as "0". Essentially, just as 'a' and 'A' both equal 10, '0' and 'G' both equal 0.

2. "Append "0" until the length is divisible by 3". This is handled in two ways. First the length of the string is rounded up to a multiple of 3 (using (length + 2) / 3 * 3), then if the end of the string is encountered, treat it as if it were 0 without incrementing the pointer (Here be UB).

3. "Split into 3 parts of equal size". This is simply dividing the rounded up length by 3 to get the component length (which simplifies into (length + 2) / 3 since the rounded up length is not used anywhere else) and counting this many characters for each component.

4. "Remove chars from the front of each part until they are 8 chars each". This is handled with integer overflow; the leading digits get shifted out once 8 characters are parsed.

5. "Remove chars from the front of each part until one of them starts with something that isn't "0" or until all parts are empty". This is done for free with integers.

6. "Look at the length of the parts: Are shorter than 2? If so, prepend each part with a "0" until each part is 2 chars long. If not, remove characters from the back of each part until they are 2 chars each". The prepending part is also done for free. The while loop handles the case where the components are longer than 2 characters.

As for the case with 3 or fewer characters: those steps actually turn out to line up with the steps for 4 or more anyway:

1. "Replace chars not matching [0-9a-fA-F] with "0"". Literally identical.

2. "Append "0" until the length is 3 chars". For lengths 1-3, this is equivalent to step 2 of the general case. For length of 0 the general step would result in "" rather than "000" at this stage.

3. "Split into 3 parts, 1 char each". This is also equivalent to step 3 of the general case, although "" would be split into the parts "", "", "", whereas "000" would become "0", "0", "0".

4. "Prepend each part with "0"". This is essentially equivalent to step 6 of the general case. As the maximum length of a component is 1, step 4 of the general case does not apply. Step 5 would turn "0", "0", "0" into "", "", "" making the two equivalent again. Then, since they are shorter than 2, they would be prepended with a 0. "", "", "" would be prepended with 2 zeros giving us "00", "00", "00", which is what it should be.

Notes

• This could be how it was implemented in IE. The implementation seems sensible for a hex code parser. That it accepts "long" colours could be future proofing for if 36-bit or 48-bit colour becomes popular (In that case I think the fact that #ABC becomes #0A0B0C rather than something like #AABBCC may have been a bug). That it "pads" may have been some sort of fix. That it accepts invalid characters may just be the nature of their hex parsing implementation.

• For some reason (TIO) clang does not like it when I do !j?r=v:j==1?g=v:b=v, forcing me to add parentheses to make !j?r=v:j==1?g=v:(b=v). In contrast MSVC didn't care. I am not digging through the spec to find out who is wrong, so I took the 2 byte penalty instead.

Japt v2.0a0, 52 bytes

u r/[^1-9A-F]/S
Êc3 ª3
úSV òVz3)®t8n)x2Ãù ®¯2 ù2 rS0

Try it

Outputs as an array of hex values.

Explanation:

u r/[^1-9A-F]/S 
u               # Convert input to uppercase
  r/[^1-9A-F]/S # Replace everything other than 1-9 and A-F with spaces
                # Store as U

Êc3 ª3 
Ê      # The length of U
 c3    # Rounded up to the nearest multiple of 3
    ª3 # If the result is 0, use 3 instead
       # Store as V

úSV òVz3)®t8n)x2Ãù ®¯2 ù2 rS0
úSV                           # Right-pad U with spaces until the length is V
    ò   )                     # Split into substrings with length:
     Vz3                      #  V divided by 3
         ®      Ã             # For each substring:
          t8n)                #  Get the last 8 characters
              x2              #  Trim any leading spaces
                 ù            # Left-pad each substring to the same length
                   ®          # For each padded substring:
                    ¯2        #  Get the first 2 characters
                       ù2     #  Left-pad with spaces to length 2
                          rS0 #  Replace all spaces with "0"

Go, 359 bytes

import."regexp"
func f(s string)(k string){s=MustCompile(`[^0-9a-fA-F]`).ReplaceAllString(s,"0")
for;len(s)%3>0;s+="0"{}
L:=len(s)/3
S:=[]string{s[:L],s[L:2*L],s[2*L:]}
if len(s)<4{return "0"+S[0]+"0"+S[1]+"0"+S[2]}
for i:=range S{a:=S[i]
for;len(a)>8;a=a[1:]{}
for;a!=""&&a[0]!='0';a=a[1:]{}
for;len(a)<2;a="0"+a{}
for;len(a)>2;a=a[:len(a)-1]{}
k+=a}
return}

Attempt This Online!

Python 3, 339 337 bytes

def f(s):
 s=re.sub('[g-zG-Z ]','0',s)
 if len(s)<4:s=s.ljust(3,'0');return['0'+h for h in s]
 while len(s)%3:s+='0'
 L=len(s);l=[s[i:i+L//3]for i in range(0,L,L//3)];l=[j[len(j)-8:]for j in l]
 while l[0]and all(k[0]=='0'for k in l):l=[m[1:]for m in l]
 if len(l[0])<2:return[n.rjust(2,'0')for n in l]
 return[o[:2]for o in l]
import re

Try it online!

Commented

def f(s):                      # Create a function, f, which takes in a string, s
 s=re.sub('[g-zG-Z ]','0',s)   # Apply the regex so all non-hex characters are replaced with '0'
 if len(s)<4:                  # If the length is less than 4:
  s=s.ljust(3,'0')             #  Append '0' until the length is 3
  return['0'+h for h in s]     #  Return the elements, prepended by a '0'
 while len(s)%3:               # Until the length is divisible by 3:
  s+='0'                       #  Append '0'
 L=len(s)                      # Assign L to the length of s
 l=[s[i:i+L//3]for i in        # Split s into
    range(0,L,L//3)]           # three equal chunks
 l=[j[len(j)-8:]for j in l]    # Remove characters from the front if the length is more than 8
 while l[0]and all(            # While the strings are not empty
       k[0]=='0'for k in l):   # And all the strings start with '0':
  l=[m[1:]for m in l]          #  Remove the first character from each string
 if len(l[0])<2:               # If the length of the first string is less than 2:
  return[n.rjust(2,'0')        # Return each string,
         for n in l]           # with a '0' prepended until the length is 2
 return[o[:2]for o in l]       # Return the first two characters of each string
import re                      # Import the re module for regex handling

C#, 235 229 227 bytes

-6 bytes thanks to Kevin Cruijssen

s=>{int l;for(s=Regex.Replace(s,"[^0-9a-fA-F]|^$","0");(l=s.Length)%3>0;s+="0");for(s=l<4?"0"+s[0]+0+s[1]+0+s[2]:s;(l=s.Length/3)>8|s[0]+s[l]+s[2*l]<145&l>2;s=string.Concat(s.Where((_,i)=>i%l>0)));return s.Where((_,i)=>i%l<2);}

Try it online!

Ungolfed:

input => {
    //the three while loops are golfed into for loops in the submission, with the last two then being combined into one

    // replace non hex chars with 0
    input = Regex.Replace(input, "[^0-9a-fA-F]|^$", "0");

    // pad input to next multiple of 3
    while (input.Length % 3 > 0) {
        input += "0"; 
    }
    // add 0 at the start of each section if the input is now 3 chars long
    // the final two zeroes here get coerced to strings
    if (input.Length == 3) input = "0" + input[0] + 0 + input[1] + 0 + input[2];

    // used to keep track of the current section length (changes as 0s are removed)
    int partLength = input.Length / 3;

    // if part length is >8 continuously remove the first char from each part (notice the first char of each part is always 0 mod partLength)
    while (partLength > 8) {
        // because LINQ, the .Where has to be cast to something to actually evaluate within a while, string.Concat is the shortest way (including other changes to have other variables be the correct type)
        input = string.Concat(input.Where((_, i) => i % partLength > 0));
    }

    //if every part starts with 0, remove the first char from each
    //this is golfed into a sum using char codes in the submission ((int)'0' = 48)
    while (input[0] == '0' && input[partLength] == '0' && input[2 * partLength] == '0' && partLength > 2) {
        input = string.Concat(input.Where((_, i) => i % partLength > 0));
    }
    //return the first two chars from each part
    return input.Where((_, i) => i % partLength < 2);
}

Vyxal, 29 bytes

S⇩3ẇ3↲∑k6Ǐ~vḟİ∑3/⟑8NȯHH2∆Z2Ẏ₴

Try it Online!

S⇩                            # Stringify ("" -> 0 edgecase) and lowercase
  3ẇ                          # Cut into chunks of length 3
     3↲                       # Pad each to length 3 with spaces (will be zeroed)
      ∑                       # Concatenate
       k6                     # Lowercase hex
          Ǐ                   # Append the leading zero
          ~vḟ                 # Without popping, for each char in input, find it in the hex
             İ                # Index into the hex + 0 - ones not found, -1s, get indexed into the end
              ∑               # Concat into a string
               3/             # Divide into 3 parts
                 ⟑            # Over each...
                  8Nȯ         # Get last 8 chars
                     HH       # Convert to/from hex to remove leading zeroes
                       2∆Z    # zfill each to length 2
                          2Ẏ  # Get the first two chars of each
                            ₴ # Print 

JavaScript + HTML, 50 bytes

document.body.setAttribute('bgcolor','#'+prompt())

Works in any browser, as the legacy behavior was kept with bgcolor (even though it won't work with CSS). Displays as graphical output.

<script>document.body.setAttribute("bgcolor",'#'+prompt());</script>

If graphical output is not allowed, then:

JavaScript + HTML, 97 bytes

c=document.body;c.setAttribute('bgcolor','#'+prompt());alert(getComputedStyle(c).backgroundColor)

<script>c=document.body;c.setAttribute('bgcolor','#'+prompt());alert(getComputedStyle(c).backgroundColor)</script>

Uses the computed styles generated. Will output like rgb(X, Y, Z).

To output a hex code, this would be 156 bytes:

c=document.body;c.setAttribute('bgcolor','#'+prompt());alert(getComputedStyle(c).backgroundColor.match(/\d+/g).map(x=>(x|256).toString(16).slice(1)).join``)

<script>c=document.body;c.setAttribute('bgcolor','#'+prompt());alert(getComputedStyle(c).backgroundColor.match(/\d+/g).map(x=>(x|256).toString(16).slice(1)).join``)</script>

JavaScript, 238 234 bytes

Way longer than the others, but actually competing!

c=>(M=Math,s='slice',t=M.ceil(c.length/3),c=c.padEnd(M.max(t*3,3)).replace(/[g-zG-Z ]/g,'0'),c=[c[s](0,t),c[s](t,t*2),c[s](t*2)]).map(p=>p[s](i=M.min(...c.map(p=>(r=p.search(/[^0].{0,7}$/),r<0?3e333:r))),i+2).padStart(2,'0')).join('')

Less golfed:

c => (
  M = Math, s = 'slice', // these are shorter
  t = M.ceil(c.length / 3), // prepare length for the parts
  c = c
    .padEnd(M.max(t*3, 3)) // pad with spaces until length divisible by 3, with a minimum of 3
    .replace(/[g-zG-Z ]/g, '0'), // replace all non-hex characters with '0'
  c = [c[s](0, t), c[s](t, t*2), c[s](t*2)] // split into 3 equal parts
).map(p => (
  i = M.min(...c.map(p=>(r=p.search(/[^0].{0,7}$/),r<0?3e333:r))), // search for a character that's not '0' in the last 8 characters, returning its index, or infinity when negative (no match found), and take the lowest of the parts
  p[s](i, i+2).padStart(2, '0') // slice max two characters from the lowest non-'0', and pad to two characters
)).join('')

Vyxal, 51 50 47 44 bytes

⇧ƛk^$c∧;ṅ3/\0ÞḞṅ3/8Nvȯ∩\03*:£øl:L2∵Ẏ¥p¥p2Nȯ∩

Try it Online!

A huge mess, but at least I outgolfed lyxal.

Rust, 372 367 340 bytes

|mut s:Vec<u8>|{for c in &mut s{if!c.is_ascii_hexdigit(){*c=48}}s.extend(b"000");let d:Vec<_>=s.chunks((s.len()-1).max(3)/3).take(3).map(|c|&c[c.len().saturating_sub(8)..]).collect();d.iter().flat_map(|c|match&c[d.iter().map(|c|c.iter().take_while(|c|**c==48).count()).min().unwrap()..]{[]=>*b"00",&[c]=>[48,c],&[x,y,..]=>[x,y]}).collect()}

Ungolfed:

fn f(mut s: Vec<u8>) -> Vec<u8> {
    for c in &mut s {
        if !c.is_ascii_hexdigit() {
            *c = b'0'
        }
    }
    s.extend(b"000");

    let chunks: Vec<_> = s
        .chunks((s.len() - 1).max(3) / 3)
        .take(3)
        .map(|c| &c[c.len().saturating_sub(8)..])
        .collect();

    let to_remove = chunks
        .iter()
        .map(|c| c.iter().take_while(|c| **c == b'0').count())
        .min()
        .unwrap();

    chunks
        .iter()
        .flat_map(|c| match &c[to_remove..] {
            [] => *b"00",
            &[c] => [b'0', c],
            &[c1, c2, ..] => [c1, c2],
        })
        .collect()
}

Playground

The chunking is a little tricky, but otherwise a relatively boring solution.

JavaScript (Browser), 138 bytes, non-competitive

c=>(document.body.innerHTML=`<font id=g color=${c}>`,getComputedStyle(g)).color.match(/\d+/g).map(n=>(n|256).toString(16).slice(1)).join``

The same color parsing algorithm also works on browsers other than IE (I had tested on my Firefox 103 and it works). But the code will output #ff0000 for input red so marked non-competitive.

f=

c=>(document.body.innerHTML=`<font id=g color=${c}>`,getComputedStyle(g)).color.match(/\d+/g).map(n=>(n|256).toString(16).slice(1)).join``

console.log(f('red                             '));
console.log(f('rules                           '));
console.log(f('1234567890ABCDE1234567890ABCDE  '));
console.log(f('rADioACtivE                     '));
console.log(f('FLUFF                           '));
console.log(f('F                               '));
console.log(f('                                '));
console.log(f('zqbttv                          '));
console.log(f('6db6ec49efd278cd0bc92d1e5e072d68'));
console.log(f('102300450067                    '));

Save ~30 bytes by Kaiido

Jelly,  44  39 bytes

-5 thanks to UnrelatedString (clever use of ;Ṫ for the empty string special case).

ØhḊiⱮŒla;Ṫ$œs3z0ZŻ€ṫ€-7ZḊẸ€Ḣ¬Ɗ¡ƬḊƇṪḣ2ZF

A full program that accepts a string and prints the resulting colour code (keeping character casing from the input*).

Try it online! Or see the test-suite.

How?

ØhḊiⱮŒla;Ṫ$œs3z0ZŻ€ṫ€-7ZḊẸ€Ḣ¬Ɗ¡ƬḊƇṪḣ2ZF - ...f(X)
Øh                                     - hex characters = "0123456789abcdef"
  Ḋ                                    - dequeue -> "123456789abcdef"
     Œl                                - lower-case X
    Ɱ                                  - map with:
   i                                   -   first 1-indexed index or 0 if not found
          $                            - last two links as a monad - f(X):
         Ṫ                             -   tail (yields zero when X is empty)
        ;                              -   X (without its tail) concatenated with that (its tail or a zero)
       a                               - logical AND the hex-char-indicator list with X or [0] (vectorises)
           œs3                         - split into three equal chunks
                 Ż€                    - prefix each with a zero
                   ṫ€-7                - tail each from index -7 -> last (up to) eight values
                       Z               - transpose
                              Ƭ        - collect up while distinct, applying:
                             ¡         -   repeat...
                            Ɗ          -   ...number of times: last three links as a monad:
                        Ẹ€             -     any? for each
                          Ḣ            -     head
                           ¬           -     logical NOT
                       Ḋ               -   ...action: dequeue
                                Ƈ      - keep those which are truthy under:
                               Ḋ       -   dequeue
                                 Ṫ     - tail
                                  ḣ2   - head to index two
                                    Z  - transpose
                                     F - flatten
                                       - implicit, smashing print

05AB1E, 48 47 bytes

lA6.$S0:₄¦«3ô¨S3ä8δ.£'e¶:øJ0Û¶'e:2£₄¦D‚ì2.£€Sø˜

Output as a lowercase list of characters without #.

Try it online or verify all test cases.

Explanation:

Step 1: Convert the input to lowercase, and replace all invalid letters with a 0:

l              # Lowercase the (implicit) input-string
 A             # Push the lowercase alphabet
  6.$          # Remove its first 6 characters: "ghijklmnopqrstuvwxyz"
     S         # Convert it to a list of characters
      0:       # And replace all those characters with a "0" in the uppercase input

Try just step 1 online.

Step 2: Pad with trailing 0s, and split it into three equal-sized parts (with each part as a list of characters):

₄              # Push 1000
 ¦             # Remove it's first character: "000"
  «            # Append it
   3ô          # Split it into parts of size 3
     ¨         # Remove the last (potentially shorter) part
      S        # Convert it to a flattened list of characters
       3ä      # Split it into 3 equal-sized parts

Try just the first two steps online.

Step 3: Only keep the last 8 characters of each inner list:

 δ             # Map over each inner list
8 .£           # Only keep (up to) the last 8 characters

Try just the first three steps online.

Step 4: Remove leading 0s from each list, until a column doesn't start with a 0 anymore:

'e¶:          '# Replace all "e" with a newline
               # (workaround, because "0e0" would be interpret as 0 in 05AB1E)
    ø          # Zip/transpose; swapping rows/columns
     J         # Join each inner list together
      0Û       # Trim all leading 0s
        ¶'e:  '# Undo the workaround, by replacing all newlines back to "e"s

Try just the first four steps online.

Step 5: Only keep the first two characters of each row, potentially prepending the result with 0s, and output the final result:

2£             # Only keep the first two triplets of the list
  ₄¦           # Push "000" again
    D‚         # Pair it with itself: ["000","000"]
      ì        # Prepend this in front of the list
       2.£     # Now only keep the last two triplets
          €S   # Convert each inner string back to a list of characters
            ø  # Zip/transpose back; swapping rows/columns
             ˜ # Flatten the pair of triplets of characters
               # (after which the sixtet of characters is output implicitly as result)

Retina 0.8.2, 117 bytes

T`op`a-fA-FRd
$
00
^((.)*?)((?<_-2>.)*)((?<-_>.)*)0?0?$
00$1¶00$3¶00$4¶
.+(.{8})
$1
+`^0(..+¶)0(.+¶)0
$1$2
(..).*¶
$1

Try it online! Link includes test cases. Explanation:

T`op`a-fA-FRd

Replace non-hex digits with zeros.

$
00

Append two zeros in case the length is not a multiple of 3.

^((.)*?)((?<_-2>.)*)((?<-_>.)*)0?0?$
00$1¶00$3¶00$4¶

Divide the string into three equal parts, discarding the trailing zeros just added if necessary. Prefix 00 to each part in case it is short.

.+(.{8})
$1

Trim each part to the last 8 digits.

+`^0(..+¶)0(.+¶)0
$1$2

Trim leading zeros while all parts have one, but don't trim to fewer than two digits.

(..).*¶
$1

Keep only the first two digits of each part and join everything together.

Vyxal, 67 64 bytes

k6:⇧∪ṅ\^pøB?0øṙ₅:ǒ3εǒ+↲›3/?L4<[2↳›|8Nvȯ{:vh0J≈|ƛh0=ßḢ}ƛ₃[0p|2Ẏ}ṅ

Try it Online!

A big mess of unicode and lesser-used overloads of things

Explained

There are 10 parts here:

Generating the hex-digit regex: k6:⇧∪ṅ\^pøB
Replacing non-hex with 0: ?0øṙ
Padding to nearest multiple of 3: ₅:ǒ3εǒ+↲›
Splitting into 3 equal parts: 3/
The length 4 conditional branch: ?L4<[
  Making each bit 2 digits with 0s if needed: 2↳›
|
  Getting the last 8 characters: 8Nvȯ
  Removing 0s until something doesn't start with a 0: {:vh0J≈|ƛh0=ßḢ}
  Either prepending a 0 or getting the last 2 chars: ƛ₃[0p|2Ẏ}
Outputting the result: ṅ

Generating the hex-digit regex

k6:⇧∪ṅ\^pøB
k6           # The string "0123456789abcdef"
  :⇧         # Uppercased ("0123456789ABCDEF")
    ∪ṅ       # Unioned with the lowercase string ("0123456789abcdefABCDEF")
       \^p   # With a caret prepended ("^0123456789abcdefABCDEF")
          øB # Surrounded in "[]" ("[^0123456789abcdefABCDEF]")

This regex ([^0123456789abcdefABCDEF], called re from here) will be used in the next step to determine which characters to replace with 0s.

Replacing non-hex with 0

?0øṙ # Before this part, the stack is [re]
?    # Push the input : [re, input]
 0   # Push a 0 to the stack : [re, input, 0]
  øṙ # Replace matches of re in input with 0

This completes the step of replacing invalid characters with 0s, which is common to strings less than 4 chars and strings with at least 4 chars. This string will be called 0-str from here.

Padding to nearest multiple of 3

₅:ǒ3εǒ+↲› # Before this part, the stack is [0-str]
₅:        # Push two copies of the length of 0-str without popping it
  ǒ3ε     # Modulo the first copy by 3 and take the absolute difference of that and 3 to get how far away the length is from the next multiple of 3. Note that this may result in the result being 3 because a string of length that is divisible by 3 will return 3, as 3 - (length % 3 => 0) = 3
     ǒ    # Modulo 3 again to make a 3 a 0
      +↲› # Add that to the remaining copy of the length to get how many characters should be in the padded string, left pad 0-str with spaces to that many characters and replace all spaces with 0s.

The result of this step is common to both string length possibilities and will be referred to as pad-0-str.

Splitting into 3 equal parts

3/ # Before this part, the stack is [pad-0-str]
3/ # Divide the string into 3 equal parts

Yes, there really is a built-in for this. The result will be referred to as 3-part-pad.

The length 4 conditional branch

?L4<[...|...} # Before this part, the stack is [3-pad-str]
?L            # Is the length of the input
  4<          # Less than 4?
    [...      # If so, move onto the steps for a string with less than 4 characters
        |...} # Otherwise, move onto the steps for a string with at least 4 characters. A `}` is used instead of a `]` because it saves a byte over `;]` later on.

The stack hasn't changed after this step.

Less than 4 - Making each bit 2 digits with 0s if needed

2↳› # Before this part, the stack is [3-pad-str]
2↳  # Right pad each part of 3-pad-str to be two characters, using spaces. This is needed because components of 3-pad-str will be empty if the input is the empty string. In the case of the empty string, this returns "  ". Otherwise, it returns " " + char
  › # Replace the spaces with 0s. This turns "  " into "00" and everything else into "0" + char.

The result of this will be called res.

At least 4 - Getting the last 8 characters

8Nvȯ # Before this part, the stack is [3-pad-str]
8N   # Push -8 to the stack : [3-pad-str, -8]
  vȯ # Push [part[-8:] for part in 3-pad-str]

The result of this will be referred to as last-8

At least 4 - Removing 0s until something doesn't start with a 0

{:vh0J≈|ƛh0=ßḢ}   # Before this part, the stack is [last-8]
{                  # While ...
 :vh               # the first character of each part of last-8 (leaving last-8 on the stack)...
    0J≈            # are all equal to 0:
       |ƛh0=       # For each item, check if the first character is 0
            ßḢ}    # And if so, remove the head of that part. The `}` here closes both the map lambda and the while loop.

This is probably the messiest section. The result of this will be referred to as second-last.

At least 4 - Either prepending a 0 or getting the last 2 chars

ƛ₃[0p|2Ẏ} # Before this part, the stack is [second-last]
ƛ         # For each item in second-last, which has leading 0s removed:
 ₃[       # If the length is 1:
   0p     #   Prepend a 0
     |2Ẏ} # Else: take the last two characters.
          # The `}` here closes both this inner if-statement, the map lambda, and the outer if-statement

The result of this part will be referred to as res. This is analogous to the step taken in the less than 4 branch.

Outputting the result

ṅ # Join res on empty string - concatenate into a single string.

I could have used the s flag here, but at this point, we're already play the long golf game, so I thought I might as well go flagless.