Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 19 min read

The Trust Store, Not the Server, Picks the Last Link

When you ask "is this certificate valid?", you tend to look for the answer inside the certificate. The padlock in the browser reinforces that: a thing is either valid or it isn't, and the bytes decide. They don't. The t

When you ask "is this certificate valid?", you tend to look for the answer inside the
certificate. The padlock in the browser reinforces that: a thing is either valid or it
isn't, and the bytes decide.

They don't. The thesis of this piece: validity is not a property of a certificate. It
is a result that the certificate and the verifier's own list produce together.
The same
three certificates, on the same machine, against two different root lists, give two
different chains β€” one three links long, the other four. And one of the certificates the
server sends goes completely unused in the first case while holding the whole chain up in
the second.

I have opened the root store on this blog before. The piece where
counting my Mac's keychain gave 158 certificates, 157 anchors and 155 names
was an inventory of the drawer: what is in it, and how much of it is actually trusted.
The question here is a different one. Given a chain arriving over the wire, which anchor
does verification land on, who makes that call, and which command shows it to you.

Every number below comes from my own machine (MacBook Pro M4 Pro, macOS 26 / Darwin
25.6.0, arm64) and from my own blog's live certificate chain, on 17 November 2026. There
are three OpenSSL flavours here: the system LibreSSL 3.3.6, plus OpenSSL 3.6.4 and
OpenSSL 4.1.0 from Homebrew. Every chain listing comes from the 4.1.0 binary.

What is actually on the wire

Start with what the server really sends. Pulling the raw chain takes one command:

echo | openssl s_client -connect mustafaerbay.com.tr:443 \
  -servername mustafaerbay.com.tr -showcerts 2>/dev/null > chain_raw.txt
grep -c 'BEGIN CERTIFICATE' chain_raw.txt
# 3

Three certificates. Split into cert0.pem, cert1.pem and cert2.pem, opened one by
one, the table looks like this:

# Subject Issuer Validity DER size
0 CN=mustafaerbay.com.tr GTS WE1 10 Nov 2026 – 8 Feb 2027 926 bytes
1 GTS WE1 GTS Root R4 13 Dec 2023 – 20 Feb 2029 675 bytes
2 GTS Root R4 GlobalSign Root CA 15 Nov 2023 – 28 Jan 2028 894 bytes

The third row looks odd at first. You expect the last certificate in a chain to be a
self-signed root; here the subject is GTS Root R4 but the issuer is GlobalSign Root CA.
So this is not a root certificate. It is a root signed by another root β€” a
cross-signature. Cloudflare sits in front of my blog and obtains the certificate, so the
choice of this trio isn't mine, but living with its consequences is.

The three add up to 2,495 bytes. The cross-signed root is 894 of them, which is 35.8% of
the chain that goes out on every TLS handshake. Keep that number in mind; we come back to
it near the end.

When you hand the verifier only the leaf

Let's make the most common mistake first and verify the leaf on its own:

openssl verify cert0.pem
# CN=mustafaerbay.com.tr
# error 20 at 0 depth lookup: unable to get local issuer certificate
# error cert0.pem: verification failed

The browser opens the same site without complaint, curl returns 200, and yet
openssl verify says it cannot find a local issuer. There is no contradiction here: the
browser takes the chain from the wire, while openssl verify works with whatever you
hand it. Intermediates have their own flag:

cat cert1.pem cert2.pem > sent_chain.pem
openssl verify -untrusted sent_chain.pem cert0.pem
# cert0.pem: OK

The distinction here is easy to miss, and it is the lock the whole article turns on. The
OpenSSL documentation says of -untrusted that every certificate in the file is treated
as untrusted and "may be used to construct a certificate chain".
So -untrusted does not confer trust, it paves a road. Trust comes from the root list
and nowhere else.

Don't assume where that root list comes from, either. The same page states it plainly:
"OpenSSL does not provide a default set of trust anchors." Whoever builds the binary bakes
in a path, openssl version -d prints it, and the three binaries on my machine report
three different OPENSSLDIR values β€” though two of them are symlinks into the same
Homebrew package, so there are only two effective files. On top of that, SSL_CERT_FILE and SSL_CERT_DIR
override it at runtime β€” in the
documentation's words, they "specify the default directory or file containing CA
certificates".

Where the chain ends today

Seeing OK is not enough; you want to see which path produced it. That flag is
-show_chain:

openssl verify -show_chain -untrusted sent_chain.pem cert0.pem
# cert0.pem: OK
# Chain:
# depth=0: CN=mustafaerbay.com.tr (untrusted)
# depth=1: C=US, O=Google Trust Services, CN=WE1 (untrusted)
# depth=2: C=US, O=Google Trust Services LLC, CN=GTS Root R4

Three links. The first two carry an untrusted marker because they came from the file I
supplied. The third carries none: it came from the root list.

Note that the server also sent a GTS Root R4. Verification did not use it. The reason is
in the documentation: the lookup "first searches for issuer certificates in the trust
store", and once an issuer is found there, the list of untrusted certificates "will not
be consulted anymore". Since OpenSSL 1.1.0 this behaviour can't be turned off;
-trusted_first is always enabled. In other words the third certificate the server sends
is dead weight on my machine today. The proof is simple β€” leave it out, and nothing
changes:

openssl verify -show_chain -untrusted cert1.pem cert0.pem
# cert0.pem: OK   (same depth=0..2, still ending at GTS Root R4)

I removed one line from the store and the path grew a link

Now for the real experiment. Homebrew's root list holds 183 certificates. I took a copy
with only the self-signed GTS Root R4 stripped out, leaving 182. A short filter that walks
the bundle and reads each subject is enough:

python3 - <<'EOF'
import re, subprocess
SRC = '/opt/homebrew/etc/ca-certificates/cert.pem'
raw = open(SRC, encoding='utf-8', errors='replace').read()
keep, drop = [], []
for c in re.findall(r'-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----', raw, re.S):
    subj = subprocess.run(['openssl','x509','-noout','-subject'],
                          input=c, capture_output=True, text=True).stdout
    (drop if 'GTS Root R4' in subj else keep).append(c)
open('store_no_r4.pem','w').write('\n'.join(keep) + '\n')
open('r4_brew.pem','w').write(drop[0] + '\n')      # single-root file
print('dropped', len(drop), 'kept', len(keep))
EOF
# dropped 1 kept 182

Running the same filter over /etc/ssl/cert.pem with GTS Root R4 produces
r4_apple.pem, and with GlobalSign Root CA it produces only_globalsign.pem. Then the
same verification, against the 182-certificate list:

openssl verify -show_chain -CAfile store_no_r4.pem -untrusted sent_chain.pem cert0.pem
# cert0.pem: OK
# Chain:
# depth=0: CN=mustafaerbay.com.tr (untrusted)
# depth=1: C=US, O=Google Trust Services, CN=WE1 (untrusted)
# depth=2: C=US, O=Google Trust Services LLC, CN=GTS Root R4 (untrusted)
# depth=3: C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA

Same leaf, same handshake, same three certificates β€” and a completely different chain.
The path is four links now, R4 carries the untrusted marker (this time it came from the
wire, not the store), and the anchor is GlobalSign Root CA. That 894-byte certificate I
called dead weight a minute ago is, against this list, the only thing holding the chain
together.

That it is load-bearing is measurable too: ask the same list without the cross-signature.

openssl verify -show_chain -CAfile store_no_r4.pem -untrusted cert1.pem cert0.pem
# C=US, O=Google Trust Services, CN=WE1
# error 20 at 1 depth lookup: unable to get local issuer certificate
# error cert0.pem: verification failed

In short: in a list that knows R4, the third certificate is redundant; in a
list that doesn't, it is mandatory. The server knows neither, and cannot β€” it can't see
the client's list. So it sends something that covers both.

Diagram

The standard says exactly this. Per section 6 of RFC 5280,
"valid paths begin with certificates issued by a trust anchor",
and the same document adds that there is no requirement for every path to be validated
with the same anchor: different paths may use different anchors. There is no such thing as
"the correct chain"; each verifier builds its own.

Same key, three different certificates

Here my own measurement walked into a trap, and leaving out where it walked in would make
this article dishonest.

To compare the root lists on my machine, I took the SHA-256 fingerprint of every
certificate and intersected the sets. The result said GTS Root R4 is absent from
Apple's /etc/ssl/cert.pem. Yet one step earlier a line reading R4 had shown up inside
that very file. Both cannot be true.

What is true: the file does hold a GTS Root R4, but a different certificate from the R4
in my Homebrew list. Put the three R4s side by side:

From Issuer Serial Signature algorithm DER SHA-256 (first 8)
Apple /etc/ssl/cert.pem itself 6E47A9C8… ecdsa-with-SHA384 526 bytes 71:CC:A5:39
Homebrew / Node / Python itself 0203E5C0… ecdsa-with-SHA384 525 bytes 34:9D:FA:40
Sent by my server GlobalSign Root CA 7FE530BF… sha256WithRSA 894 bytes 76:B2:7B:80

Three different certificates, three different fingerprints. But the public key is the same
in all three. The command that shows it digests the key's DER encoding, not the
certificate:

openssl x509 -in r4_apple.pem -noout -pubkey \
  | openssl pkey -pubin -outform DER | shasum -a 256
# 9847e5653e5e9e84...   (identical for all three)

The genuinely surprising part is the first two rows. Both are self-signed, both are valid
from 22 June 2016 to 22 June 2036, both are signed with the same algorithm and carry the
same key. Seeing one in one list and the other in another does not mean a root is missing.

They are not identical twins either, and the difference is not cosmetic. Apple's copy
carries a keyUsage of Certificate Sign, CRL Sign; Homebrew's carries one bit more:

openssl x509 -in r4_apple.pem -noout -ext keyUsage
#     Certificate Sign, CRL Sign
openssl x509 -in r4_brew.pem  -noout -ext keyUsage
#     Digital Signature, Certificate Sign, CRL Sign

So the field that states what the key may be used for differs between the two copies. The
size gap isn't one byte either: the serial is 16 bytes against 13, the signature block 103
against 105, the extension one byte longer β€” 526 and 525 in total.

The anchor is a key, not a certificate

RFC 5280 spells out why that trap is a trap β€” and this blog did not meet the concept
today: the 158-roots piece already noted that an anchor is represented by a key and is an
input to path validation rather than a link in the chain. There it was a definition; here
it has a measured price. Trust anchor information, by the document's
own definition, has four parts: the trusted issuer name, the trusted public key algorithm,
the trusted public key, and optionally the key parameters. A certificate is not on that
list. The standard's phrasing is that the anchor information may be supplied to the path
processing procedure "in the form of a self-signed certificate" β€” may be, which makes
the certificate a transport format rather than the anchor itself.

The practical consequence: comparing two root lists by fingerprint gives the wrong answer.
Here are the same two files on the same machine, compared two ways:

Comparison Shared by fingerprint Shared by public key
Apple (128) ∩ Homebrew (183) 91 96
Apple (128) ∩ Node 22 bundled (150) 98 104
Homebrew ∩ Node 22 bundled 132 132

The gap of five between Apple and Homebrew comes from five anchors that share a key
without sharing a single certificate: GTS Root R1, R2, R3, R4 and GlobalSign ECC Root
CA - R4. An audit that looks at fingerprints reports those five as "missing roots" and
raises five false alarms. The arithmetic closes, too: 91 shared fingerprints + 5 = 96
shared keys.

There is a sixth case, slightly more tangled. The Spanish Firmaprofesional root's key sits
in both lists and they do share a certificate for it β€” but Homebrew carries that key twice,
in two different certificates. That is why the list shows 183 certificates and 182 distinct
keys. It also explains the gap of six in the table's second row: compared against Node,
Apple's Firmaprofesional variant overlaps with neither of the other two, so six anchors,
not five, slip past a fingerprint scan.

You can see this with everyday curl rather than a lab tool. The --cacert flag
narrows the trust list down to a single file, and each of the three files below holds
exactly one root:

curl -s --cacert only_globalsign.pem -o /dev/null -w "%{http_code}\n" https://mustafaerbay.com.tr/
# 200   the only root in the list is GlobalSign; the cross-signed path carried it
curl -s --cacert r4_brew.pem -o /dev/null -w "%{http_code}\n" https://mustafaerbay.com.tr/
# 200   the only root is the self-signed R4 (Homebrew copy)
curl -s --cacert r4_apple.pem -o /dev/null -w "%{http_code}\n" https://mustafaerbay.com.tr/
# 200   Apple's copy of the same key: different certificate, same outcome

The third line is the most concrete proof that the anchor is a key: not through
openssl verify but over a real HTTPS request, two byte-for-byte different certificates
validate the same connection. The first line says plenty too β€” a client holding nothing
but the GlobalSign root still reaches the site, thanks to the cross-signature the server
sends.

In the earlier piece I wrote that all 158 fingerprints differed and the thing repeating
was the name. This case goes a step further: the name repeats again, the fingerprints
differ again, but what really repeats is the key β€” and the key is what verification runs on.

Which engine reads which list

There is no single "root list of my machine". This machine has at least four separate
files plus a Keychain, each carrying a different number of roots:

Engine List Certificates Distinct keys
LibreSSL 3.3.6 (/usr/bin/openssl) /etc/ssl/cert.pem 128 128
OpenSSL 3.6.4 and 4.1.0 (brew) ca-certificates/cert.pem 183 182
Node.js 22.16.0 (default) bundled Mozilla copy 150 150
Python 3.13.3 (python.org framework) cert.pem under the framework 121 121
curl 8.7.1 SecureTransport β†’ Keychain no file β€”

The curl row in the table carries a caveat: it uses the Keychain by default, but the
moment you pass --cacert or set SSL_CERT_FILE it switches to that file. Pointing it at
an empty file stopped it with curl: (77) before the handshake even began.

On the Node side you can count this with its own API: tls.getCACertificates('bundled')
returns 150, while 'system' returns just 13 on this machine. The Node documentation
describes the bundled list as the "bundled Mozilla CA store as supplied by current
Node.js version" and says the default store is selected at build time. So what Node trusts
is not your system administrator's call; it is your Node version's call.

SSL_CERT_FILE is not free in Node:
the docs say the variable only applies when --use-openssl-ca is enabled (or with
--use-system-ca on platforms other than macOS and Windows). A run with an empty file in
SSL_CERT_FILE confirms it β€” under that flag a fetch to my own blog fails, without the
flag nothing changes. NODE_EXTRA_CA_CERTS, for its part, extends the list rather than
replacing it; in the documentation's wording the well-known roots get "extended with the
extra certificates in file". You cannot disable a root with it.

This sprawl isn't a defect, it is the architecture. The current text of TLS 1.3 β€”
RFC 9846, which retired RFC 8446 in July 2026
and moved this material to section 4.5.1 β€” assumes anchors are distributed independently
and says "a certificate that specifies a trust anchor MAY be omitted from the chain". It
has one condition: knowing that the peer already has that certificate. The server cannot
know, so it guesses.

The bridge and the shore close on the same day

Sending the cross-signed root has a cost and an expiry date. The cost is settled: 894 bytes
per handshake, 35.8% of the chain. The date is where my first calculation went wrong, and
fixing it turned out to be the most useful part of this piece.

The cross-signed R4 expires on 28 January 2028. The self-signed R4 runs to 2036, so my note
read "the bridge closes eight years before the shore it reaches". That is not the shore it
reaches. Path B's anchor is GlobalSign Root CA, and its dates look like this:

openssl x509 -in only_globalsign.pem -noout -dates
# notBefore=Sep  1 12:00:00 1998 GMT
# notAfter=Jan 28 12:00:00 2028 GMT

The same day. The cross-signature expires on 28 January 2028 at 00:00:42, the root it
reaches at noon the same day β€” the cross certificate's life is tucked just under its
issuer's. So what disappears on that date is not the bridge but the whole of Path B.

You don't have to guess; -attime will tell you, described in the docs as performing
validation checks with the given timestamp instead of system time. The one trick is
choosing the target: ask about the leaf and its own calendar hides the answer, ask about
the intermediate and the boundary stands alone.

# today, against the store that does not know R4: Path B works
openssl verify -show_chain -CAfile store_no_r4.pem -untrusted cert2.pem cert1.pem
# cert1.pem: OK
# depth=0: ... CN=WE1 (untrusted)
# depth=1: ... CN=GTS Root R4 (untrusted)
# depth=2: ... CN=GlobalSign Root CA

# 1 February 2028, same store: two links expired at once
openssl verify -attime 1832976000 -CAfile store_no_r4.pem -untrusted cert2.pem cert1.pem
# C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA
# error 10 at 2 depth lookup: certificate has expired
# C=US, O=Google Trust Services LLC, CN=GTS Root R4
# error 10 at 1 depth lookup: certificate has expired
# error cert1.pem: verification failed

# 1 February 2028, a store carrying the self-signed R4: untouched
openssl verify -show_chain -attime 1832976000 -untrusted cert2.pem cert1.pem
# cert1.pem: OK   (depth=1 is GTS Root R4, valid to 2036)

My leaf plays no part in this story: it expires on 8 February 2027 and, with its 90-day
lifetime, will be renewed close to five times before 28 January 2028 (437 days). The point
is elsewhere. At the end of that January, a client holding nothing but the GlobalSign root
can no longer reach this site, and the only thing that saves it is having taken the
self-signed R4 into its list. The most likely scenario is Cloudflare quietly swapping the
chain before then; my responsibility is whether I have a check that notices.

Making an intermediate the anchor

Sometimes the public chain is not what you want: an internal PKI, pinning, or a service
that should trust exactly one intermediate CA. OpenSSL has a flag for that,
-partial_chain, and its documented definition is crisp: "Allow verification to succeed
if an incomplete chain can be built."

# make WE1 the anchor and shut the default store off entirely
openssl verify -no-CAfile -no-CApath -trusted cert1.pem cert0.pem
# error 2 at 1 depth lookup: unable to get issuer certificate

openssl verify -show_chain -partial_chain -no-CAfile -no-CApath -trusted cert1.pem cert0.pem
# cert0.pem: OK
# Chain:
# depth=0: CN=mustafaerbay.com.tr (untrusted)
# depth=1: C=US, O=Google Trust Services, CN=WE1

Two links. The chain ends at the intermediate, because that is the anchor now. Without
the flag the same command says it cannot find an issuer, since the default behaviour is to
climb until it reaches a self-signed certificate. The documentation states that stopping
rule as well: chain construction stops once an apparently self-signed certificate has been
added.

-partial_chain is powerful and double-edged: a renewal of the intermediate CA breaks
your trust, because your anchor is no longer a root's long life but a middleman's short
one. In an internal PKI that is usually what I want; for a client facing the open internet
it is not.

What this recipe does not measure

One warning before the recipe: openssl verify validates a chain, not a connection. The
biggest gap is the name β€” by default it isn't checked at all:

openssl verify -untrusted sent_chain.pem cert0.pem
# cert0.pem: OK
openssl verify -verify_hostname example.com -untrusted sent_chain.pem cert0.pem
# CN=mustafaerbay.com.tr
# error 62 at 0 depth lookup: hostname mismatch
# error cert0.pem: verification failed

Same certificate, same list, same date; one command says OK, the other says the name
doesn't match. So the OK lines below mean "this chain reaches this list", not "this
connection is safe". The other topics outside the scope are not minor either: revocation
(OCSP and CRL β€” openssl verify does not ask by default), client certificates, pinning,
the private roots of corporate interception proxies, and the ca-certificates version
baked into your container images. Every one of them changes the list side; this piece
measured only what the list does to the chain.

Auditing your own chain

The six steps below pull out where a chain lands for a domain in about ten minutes. Run
them with your own domain.

# 1) Pull the chain off the wire and split it into files
echo | openssl s_client -connect example.com:443 -servername example.com \
  -showcerts 2>/dev/null > chain_raw.txt

rm -f cert*.pem
awk 'BEGIN{n=0}
  /-----BEGIN CERTIFICATE-----/{inc=1; out=sprintf("cert%d.pem", n)}
  inc{print >> out}
  /-----END CERTIFICATE-----/{inc=0; close(out); n++}' chain_raw.txt
cat cert[1-9]*.pem > sent_chain.pem   # EVERYTHING but the leaf (chains can exceed 3)

# 2) Read who each certificate is (subject/issuer/dates)
openssl x509 -in cert1.pem -noout -subject -issuer -dates

# 3) See the path with the default store
openssl verify -show_chain -untrusted sent_chain.pem cert0.pem

# 4) Ask the same question of the store your app actually reads
openssl verify -show_chain -CAfile /etc/ssl/cert.pem \
  -untrusted sent_chain.pem cert0.pem

# 5) Are the extra certificates really needed: drop them and ask again
openssl verify -show_chain -untrusted cert1.pem cert0.pem

# 6) What happens on a future date (GNU: date -d '+6 months' +%s)
openssl verify -attime $(date -v+6m +%s) -untrusted sent_chain.pem cert0.pem

Step four is the one most often skipped. openssl verify uses the store of the binary in
your shell; the service in production may be reading a different store β€” and probably is.
On my machine the same question goes to four separate files, and the four have four
different lengths.

One caution when reading the output of step six: if the error lands at depth 0 the problem
is in the leaf, deeper than that and it is in the intermediate or root layer. The error
code carries information too β€” 20 means no local issuer was found (the chain is incomplete
or the anchor is unrecognised), 2 means there is no issuer certificate to climb to, and 10
means expired.

A decision frame

Should you send the cross-signed root? Three questions settle it for me:

  • Can you update your clients' root lists? If they are all servers and containers you operate, don't send it; the bytes are pure waste. If a mobile app, an embedded device or an old Java/Android stack is in the picture, send it β€” while knowing it has a deadline: in my chain the window where the cross-signature helps closes on 28 January 2028, because the root it reaches expires that day too.
  • How many bytes are you paying? In my chain, 894 bytes, 35.8%. On a small, chatty API that is a measurable difference; for a blog making a handful of handshakes per page it is noise.
  • Is the bridge's end date in your calendar? Mine is 28 January 2028 β€” and what ends that day is not one certificate but the entire path that cross-signature opens. Point -attime at the intermediate and you can ask about that day today, so it won't be a surprise.

And one rule for the audit side: do not compare root lists by fingerprint. Compare
them by the digest of the public key, or you will get the five false alarms this run
produced. A fingerprint is the right tool for identifying a certificate; for identifying
an anchor, the key is.

Trust is the thing you already hold

Three certificates arrived, one of them went unused, and then one line left the list and
that unused certificate became the backbone of the chain. The only thing that changed was
the list in my hands.

The lesson I take from this: a chain is not proof, it is a suggestion. The server says
"would you like to try this path", and the decision belongs to those keys that have been
sitting on your machine all along. "The certificate is valid" is an incomplete sentence.
The full one reads: this certificate, with this list, on this date, for this name, is
valid.
Change any of the four and the answer changes β€” and the way to see it coming is to look at your own
list rather than at the chain.

The day it became clear that timeouts run on
the gap between two bytes rather than the number you wrote,
and the day it became clear that
the CA, not the server, now dictates renewal timing,
both ended up in the same place: in TLS, the decision almost never belongs to the side you
thought you were writing it on.

Official Sources

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.