rustc_metadata/
locator.rs

1//! Finds crate binaries and loads their metadata
2//!
3//! Might I be the first to welcome you to a world of platform differences,
4//! version requirements, dependency graphs, conflicting desires, and fun! This
5//! is the major guts (along with metadata::creader) of the compiler for loading
6//! crates and resolving dependencies. Let's take a tour!
7//!
8//! # The problem
9//!
10//! Each invocation of the compiler is immediately concerned with one primary
11//! problem, to connect a set of crates to resolved crates on the filesystem.
12//! Concretely speaking, the compiler follows roughly these steps to get here:
13//!
14//! 1. Discover a set of `extern crate` statements.
15//! 2. Transform these directives into crate names. If the directive does not
16//!    have an explicit name, then the identifier is the name.
17//! 3. For each of these crate names, find a corresponding crate on the
18//!    filesystem.
19//!
20//! Sounds easy, right? Let's walk into some of the nuances.
21//!
22//! ## Transitive Dependencies
23//!
24//! Let's say we've got three crates: A, B, and C. A depends on B, and B depends
25//! on C. When we're compiling A, we primarily need to find and locate B, but we
26//! also end up needing to find and locate C as well.
27//!
28//! The reason for this is that any of B's types could be composed of C's types,
29//! any function in B could return a type from C, etc. To be able to guarantee
30//! that we can always type-check/translate any function, we have to have
31//! complete knowledge of the whole ecosystem, not just our immediate
32//! dependencies.
33//!
34//! So now as part of the "find a corresponding crate on the filesystem" step
35//! above, this involves also finding all crates for *all upstream
36//! dependencies*. This includes all dependencies transitively.
37//!
38//! ## Rlibs and Dylibs
39//!
40//! The compiler has two forms of intermediate dependencies. These are dubbed
41//! rlibs and dylibs for the static and dynamic variants, respectively. An rlib
42//! is a rustc-defined file format (currently just an ar archive) while a dylib
43//! is a platform-defined dynamic library. Each library has a metadata somewhere
44//! inside of it.
45//!
46//! A third kind of dependency is an rmeta file. These are metadata files and do
47//! not contain any code, etc. To a first approximation, these are treated in the
48//! same way as rlibs. Where there is both an rlib and an rmeta file, the rlib
49//! gets priority (even if the rmeta file is newer). An rmeta file is only
50//! useful for checking a downstream crate, attempting to link one will cause an
51//! error.
52//!
53//! When translating a crate name to a crate on the filesystem, we all of a
54//! sudden need to take into account both rlibs and dylibs! Linkage later on may
55//! use either one of these files, as each has their pros/cons. The job of crate
56//! loading is to discover what's possible by finding all candidates.
57//!
58//! Most parts of this loading systems keep the dylib/rlib as just separate
59//! variables.
60//!
61//! ## Where to look?
62//!
63//! We can't exactly scan your whole hard drive when looking for dependencies,
64//! so we need to places to look. Currently the compiler will implicitly add the
65//! target lib search path ($prefix/lib/rustlib/$target/lib) to any compilation,
66//! and otherwise all -L flags are added to the search paths.
67//!
68//! ## What criterion to select on?
69//!
70//! This is a pretty tricky area of loading crates. Given a file, how do we know
71//! whether it's the right crate? Currently, the rules look along these lines:
72//!
73//! 1. Does the filename match an rlib/dylib pattern? That is to say, does the
74//!    filename have the right prefix/suffix?
75//! 2. Does the filename have the right prefix for the crate name being queried?
76//!    This is filtering for files like `libfoo*.rlib` and such. If the crate
77//!    we're looking for was originally compiled with -C extra-filename, the
78//!    extra filename will be included in this prefix to reduce reading
79//!    metadata from crates that would otherwise share our prefix.
80//! 3. Is the file an actual rust library? This is done by loading the metadata
81//!    from the library and making sure it's actually there.
82//! 4. Does the name in the metadata agree with the name of the library?
83//! 5. Does the target in the metadata agree with the current target?
84//! 6. Does the SVH match? (more on this later)
85//!
86//! If the file answers `yes` to all these questions, then the file is
87//! considered as being *candidate* for being accepted. It is illegal to have
88//! more than two candidates as the compiler has no method by which to resolve
89//! this conflict. Additionally, rlib/dylib candidates are considered
90//! separately.
91//!
92//! After all this has happened, we have 1 or two files as candidates. These
93//! represent the rlib/dylib file found for a library, and they're returned as
94//! being found.
95//!
96//! ### What about versions?
97//!
98//! A lot of effort has been put forth to remove versioning from the compiler.
99//! There have been forays in the past to have versioning baked in, but it was
100//! largely always deemed insufficient to the point that it was recognized that
101//! it's probably something the compiler shouldn't do anyway due to its
102//! complicated nature and the state of the half-baked solutions.
103//!
104//! With a departure from versioning, the primary criterion for loading crates
105//! is just the name of a crate. If we stopped here, it would imply that you
106//! could never link two crates of the same name from different sources
107//! together, which is clearly a bad state to be in.
108//!
109//! To resolve this problem, we come to the next section!
110//!
111//! # Expert Mode
112//!
113//! A number of flags have been added to the compiler to solve the "version
114//! problem" in the previous section, as well as generally enabling more
115//! powerful usage of the crate loading system of the compiler. The goal of
116//! these flags and options are to enable third-party tools to drive the
117//! compiler with prior knowledge about how the world should look.
118//!
119//! ## The `--extern` flag
120//!
121//! The compiler accepts a flag of this form a number of times:
122//!
123//! ```text
124//! --extern crate-name=path/to/the/crate.rlib
125//! ```
126//!
127//! This flag is basically the following letter to the compiler:
128//!
129//! > Dear rustc,
130//! >
131//! > When you are attempting to load the immediate dependency `crate-name`, I
132//! > would like you to assume that the library is located at
133//! > `path/to/the/crate.rlib`, and look nowhere else. Also, please do not
134//! > assume that the path I specified has the name `crate-name`.
135//!
136//! This flag basically overrides most matching logic except for validating that
137//! the file is indeed a rust library. The same `crate-name` can be specified
138//! twice to specify the rlib/dylib pair.
139//!
140//! ## Enabling "multiple versions"
141//!
142//! This basically boils down to the ability to specify arbitrary packages to
143//! the compiler. For example, if crate A wanted to use Bv1 and Bv2, then it
144//! would look something like:
145//!
146//! ```compile_fail,E0463
147//! extern crate b1;
148//! extern crate b2;
149//!
150//! fn main() {}
151//! ```
152//!
153//! and the compiler would be invoked as:
154//!
155//! ```text
156//! rustc a.rs --extern b1=path/to/libb1.rlib --extern b2=path/to/libb2.rlib
157//! ```
158//!
159//! In this scenario there are two crates named `b` and the compiler must be
160//! manually driven to be informed where each crate is.
161//!
162//! ## Frobbing symbols
163//!
164//! One of the immediate problems with linking the same library together twice
165//! in the same problem is dealing with duplicate symbols. The primary way to
166//! deal with this in rustc is to add hashes to the end of each symbol.
167//!
168//! In order to force hashes to change between versions of a library, if
169//! desired, the compiler exposes an option `-C metadata=foo`, which is used to
170//! initially seed each symbol hash. The string `foo` is prepended to each
171//! string-to-hash to ensure that symbols change over time.
172//!
173//! ## Loading transitive dependencies
174//!
175//! Dealing with same-named-but-distinct crates is not just a local problem, but
176//! one that also needs to be dealt with for transitive dependencies. Note that
177//! in the letter above `--extern` flags only apply to the *local* set of
178//! dependencies, not the upstream transitive dependencies. Consider this
179//! dependency graph:
180//!
181//! ```text
182//! A.1   A.2
183//! |     |
184//! |     |
185//! B     C
186//!  \   /
187//!   \ /
188//!    D
189//! ```
190//!
191//! In this scenario, when we compile `D`, we need to be able to distinctly
192//! resolve `A.1` and `A.2`, but an `--extern` flag cannot apply to these
193//! transitive dependencies.
194//!
195//! Note that the key idea here is that `B` and `C` are both *already compiled*.
196//! That is, they have already resolved their dependencies. Due to unrelated
197//! technical reasons, when a library is compiled, it is only compatible with
198//! the *exact same* version of the upstream libraries it was compiled against.
199//! We use the "Strict Version Hash" to identify the exact copy of an upstream
200//! library.
201//!
202//! With this knowledge, we know that `B` and `C` will depend on `A` with
203//! different SVH values, so we crawl the normal `-L` paths looking for
204//! `liba*.rlib` and filter based on the contained SVH.
205//!
206//! In the end, this ends up not needing `--extern` to specify upstream
207//! transitive dependencies.
208//!
209//! # Wrapping up
210//!
211//! That's the general overview of loading crates in the compiler, but it's by
212//! no means all of the necessary details. Take a look at the rest of
213//! metadata::locator or metadata::creader for all the juicy details!
214
215use std::borrow::Cow;
216use std::io::{Result as IoResult, Write};
217use std::ops::Deref;
218use std::path::{Path, PathBuf};
219use std::{cmp, fmt};
220
221use rustc_data_structures::fx::{FxHashSet, FxIndexMap};
222use rustc_data_structures::memmap::Mmap;
223use rustc_data_structures::owned_slice::{OwnedSlice, slice_owned};
224use rustc_data_structures::svh::Svh;
225use rustc_errors::{DiagArgValue, IntoDiagArg};
226use rustc_fs_util::try_canonicalize;
227use rustc_session::Session;
228use rustc_session::cstore::CrateSource;
229use rustc_session::filesearch::FileSearch;
230use rustc_session::search_paths::PathKind;
231use rustc_session::utils::CanonicalizedPath;
232use rustc_span::{Span, Symbol};
233use rustc_target::spec::{Target, TargetTuple};
234use tempfile::Builder as TempFileBuilder;
235use tracing::{debug, info};
236
237use crate::creader::{Library, MetadataLoader};
238use crate::errors;
239use crate::rmeta::{METADATA_HEADER, MetadataBlob, rustc_version};
240
241#[derive(Clone)]
242pub(crate) struct CrateLocator<'a> {
243    // Immutable per-session configuration.
244    only_needs_metadata: bool,
245    sysroot: &'a Path,
246    metadata_loader: &'a dyn MetadataLoader,
247    cfg_version: &'static str,
248
249    // Immutable per-search configuration.
250    crate_name: Symbol,
251    exact_paths: Vec<CanonicalizedPath>,
252    pub hash: Option<Svh>,
253    extra_filename: Option<&'a str>,
254    pub target: &'a Target,
255    pub tuple: TargetTuple,
256    pub filesearch: &'a FileSearch,
257    pub is_proc_macro: bool,
258
259    pub path_kind: PathKind,
260    // Mutable in-progress state or output.
261    crate_rejections: CrateRejections,
262}
263
264#[derive(Clone, Debug)]
265pub(crate) struct CratePaths {
266    pub(crate) name: Symbol,
267    source: CrateSource,
268}
269
270impl CratePaths {
271    pub(crate) fn new(name: Symbol, source: CrateSource) -> CratePaths {
272        CratePaths { name, source }
273    }
274}
275
276#[derive(Copy, Clone, Debug, PartialEq)]
277pub(crate) enum CrateFlavor {
278    Rlib,
279    Rmeta,
280    Dylib,
281    SDylib,
282}
283
284impl fmt::Display for CrateFlavor {
285    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
286        f.write_str(match *self {
287            CrateFlavor::Rlib => "rlib",
288            CrateFlavor::Rmeta => "rmeta",
289            CrateFlavor::Dylib => "dylib",
290            CrateFlavor::SDylib => "sdylib",
291        })
292    }
293}
294
295impl IntoDiagArg for CrateFlavor {
296    fn into_diag_arg(self, _: &mut Option<std::path::PathBuf>) -> rustc_errors::DiagArgValue {
297        match self {
298            CrateFlavor::Rlib => DiagArgValue::Str(Cow::Borrowed("rlib")),
299            CrateFlavor::Rmeta => DiagArgValue::Str(Cow::Borrowed("rmeta")),
300            CrateFlavor::Dylib => DiagArgValue::Str(Cow::Borrowed("dylib")),
301            CrateFlavor::SDylib => DiagArgValue::Str(Cow::Borrowed("sdylib")),
302        }
303    }
304}
305
306impl<'a> CrateLocator<'a> {
307    pub(crate) fn new(
308        sess: &'a Session,
309        metadata_loader: &'a dyn MetadataLoader,
310        crate_name: Symbol,
311        is_rlib: bool,
312        hash: Option<Svh>,
313        extra_filename: Option<&'a str>,
314        path_kind: PathKind,
315    ) -> CrateLocator<'a> {
316        let needs_object_code = sess.opts.output_types.should_codegen();
317        // If we're producing an rlib, then we don't need object code.
318        // Or, if we're not producing object code, then we don't need it either
319        // (e.g., if we're a cdylib but emitting just metadata).
320        let only_needs_metadata = is_rlib || !needs_object_code;
321
322        CrateLocator {
323            only_needs_metadata,
324            sysroot: &sess.sysroot,
325            metadata_loader,
326            cfg_version: sess.cfg_version,
327            crate_name,
328            exact_paths: if hash.is_none() {
329                sess.opts
330                    .externs
331                    .get(crate_name.as_str())
332                    .into_iter()
333                    .filter_map(|entry| entry.files())
334                    .flatten()
335                    .cloned()
336                    .collect()
337            } else {
338                // SVH being specified means this is a transitive dependency,
339                // so `--extern` options do not apply.
340                Vec::new()
341            },
342            hash,
343            extra_filename,
344            target: &sess.target,
345            tuple: sess.opts.target_triple.clone(),
346            filesearch: sess.target_filesearch(),
347            path_kind,
348            is_proc_macro: false,
349            crate_rejections: CrateRejections::default(),
350        }
351    }
352
353    pub(crate) fn reset(&mut self) {
354        self.crate_rejections.via_hash.clear();
355        self.crate_rejections.via_triple.clear();
356        self.crate_rejections.via_kind.clear();
357        self.crate_rejections.via_version.clear();
358        self.crate_rejections.via_filename.clear();
359        self.crate_rejections.via_invalid.clear();
360    }
361
362    pub(crate) fn maybe_load_library_crate(&mut self) -> Result<Option<Library>, CrateError> {
363        if !self.exact_paths.is_empty() {
364            return self.find_commandline_library();
365        }
366        let mut seen_paths = FxHashSet::default();
367        if let Some(extra_filename) = self.extra_filename {
368            if let library @ Some(_) = self.find_library_crate(extra_filename, &mut seen_paths)? {
369                return Ok(library);
370            }
371        }
372        self.find_library_crate("", &mut seen_paths)
373    }
374
375    fn find_library_crate(
376        &mut self,
377        extra_prefix: &str,
378        seen_paths: &mut FxHashSet<PathBuf>,
379    ) -> Result<Option<Library>, CrateError> {
380        let rmeta_prefix = &format!("lib{}{}", self.crate_name, extra_prefix);
381        let rlib_prefix = rmeta_prefix;
382        let dylib_prefix =
383            &format!("{}{}{}", self.target.dll_prefix, self.crate_name, extra_prefix);
384        let staticlib_prefix =
385            &format!("{}{}{}", self.target.staticlib_prefix, self.crate_name, extra_prefix);
386        let interface_prefix = rmeta_prefix;
387
388        let rmeta_suffix = ".rmeta";
389        let rlib_suffix = ".rlib";
390        let dylib_suffix = &self.target.dll_suffix;
391        let staticlib_suffix = &self.target.staticlib_suffix;
392        let interface_suffix = ".rs";
393
394        let mut candidates: FxIndexMap<
395            _,
396            (FxIndexMap<_, _>, FxIndexMap<_, _>, FxIndexMap<_, _>, FxIndexMap<_, _>),
397        > = Default::default();
398
399        // First, find all possible candidate rlibs and dylibs purely based on
400        // the name of the files themselves. We're trying to match against an
401        // exact crate name and a possibly an exact hash.
402        //
403        // During this step, we can filter all found libraries based on the
404        // name and id found in the crate id (we ignore the path portion for
405        // filename matching), as well as the exact hash (if specified). If we
406        // end up having many candidates, we must look at the metadata to
407        // perform exact matches against hashes/crate ids. Note that opening up
408        // the metadata is where we do an exact match against the full contents
409        // of the crate id (path/name/id).
410        //
411        // The goal of this step is to look at as little metadata as possible.
412        // Unfortunately, the prefix-based matching sometimes is over-eager.
413        // E.g. if `rlib_suffix` is `libstd` it'll match the file
414        // `libstd_detect-8d6701fb958915ad.rlib` (incorrect) as well as
415        // `libstd-f3ab5b1dea981f17.rlib` (correct). But this is hard to avoid
416        // given that `extra_filename` comes from the `-C extra-filename`
417        // option and thus can be anything, and the incorrect match will be
418        // handled safely in `extract_one`.
419        for search_path in self.filesearch.search_paths(self.path_kind) {
420            debug!("searching {}", search_path.dir.display());
421            let spf = &search_path.files;
422
423            let mut should_check_staticlibs = true;
424            for (prefix, suffix, kind) in [
425                (rlib_prefix.as_str(), rlib_suffix, CrateFlavor::Rlib),
426                (rmeta_prefix.as_str(), rmeta_suffix, CrateFlavor::Rmeta),
427                (dylib_prefix, dylib_suffix, CrateFlavor::Dylib),
428                (interface_prefix, interface_suffix, CrateFlavor::SDylib),
429            ] {
430                if prefix == staticlib_prefix && suffix == staticlib_suffix {
431                    should_check_staticlibs = false;
432                }
433                if let Some(matches) = spf.query(prefix, suffix) {
434                    for (hash, spf) in matches {
435                        info!("lib candidate: {}", spf.path.display());
436
437                        let (rlibs, rmetas, dylibs, interfaces) =
438                            candidates.entry(hash).or_default();
439                        {
440                            // As a perforamnce optimisation we canonicalize the path and skip
441                            // ones we've already seeen. This allows us to ignore crates
442                            // we know are exactual equal to ones we've already found.
443                            // Going to the same crate through different symlinks does not change the result.
444                            let path = try_canonicalize(&spf.path)
445                                .unwrap_or_else(|_| spf.path.to_path_buf());
446                            if seen_paths.contains(&path) {
447                                continue;
448                            };
449                            seen_paths.insert(path);
450                        }
451                        // Use the original path (potentially with unresolved symlinks),
452                        // filesystem code should not care, but this is nicer for diagnostics.
453                        let path = spf.path.to_path_buf();
454                        match kind {
455                            CrateFlavor::Rlib => rlibs.insert(path, search_path.kind),
456                            CrateFlavor::Rmeta => rmetas.insert(path, search_path.kind),
457                            CrateFlavor::Dylib => dylibs.insert(path, search_path.kind),
458                            CrateFlavor::SDylib => interfaces.insert(path, search_path.kind),
459                        };
460                    }
461                }
462            }
463            if let Some(static_matches) = should_check_staticlibs
464                .then(|| spf.query(staticlib_prefix, staticlib_suffix))
465                .flatten()
466            {
467                for (_, spf) in static_matches {
468                    self.crate_rejections.via_kind.push(CrateMismatch {
469                        path: spf.path.to_path_buf(),
470                        got: "static".to_string(),
471                    });
472                }
473            }
474        }
475
476        // We have now collected all known libraries into a set of candidates
477        // keyed of the filename hash listed. For each filename, we also have a
478        // list of rlibs/dylibs that apply. Here, we map each of these lists
479        // (per hash), to a Library candidate for returning.
480        //
481        // A Library candidate is created if the metadata for the set of
482        // libraries corresponds to the crate id and hash criteria that this
483        // search is being performed for.
484        let mut libraries = FxIndexMap::default();
485        for (_hash, (rlibs, rmetas, dylibs, interfaces)) in candidates {
486            if let Some((svh, lib)) = self.extract_lib(rlibs, rmetas, dylibs, interfaces)? {
487                libraries.insert(svh, lib);
488            }
489        }
490
491        // Having now translated all relevant found hashes into libraries, see
492        // what we've got and figure out if we found multiple candidates for
493        // libraries or not.
494        match libraries.len() {
495            0 => Ok(None),
496            1 => Ok(Some(libraries.into_iter().next().unwrap().1)),
497            _ => {
498                let mut libraries: Vec<_> = libraries.into_values().collect();
499
500                libraries.sort_by_cached_key(|lib| lib.source.paths().next().unwrap().clone());
501                let candidates = libraries
502                    .iter()
503                    .map(|lib| lib.source.paths().next().unwrap().clone())
504                    .collect::<Vec<_>>();
505
506                Err(CrateError::MultipleCandidates(
507                    self.crate_name,
508                    // these are the same for all candidates
509                    get_flavor_from_path(candidates.first().unwrap()),
510                    candidates,
511                ))
512            }
513        }
514    }
515
516    fn extract_lib(
517        &mut self,
518        rlibs: FxIndexMap<PathBuf, PathKind>,
519        rmetas: FxIndexMap<PathBuf, PathKind>,
520        dylibs: FxIndexMap<PathBuf, PathKind>,
521        interfaces: FxIndexMap<PathBuf, PathKind>,
522    ) -> Result<Option<(Svh, Library)>, CrateError> {
523        let mut slot = None;
524        // Order here matters, rmeta should come first.
525        //
526        // Make sure there's at most one rlib and at most one dylib.
527        //
528        // See comment in `extract_one` below.
529        let rmeta = self.extract_one(rmetas, CrateFlavor::Rmeta, &mut slot)?;
530        let rlib = self.extract_one(rlibs, CrateFlavor::Rlib, &mut slot)?;
531        let sdylib_interface = self.extract_one(interfaces, CrateFlavor::SDylib, &mut slot)?;
532        let dylib = self.extract_one(dylibs, CrateFlavor::Dylib, &mut slot)?;
533
534        if sdylib_interface.is_some() && dylib.is_none() {
535            return Err(CrateError::FullMetadataNotFound(self.crate_name, CrateFlavor::SDylib));
536        }
537
538        let source = CrateSource { rmeta, rlib, dylib, sdylib_interface };
539        Ok(slot.map(|(svh, metadata, _, _)| (svh, Library { source, metadata })))
540    }
541
542    fn needs_crate_flavor(&self, flavor: CrateFlavor) -> bool {
543        if flavor == CrateFlavor::Dylib && self.is_proc_macro {
544            return true;
545        }
546
547        if self.only_needs_metadata {
548            flavor == CrateFlavor::Rmeta
549        } else {
550            // we need all flavors (perhaps not true, but what we do for now)
551            true
552        }
553    }
554
555    // Attempts to extract *one* library from the set `m`. If the set has no
556    // elements, `None` is returned. If the set has more than one element, then
557    // the errors and notes are emitted about the set of libraries.
558    //
559    // With only one library in the set, this function will extract it, and then
560    // read the metadata from it if `*slot` is `None`. If the metadata couldn't
561    // be read, it is assumed that the file isn't a valid rust library (no
562    // errors are emitted).
563    //
564    // The `PathBuf` in `slot` will only be used for diagnostic purposes.
565    fn extract_one(
566        &mut self,
567        m: FxIndexMap<PathBuf, PathKind>,
568        flavor: CrateFlavor,
569        slot: &mut Option<(Svh, MetadataBlob, PathBuf, CrateFlavor)>,
570    ) -> Result<Option<(PathBuf, PathKind)>, CrateError> {
571        // If we are producing an rlib, and we've already loaded metadata, then
572        // we should not attempt to discover further crate sources (unless we're
573        // locating a proc macro; exact logic is in needs_crate_flavor). This means
574        // that under -Zbinary-dep-depinfo we will not emit a dependency edge on
575        // the *unused* rlib, and by returning `None` here immediately we
576        // guarantee that we do indeed not use it.
577        //
578        // See also #68149 which provides more detail on why emitting the
579        // dependency on the rlib is a bad thing.
580        if slot.is_some() {
581            if m.is_empty() || !self.needs_crate_flavor(flavor) {
582                return Ok(None);
583            }
584        }
585
586        let mut ret: Option<(PathBuf, PathKind)> = None;
587        let mut err_data: Option<Vec<PathBuf>> = None;
588        for (lib, kind) in m {
589            info!("{} reading metadata from: {}", flavor, lib.display());
590            if flavor == CrateFlavor::Rmeta && lib.metadata().is_ok_and(|m| m.len() == 0) {
591                // Empty files will cause get_metadata_section to fail. Rmeta
592                // files can be empty, for example with binaries (which can
593                // often appear with `cargo check` when checking a library as
594                // a unittest). We don't want to emit a user-visible warning
595                // in this case as it is not a real problem.
596                debug!("skipping empty file");
597                continue;
598            }
599            let (hash, metadata) = match get_metadata_section(
600                self.target,
601                flavor,
602                &lib,
603                self.metadata_loader,
604                self.cfg_version,
605                Some(self.crate_name),
606            ) {
607                Ok(blob) => {
608                    if let Some(h) = self.crate_matches(&blob, &lib) {
609                        (h, blob)
610                    } else {
611                        info!("metadata mismatch");
612                        continue;
613                    }
614                }
615                Err(MetadataError::VersionMismatch { expected_version, found_version }) => {
616                    // The file was present and created by the same compiler version, but we
617                    // couldn't load it for some reason. Give a hard error instead of silently
618                    // ignoring it, but only if we would have given an error anyway.
619                    info!(
620                        "Rejecting via version: expected {} got {}",
621                        expected_version, found_version
622                    );
623                    self.crate_rejections
624                        .via_version
625                        .push(CrateMismatch { path: lib, got: found_version });
626                    continue;
627                }
628                Err(MetadataError::LoadFailure(err)) => {
629                    info!("no metadata found: {}", err);
630                    // Metadata was loaded from interface file earlier.
631                    if let Some((.., CrateFlavor::SDylib)) = slot {
632                        ret = Some((lib, kind));
633                        continue;
634                    }
635                    // The file was present and created by the same compiler version, but we
636                    // couldn't load it for some reason. Give a hard error instead of silently
637                    // ignoring it, but only if we would have given an error anyway.
638                    self.crate_rejections.via_invalid.push(CrateMismatch { path: lib, got: err });
639                    continue;
640                }
641                Err(err @ MetadataError::NotPresent(_)) => {
642                    info!("no metadata found: {}", err);
643                    continue;
644                }
645            };
646            // If we see multiple hashes, emit an error about duplicate candidates.
647            if slot.as_ref().is_some_and(|s| s.0 != hash) {
648                if let Some(candidates) = err_data {
649                    return Err(CrateError::MultipleCandidates(
650                        self.crate_name,
651                        flavor,
652                        candidates,
653                    ));
654                }
655                err_data = Some(vec![slot.take().unwrap().2]);
656            }
657            if let Some(candidates) = &mut err_data {
658                candidates.push(lib);
659                continue;
660            }
661
662            // Ok so at this point we've determined that `(lib, kind)` above is
663            // a candidate crate to load, and that `slot` is either none (this
664            // is the first crate of its kind) or if some the previous path has
665            // the exact same hash (e.g., it's the exact same crate).
666            //
667            // In principle these two candidate crates are exactly the same so
668            // we can choose either of them to link. As a stupidly gross hack,
669            // however, we favor crate in the sysroot.
670            //
671            // You can find more info in rust-lang/rust#39518 and various linked
672            // issues, but the general gist is that during testing libstd the
673            // compilers has two candidates to choose from: one in the sysroot
674            // and one in the deps folder. These two crates are the exact same
675            // crate but if the compiler chooses the one in the deps folder
676            // it'll cause spurious errors on Windows.
677            //
678            // As a result, we favor the sysroot crate here. Note that the
679            // candidates are all canonicalized, so we canonicalize the sysroot
680            // as well.
681            if let Some((prev, _)) = &ret {
682                let sysroot = self.sysroot;
683                let sysroot = try_canonicalize(sysroot).unwrap_or_else(|_| sysroot.to_path_buf());
684                if prev.starts_with(&sysroot) {
685                    continue;
686                }
687            }
688
689            // We error eagerly here. If we're locating a rlib, then in theory the full metadata
690            // could still be in a (later resolved) dylib. In practice, if the rlib and dylib
691            // were produced in a way where one has full metadata and the other hasn't, it would
692            // mean that they were compiled using different compiler flags and probably also have
693            // a different SVH value.
694            if metadata.get_header().is_stub {
695                // `is_stub` should never be true for .rmeta files.
696                assert_ne!(flavor, CrateFlavor::Rmeta);
697
698                // Because rmeta files are resolved before rlib/dylib files, if this is a stub and
699                // we haven't found a slot already, it means that the full metadata is missing.
700                if slot.is_none() {
701                    return Err(CrateError::FullMetadataNotFound(self.crate_name, flavor));
702                }
703            } else {
704                *slot = Some((hash, metadata, lib.clone(), flavor));
705            }
706            ret = Some((lib, kind));
707        }
708
709        if let Some(candidates) = err_data {
710            Err(CrateError::MultipleCandidates(self.crate_name, flavor, candidates))
711        } else {
712            Ok(ret)
713        }
714    }
715
716    fn crate_matches(&mut self, metadata: &MetadataBlob, libpath: &Path) -> Option<Svh> {
717        let header = metadata.get_header();
718        if header.is_proc_macro_crate != self.is_proc_macro {
719            info!(
720                "Rejecting via proc macro: expected {} got {}",
721                self.is_proc_macro, header.is_proc_macro_crate,
722            );
723            return None;
724        }
725
726        if self.exact_paths.is_empty() && self.crate_name != header.name {
727            info!("Rejecting via crate name");
728            return None;
729        }
730
731        if header.triple != self.tuple {
732            info!("Rejecting via crate triple: expected {} got {}", self.tuple, header.triple);
733            self.crate_rejections.via_triple.push(CrateMismatch {
734                path: libpath.to_path_buf(),
735                got: header.triple.to_string(),
736            });
737            return None;
738        }
739
740        let hash = header.hash;
741        if let Some(expected_hash) = self.hash {
742            if hash != expected_hash {
743                info!("Rejecting via hash: expected {} got {}", expected_hash, hash);
744                self.crate_rejections
745                    .via_hash
746                    .push(CrateMismatch { path: libpath.to_path_buf(), got: hash.to_string() });
747                return None;
748            }
749        }
750
751        Some(hash)
752    }
753
754    fn find_commandline_library(&mut self) -> Result<Option<Library>, CrateError> {
755        // First, filter out all libraries that look suspicious. We only accept
756        // files which actually exist that have the correct naming scheme for
757        // rlibs/dylibs.
758        let mut rlibs = FxIndexMap::default();
759        let mut rmetas = FxIndexMap::default();
760        let mut dylibs = FxIndexMap::default();
761        let mut sdylib_interfaces = FxIndexMap::default();
762        for loc in &self.exact_paths {
763            let loc_canon = loc.canonicalized();
764            let loc_orig = loc.original();
765            if !loc_canon.exists() {
766                return Err(CrateError::ExternLocationNotExist(self.crate_name, loc_orig.clone()));
767            }
768            if !loc_orig.is_file() {
769                return Err(CrateError::ExternLocationNotFile(self.crate_name, loc_orig.clone()));
770            }
771            // Note to take care and match against the non-canonicalized name:
772            // some systems save build artifacts into content-addressed stores
773            // that do not preserve extensions, and then link to them using
774            // e.g. symbolic links. If we canonicalize too early, we resolve
775            // the symlink, the file type is lost and we might treat rlibs and
776            // rmetas as dylibs.
777            let Some(file) = loc_orig.file_name().and_then(|s| s.to_str()) else {
778                return Err(CrateError::ExternLocationNotFile(self.crate_name, loc_orig.clone()));
779            };
780            if file.starts_with("lib") {
781                if file.ends_with(".rlib") {
782                    rlibs.insert(loc_canon.clone(), PathKind::ExternFlag);
783                    continue;
784                }
785                if file.ends_with(".rmeta") {
786                    rmetas.insert(loc_canon.clone(), PathKind::ExternFlag);
787                    continue;
788                }
789                if file.ends_with(".rs") {
790                    sdylib_interfaces.insert(loc_canon.clone(), PathKind::ExternFlag);
791                }
792            }
793            let dll_prefix = self.target.dll_prefix.as_ref();
794            let dll_suffix = self.target.dll_suffix.as_ref();
795            if file.starts_with(dll_prefix) && file.ends_with(dll_suffix) {
796                dylibs.insert(loc_canon.clone(), PathKind::ExternFlag);
797                continue;
798            }
799            self.crate_rejections
800                .via_filename
801                .push(CrateMismatch { path: loc_orig.clone(), got: String::new() });
802        }
803
804        // Extract the dylib/rlib/rmeta triple.
805        self.extract_lib(rlibs, rmetas, dylibs, sdylib_interfaces)
806            .map(|opt| opt.map(|(_, lib)| lib))
807    }
808
809    pub(crate) fn into_error(self, dep_root: Option<CratePaths>) -> CrateError {
810        CrateError::LocatorCombined(Box::new(CombinedLocatorError {
811            crate_name: self.crate_name,
812            dep_root,
813            triple: self.tuple,
814            dll_prefix: self.target.dll_prefix.to_string(),
815            dll_suffix: self.target.dll_suffix.to_string(),
816            crate_rejections: self.crate_rejections,
817        }))
818    }
819}
820
821fn get_metadata_section<'p>(
822    target: &Target,
823    flavor: CrateFlavor,
824    filename: &'p Path,
825    loader: &dyn MetadataLoader,
826    cfg_version: &'static str,
827    crate_name: Option<Symbol>,
828) -> Result<MetadataBlob, MetadataError<'p>> {
829    if !filename.exists() {
830        return Err(MetadataError::NotPresent(filename));
831    }
832    let raw_bytes = match flavor {
833        CrateFlavor::Rlib => {
834            loader.get_rlib_metadata(target, filename).map_err(MetadataError::LoadFailure)?
835        }
836        CrateFlavor::SDylib => {
837            let compiler = std::env::current_exe().map_err(|_err| {
838                MetadataError::LoadFailure(
839                    "couldn't obtain current compiler binary when loading sdylib interface"
840                        .to_string(),
841                )
842            })?;
843
844            let tmp_path = match TempFileBuilder::new().prefix("rustc").tempdir() {
845                Ok(tmp_path) => tmp_path,
846                Err(error) => {
847                    return Err(MetadataError::LoadFailure(format!(
848                        "couldn't create a temp dir: {}",
849                        error
850                    )));
851                }
852            };
853
854            let crate_name = crate_name.unwrap();
855            debug!("compiling {}", filename.display());
856            // FIXME: This will need to be done either within the current compiler session or
857            // as a separate compiler session in the same process.
858            let res = std::process::Command::new(compiler)
859                .arg(&filename)
860                .arg("--emit=metadata")
861                .arg(format!("--crate-name={}", crate_name))
862                .arg(format!("--out-dir={}", tmp_path.path().display()))
863                .arg("-Zbuild-sdylib-interface")
864                .output()
865                .map_err(|err| {
866                    MetadataError::LoadFailure(format!("couldn't compile interface: {}", err))
867                })?;
868
869            if !res.status.success() {
870                return Err(MetadataError::LoadFailure(format!(
871                    "couldn't compile interface: {}",
872                    std::str::from_utf8(&res.stderr).unwrap_or_default()
873                )));
874            }
875
876            // Load interface metadata instead of crate metadata.
877            let interface_metadata_name = format!("lib{}.rmeta", crate_name);
878            let rmeta_file = tmp_path.path().join(interface_metadata_name);
879            debug!("loading interface metadata from {}", rmeta_file.display());
880            let rmeta = get_rmeta_metadata_section(&rmeta_file)?;
881            let _ = std::fs::remove_file(rmeta_file);
882
883            rmeta
884        }
885        CrateFlavor::Dylib => {
886            let buf =
887                loader.get_dylib_metadata(target, filename).map_err(MetadataError::LoadFailure)?;
888            let header_len = METADATA_HEADER.len();
889            // header + u64 length of data
890            let data_start = header_len + 8;
891
892            debug!("checking {} bytes of metadata-version stamp", header_len);
893            let header = &buf[..cmp::min(header_len, buf.len())];
894            if header != METADATA_HEADER {
895                return Err(MetadataError::LoadFailure(format!(
896                    "invalid metadata version found: {}",
897                    filename.display()
898                )));
899            }
900
901            // Length of the metadata - this allows linkers to pad the section if they want
902            let Ok(len_bytes) =
903                <[u8; 8]>::try_from(&buf[header_len..cmp::min(data_start, buf.len())])
904            else {
905                return Err(MetadataError::LoadFailure(
906                    "invalid metadata length found".to_string(),
907                ));
908            };
909            let metadata_len = u64::from_le_bytes(len_bytes) as usize;
910
911            // Header is okay -> inflate the actual metadata
912            buf.slice(|buf| &buf[data_start..(data_start + metadata_len)])
913        }
914        CrateFlavor::Rmeta => get_rmeta_metadata_section(filename)?,
915    };
916    let Ok(blob) = MetadataBlob::new(raw_bytes) else {
917        return Err(MetadataError::LoadFailure(format!(
918            "corrupt metadata encountered in {}",
919            filename.display()
920        )));
921    };
922    match blob.check_compatibility(cfg_version) {
923        Ok(()) => {
924            debug!("metadata blob read okay");
925            Ok(blob)
926        }
927        Err(None) => Err(MetadataError::LoadFailure(format!(
928            "invalid metadata version found: {}",
929            filename.display()
930        ))),
931        Err(Some(found_version)) => {
932            return Err(MetadataError::VersionMismatch {
933                expected_version: rustc_version(cfg_version),
934                found_version,
935            });
936        }
937    }
938}
939
940fn get_rmeta_metadata_section<'a, 'p>(filename: &'p Path) -> Result<OwnedSlice, MetadataError<'a>> {
941    // mmap the file, because only a small fraction of it is read.
942    let file = std::fs::File::open(filename).map_err(|_| {
943        MetadataError::LoadFailure(format!(
944            "failed to open rmeta metadata: '{}'",
945            filename.display()
946        ))
947    })?;
948    let mmap = unsafe { Mmap::map(file) };
949    let mmap = mmap.map_err(|_| {
950        MetadataError::LoadFailure(format!(
951            "failed to mmap rmeta metadata: '{}'",
952            filename.display()
953        ))
954    })?;
955
956    Ok(slice_owned(mmap, Deref::deref))
957}
958
959/// A diagnostic function for dumping crate metadata to an output stream.
960pub fn list_file_metadata(
961    target: &Target,
962    path: &Path,
963    metadata_loader: &dyn MetadataLoader,
964    out: &mut dyn Write,
965    ls_kinds: &[String],
966    cfg_version: &'static str,
967) -> IoResult<()> {
968    let flavor = get_flavor_from_path(path);
969    match get_metadata_section(target, flavor, path, metadata_loader, cfg_version, None) {
970        Ok(metadata) => metadata.list_crate_metadata(out, ls_kinds),
971        Err(msg) => write!(out, "{msg}\n"),
972    }
973}
974
975fn get_flavor_from_path(path: &Path) -> CrateFlavor {
976    let filename = path.file_name().unwrap().to_str().unwrap();
977
978    if filename.ends_with(".rlib") {
979        CrateFlavor::Rlib
980    } else if filename.ends_with(".rmeta") {
981        CrateFlavor::Rmeta
982    } else {
983        CrateFlavor::Dylib
984    }
985}
986
987// ------------------------------------------ Error reporting -------------------------------------
988
989#[derive(Clone, Debug)]
990struct CrateMismatch {
991    path: PathBuf,
992    got: String,
993}
994
995#[derive(Clone, Debug, Default)]
996struct CrateRejections {
997    via_hash: Vec<CrateMismatch>,
998    via_triple: Vec<CrateMismatch>,
999    via_kind: Vec<CrateMismatch>,
1000    via_version: Vec<CrateMismatch>,
1001    via_filename: Vec<CrateMismatch>,
1002    via_invalid: Vec<CrateMismatch>,
1003}
1004
1005/// Candidate rejection reasons collected during crate search.
1006/// If no candidate is accepted, then these reasons are presented to the user,
1007/// otherwise they are ignored.
1008#[derive(Debug)]
1009pub(crate) struct CombinedLocatorError {
1010    crate_name: Symbol,
1011    dep_root: Option<CratePaths>,
1012    triple: TargetTuple,
1013    dll_prefix: String,
1014    dll_suffix: String,
1015    crate_rejections: CrateRejections,
1016}
1017
1018#[derive(Debug)]
1019pub(crate) enum CrateError {
1020    NonAsciiName(Symbol),
1021    ExternLocationNotExist(Symbol, PathBuf),
1022    ExternLocationNotFile(Symbol, PathBuf),
1023    MultipleCandidates(Symbol, CrateFlavor, Vec<PathBuf>),
1024    FullMetadataNotFound(Symbol, CrateFlavor),
1025    SymbolConflictsCurrent(Symbol),
1026    StableCrateIdCollision(Symbol, Symbol),
1027    DlOpen(String, String),
1028    DlSym(String, String),
1029    LocatorCombined(Box<CombinedLocatorError>),
1030    NotFound(Symbol),
1031}
1032
1033enum MetadataError<'a> {
1034    /// The file was missing.
1035    NotPresent(&'a Path),
1036    /// The file was present and invalid.
1037    LoadFailure(String),
1038    /// The file was present, but compiled with a different rustc version.
1039    VersionMismatch { expected_version: String, found_version: String },
1040}
1041
1042impl fmt::Display for MetadataError<'_> {
1043    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
1044        match self {
1045            MetadataError::NotPresent(filename) => {
1046                f.write_str(&format!("no such file: '{}'", filename.display()))
1047            }
1048            MetadataError::LoadFailure(msg) => f.write_str(msg),
1049            MetadataError::VersionMismatch { expected_version, found_version } => {
1050                f.write_str(&format!(
1051                    "rustc version mismatch. expected {}, found {}",
1052                    expected_version, found_version,
1053                ))
1054            }
1055        }
1056    }
1057}
1058
1059impl CrateError {
1060    pub(crate) fn report(self, sess: &Session, span: Span, missing_core: bool) {
1061        let dcx = sess.dcx();
1062        match self {
1063            CrateError::NonAsciiName(crate_name) => {
1064                dcx.emit_err(errors::NonAsciiName { span, crate_name });
1065            }
1066            CrateError::ExternLocationNotExist(crate_name, loc) => {
1067                dcx.emit_err(errors::ExternLocationNotExist { span, crate_name, location: &loc });
1068            }
1069            CrateError::ExternLocationNotFile(crate_name, loc) => {
1070                dcx.emit_err(errors::ExternLocationNotFile { span, crate_name, location: &loc });
1071            }
1072            CrateError::MultipleCandidates(crate_name, flavor, candidates) => {
1073                dcx.emit_err(errors::MultipleCandidates { span, crate_name, flavor, candidates });
1074            }
1075            CrateError::FullMetadataNotFound(crate_name, flavor) => {
1076                dcx.emit_err(errors::FullMetadataNotFound { span, crate_name, flavor });
1077            }
1078            CrateError::SymbolConflictsCurrent(root_name) => {
1079                dcx.emit_err(errors::SymbolConflictsCurrent { span, crate_name: root_name });
1080            }
1081            CrateError::StableCrateIdCollision(crate_name0, crate_name1) => {
1082                dcx.emit_err(errors::StableCrateIdCollision { span, crate_name0, crate_name1 });
1083            }
1084            CrateError::DlOpen(path, err) | CrateError::DlSym(path, err) => {
1085                dcx.emit_err(errors::DlError { span, path, err });
1086            }
1087            CrateError::LocatorCombined(locator) => {
1088                let crate_name = locator.crate_name;
1089                let add_info = match &locator.dep_root {
1090                    None => String::new(),
1091                    Some(r) => format!(" which `{}` depends on", r.name),
1092                };
1093                if !locator.crate_rejections.via_filename.is_empty() {
1094                    let mismatches = locator.crate_rejections.via_filename.iter();
1095                    for CrateMismatch { path, .. } in mismatches {
1096                        dcx.emit_err(errors::CrateLocationUnknownType { span, path, crate_name });
1097                        dcx.emit_err(errors::LibFilenameForm {
1098                            span,
1099                            dll_prefix: &locator.dll_prefix,
1100                            dll_suffix: &locator.dll_suffix,
1101                        });
1102                    }
1103                }
1104                let mut found_crates = String::new();
1105                if !locator.crate_rejections.via_hash.is_empty() {
1106                    let mismatches = locator.crate_rejections.via_hash.iter();
1107                    for CrateMismatch { path, .. } in mismatches {
1108                        found_crates.push_str(&format!(
1109                            "\ncrate `{}`: {}",
1110                            crate_name,
1111                            path.display()
1112                        ));
1113                    }
1114                    if let Some(r) = locator.dep_root {
1115                        for path in r.source.paths() {
1116                            found_crates.push_str(&format!(
1117                                "\ncrate `{}`: {}",
1118                                r.name,
1119                                path.display()
1120                            ));
1121                        }
1122                    }
1123                    dcx.emit_err(errors::NewerCrateVersion {
1124                        span,
1125                        crate_name,
1126                        add_info,
1127                        found_crates,
1128                    });
1129                } else if !locator.crate_rejections.via_triple.is_empty() {
1130                    let mismatches = locator.crate_rejections.via_triple.iter();
1131                    for CrateMismatch { path, got } in mismatches {
1132                        found_crates.push_str(&format!(
1133                            "\ncrate `{}`, target triple {}: {}",
1134                            crate_name,
1135                            got,
1136                            path.display(),
1137                        ));
1138                    }
1139                    dcx.emit_err(errors::NoCrateWithTriple {
1140                        span,
1141                        crate_name,
1142                        locator_triple: locator.triple.tuple(),
1143                        add_info,
1144                        found_crates,
1145                    });
1146                } else if !locator.crate_rejections.via_kind.is_empty() {
1147                    let mismatches = locator.crate_rejections.via_kind.iter();
1148                    for CrateMismatch { path, .. } in mismatches {
1149                        found_crates.push_str(&format!(
1150                            "\ncrate `{}`: {}",
1151                            crate_name,
1152                            path.display()
1153                        ));
1154                    }
1155                    dcx.emit_err(errors::FoundStaticlib {
1156                        span,
1157                        crate_name,
1158                        add_info,
1159                        found_crates,
1160                    });
1161                } else if !locator.crate_rejections.via_version.is_empty() {
1162                    let mismatches = locator.crate_rejections.via_version.iter();
1163                    for CrateMismatch { path, got } in mismatches {
1164                        found_crates.push_str(&format!(
1165                            "\ncrate `{}` compiled by {}: {}",
1166                            crate_name,
1167                            got,
1168                            path.display(),
1169                        ));
1170                    }
1171                    dcx.emit_err(errors::IncompatibleRustc {
1172                        span,
1173                        crate_name,
1174                        add_info,
1175                        found_crates,
1176                        rustc_version: rustc_version(sess.cfg_version),
1177                    });
1178                } else if !locator.crate_rejections.via_invalid.is_empty() {
1179                    let mut crate_rejections = Vec::new();
1180                    for CrateMismatch { path: _, got } in locator.crate_rejections.via_invalid {
1181                        crate_rejections.push(got);
1182                    }
1183                    dcx.emit_err(errors::InvalidMetadataFiles {
1184                        span,
1185                        crate_name,
1186                        add_info,
1187                        crate_rejections,
1188                    });
1189                } else {
1190                    let error = errors::CannotFindCrate {
1191                        span,
1192                        crate_name,
1193                        add_info,
1194                        missing_core,
1195                        current_crate: sess
1196                            .opts
1197                            .crate_name
1198                            .clone()
1199                            .unwrap_or("<unknown>".to_string()),
1200                        is_nightly_build: sess.is_nightly_build(),
1201                        profiler_runtime: Symbol::intern(&sess.opts.unstable_opts.profiler_runtime),
1202                        locator_triple: locator.triple,
1203                        is_ui_testing: sess.opts.unstable_opts.ui_testing,
1204                    };
1205                    // The diagnostic for missing core is very good, but it is followed by a lot of
1206                    // other diagnostics that do not add information.
1207                    if missing_core {
1208                        dcx.emit_fatal(error);
1209                    } else {
1210                        dcx.emit_err(error);
1211                    }
1212                }
1213            }
1214            CrateError::NotFound(crate_name) => {
1215                let error = errors::CannotFindCrate {
1216                    span,
1217                    crate_name,
1218                    add_info: String::new(),
1219                    missing_core,
1220                    current_crate: sess.opts.crate_name.clone().unwrap_or("<unknown>".to_string()),
1221                    is_nightly_build: sess.is_nightly_build(),
1222                    profiler_runtime: Symbol::intern(&sess.opts.unstable_opts.profiler_runtime),
1223                    locator_triple: sess.opts.target_triple.clone(),
1224                    is_ui_testing: sess.opts.unstable_opts.ui_testing,
1225                };
1226                // The diagnostic for missing core is very good, but it is followed by a lot of
1227                // other diagnostics that do not add information.
1228                if missing_core {
1229                    dcx.emit_fatal(error);
1230                } else {
1231                    dcx.emit_err(error);
1232                }
1233            }
1234        }
1235    }
1236}