rustc_metadata/locator.rs
1//! Finds crate binaries and loads their metadata
2//!
3//! Might I be the first to welcome you to a world of platform differences,
4//! version requirements, dependency graphs, conflicting desires, and fun! This
5//! is the major guts (along with metadata::creader) of the compiler for loading
6//! crates and resolving dependencies. Let's take a tour!
7//!
8//! # The problem
9//!
10//! Each invocation of the compiler is immediately concerned with one primary
11//! problem, to connect a set of crates to resolved crates on the filesystem.
12//! Concretely speaking, the compiler follows roughly these steps to get here:
13//!
14//! 1. Discover a set of `extern crate` statements.
15//! 2. Transform these directives into crate names. If the directive does not
16//! have an explicit name, then the identifier is the name.
17//! 3. For each of these crate names, find a corresponding crate on the
18//! filesystem.
19//!
20//! Sounds easy, right? Let's walk into some of the nuances.
21//!
22//! ## Transitive Dependencies
23//!
24//! Let's say we've got three crates: A, B, and C. A depends on B, and B depends
25//! on C. When we're compiling A, we primarily need to find and locate B, but we
26//! also end up needing to find and locate C as well.
27//!
28//! The reason for this is that any of B's types could be composed of C's types,
29//! any function in B could return a type from C, etc. To be able to guarantee
30//! that we can always type-check/translate any function, we have to have
31//! complete knowledge of the whole ecosystem, not just our immediate
32//! dependencies.
33//!
34//! So now as part of the "find a corresponding crate on the filesystem" step
35//! above, this involves also finding all crates for *all upstream
36//! dependencies*. This includes all dependencies transitively.
37//!
38//! ## Rlibs and Dylibs
39//!
40//! The compiler has two forms of intermediate dependencies. These are dubbed
41//! rlibs and dylibs for the static and dynamic variants, respectively. An rlib
42//! is a rustc-defined file format (currently just an ar archive) while a dylib
43//! is a platform-defined dynamic library. Each library has a metadata somewhere
44//! inside of it.
45//!
46//! A third kind of dependency is an rmeta file. These are metadata files and do
47//! not contain any code, etc. To a first approximation, these are treated in the
48//! same way as rlibs. Where there is both an rlib and an rmeta file, the rlib
49//! gets priority (even if the rmeta file is newer). An rmeta file is only
50//! useful for checking a downstream crate, attempting to link one will cause an
51//! error.
52//!
53//! When translating a crate name to a crate on the filesystem, we all of a
54//! sudden need to take into account both rlibs and dylibs! Linkage later on may
55//! use either one of these files, as each has their pros/cons. The job of crate
56//! loading is to discover what's possible by finding all candidates.
57//!
58//! Most parts of this loading systems keep the dylib/rlib as just separate
59//! variables.
60//!
61//! ## Where to look?
62//!
63//! We can't exactly scan your whole hard drive when looking for dependencies,
64//! so we need to places to look. Currently the compiler will implicitly add the
65//! target lib search path ($prefix/lib/rustlib/$target/lib) to any compilation,
66//! and otherwise all -L flags are added to the search paths.
67//!
68//! ## What criterion to select on?
69//!
70//! This is a pretty tricky area of loading crates. Given a file, how do we know
71//! whether it's the right crate? Currently, the rules look along these lines:
72//!
73//! 1. Does the filename match an rlib/dylib pattern? That is to say, does the
74//! filename have the right prefix/suffix?
75//! 2. Does the filename have the right prefix for the crate name being queried?
76//! This is filtering for files like `libfoo*.rlib` and such. If the crate
77//! we're looking for was originally compiled with -C extra-filename, the
78//! extra filename will be included in this prefix to reduce reading
79//! metadata from crates that would otherwise share our prefix.
80//! 3. Is the file an actual rust library? This is done by loading the metadata
81//! from the library and making sure it's actually there.
82//! 4. Does the name in the metadata agree with the name of the library?
83//! 5. Does the target in the metadata agree with the current target?
84//! 6. Does the SVH match? (more on this later)
85//!
86//! If the file answers `yes` to all these questions, then the file is
87//! considered as being *candidate* for being accepted. It is illegal to have
88//! more than two candidates as the compiler has no method by which to resolve
89//! this conflict. Additionally, rlib/dylib candidates are considered
90//! separately.
91//!
92//! After all this has happened, we have 1 or two files as candidates. These
93//! represent the rlib/dylib file found for a library, and they're returned as
94//! being found.
95//!
96//! ### What about versions?
97//!
98//! A lot of effort has been put forth to remove versioning from the compiler.
99//! There have been forays in the past to have versioning baked in, but it was
100//! largely always deemed insufficient to the point that it was recognized that
101//! it's probably something the compiler shouldn't do anyway due to its
102//! complicated nature and the state of the half-baked solutions.
103//!
104//! With a departure from versioning, the primary criterion for loading crates
105//! is just the name of a crate. If we stopped here, it would imply that you
106//! could never link two crates of the same name from different sources
107//! together, which is clearly a bad state to be in.
108//!
109//! To resolve this problem, we come to the next section!
110//!
111//! # Expert Mode
112//!
113//! A number of flags have been added to the compiler to solve the "version
114//! problem" in the previous section, as well as generally enabling more
115//! powerful usage of the crate loading system of the compiler. The goal of
116//! these flags and options are to enable third-party tools to drive the
117//! compiler with prior knowledge about how the world should look.
118//!
119//! ## The `--extern` flag
120//!
121//! The compiler accepts a flag of this form a number of times:
122//!
123//! ```text
124//! --extern crate-name=path/to/the/crate.rlib
125//! ```
126//!
127//! This flag is basically the following letter to the compiler:
128//!
129//! > Dear rustc,
130//! >
131//! > When you are attempting to load the immediate dependency `crate-name`, I
132//! > would like you to assume that the library is located at
133//! > `path/to/the/crate.rlib`, and look nowhere else. Also, please do not
134//! > assume that the path I specified has the name `crate-name`.
135//!
136//! This flag basically overrides most matching logic except for validating that
137//! the file is indeed a rust library. The same `crate-name` can be specified
138//! twice to specify the rlib/dylib pair.
139//!
140//! ## Enabling "multiple versions"
141//!
142//! This basically boils down to the ability to specify arbitrary packages to
143//! the compiler. For example, if crate A wanted to use Bv1 and Bv2, then it
144//! would look something like:
145//!
146//! ```compile_fail,E0463
147//! extern crate b1;
148//! extern crate b2;
149//!
150//! fn main() {}
151//! ```
152//!
153//! and the compiler would be invoked as:
154//!
155//! ```text
156//! rustc a.rs --extern b1=path/to/libb1.rlib --extern b2=path/to/libb2.rlib
157//! ```
158//!
159//! In this scenario there are two crates named `b` and the compiler must be
160//! manually driven to be informed where each crate is.
161//!
162//! ## Frobbing symbols
163//!
164//! One of the immediate problems with linking the same library together twice
165//! in the same problem is dealing with duplicate symbols. The primary way to
166//! deal with this in rustc is to add hashes to the end of each symbol.
167//!
168//! In order to force hashes to change between versions of a library, if
169//! desired, the compiler exposes an option `-C metadata=foo`, which is used to
170//! initially seed each symbol hash. The string `foo` is prepended to each
171//! string-to-hash to ensure that symbols change over time.
172//!
173//! ## Loading transitive dependencies
174//!
175//! Dealing with same-named-but-distinct crates is not just a local problem, but
176//! one that also needs to be dealt with for transitive dependencies. Note that
177//! in the letter above `--extern` flags only apply to the *local* set of
178//! dependencies, not the upstream transitive dependencies. Consider this
179//! dependency graph:
180//!
181//! ```text
182//! A.1 A.2
183//! | |
184//! | |
185//! B C
186//! \ /
187//! \ /
188//! D
189//! ```
190//!
191//! In this scenario, when we compile `D`, we need to be able to distinctly
192//! resolve `A.1` and `A.2`, but an `--extern` flag cannot apply to these
193//! transitive dependencies.
194//!
195//! Note that the key idea here is that `B` and `C` are both *already compiled*.
196//! That is, they have already resolved their dependencies. Due to unrelated
197//! technical reasons, when a library is compiled, it is only compatible with
198//! the *exact same* version of the upstream libraries it was compiled against.
199//! We use the "Strict Version Hash" to identify the exact copy of an upstream
200//! library.
201//!
202//! With this knowledge, we know that `B` and `C` will depend on `A` with
203//! different SVH values, so we crawl the normal `-L` paths looking for
204//! `liba*.rlib` and filter based on the contained SVH.
205//!
206//! In the end, this ends up not needing `--extern` to specify upstream
207//! transitive dependencies.
208//!
209//! # Wrapping up
210//!
211//! That's the general overview of loading crates in the compiler, but it's by
212//! no means all of the necessary details. Take a look at the rest of
213//! metadata::locator or metadata::creader for all the juicy details!
214
215use std::borrow::Cow;
216use std::io::{Result as IoResult, Write};
217use std::ops::Deref;
218use std::path::{Path, PathBuf};
219use std::{cmp, fmt};
220
221use rustc_data_structures::fx::{FxHashSet, FxIndexMap};
222use rustc_data_structures::memmap::Mmap;
223use rustc_data_structures::owned_slice::{OwnedSlice, slice_owned};
224use rustc_data_structures::svh::Svh;
225use rustc_errors::{DiagArgValue, IntoDiagArg};
226use rustc_fs_util::try_canonicalize;
227use rustc_session::Session;
228use rustc_session::cstore::CrateSource;
229use rustc_session::filesearch::FileSearch;
230use rustc_session::search_paths::PathKind;
231use rustc_session::utils::CanonicalizedPath;
232use rustc_span::{Span, Symbol};
233use rustc_target::spec::{Target, TargetTuple};
234use tempfile::Builder as TempFileBuilder;
235use tracing::{debug, info};
236
237use crate::creader::{Library, MetadataLoader};
238use crate::errors;
239use crate::rmeta::{METADATA_HEADER, MetadataBlob, rustc_version};
240
241#[derive(Clone)]
242pub(crate) struct CrateLocator<'a> {
243 // Immutable per-session configuration.
244 only_needs_metadata: bool,
245 sysroot: &'a Path,
246 metadata_loader: &'a dyn MetadataLoader,
247 cfg_version: &'static str,
248
249 // Immutable per-search configuration.
250 crate_name: Symbol,
251 exact_paths: Vec<CanonicalizedPath>,
252 pub hash: Option<Svh>,
253 extra_filename: Option<&'a str>,
254 pub target: &'a Target,
255 pub tuple: TargetTuple,
256 pub filesearch: &'a FileSearch,
257 pub is_proc_macro: bool,
258
259 pub path_kind: PathKind,
260 // Mutable in-progress state or output.
261 crate_rejections: CrateRejections,
262}
263
264#[derive(Clone, Debug)]
265pub(crate) struct CratePaths {
266 pub(crate) name: Symbol,
267 source: CrateSource,
268}
269
270impl CratePaths {
271 pub(crate) fn new(name: Symbol, source: CrateSource) -> CratePaths {
272 CratePaths { name, source }
273 }
274}
275
276#[derive(Copy, Clone, Debug, PartialEq)]
277pub(crate) enum CrateFlavor {
278 Rlib,
279 Rmeta,
280 Dylib,
281 SDylib,
282}
283
284impl fmt::Display for CrateFlavor {
285 fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
286 f.write_str(match *self {
287 CrateFlavor::Rlib => "rlib",
288 CrateFlavor::Rmeta => "rmeta",
289 CrateFlavor::Dylib => "dylib",
290 CrateFlavor::SDylib => "sdylib",
291 })
292 }
293}
294
295impl IntoDiagArg for CrateFlavor {
296 fn into_diag_arg(self, _: &mut Option<std::path::PathBuf>) -> rustc_errors::DiagArgValue {
297 match self {
298 CrateFlavor::Rlib => DiagArgValue::Str(Cow::Borrowed("rlib")),
299 CrateFlavor::Rmeta => DiagArgValue::Str(Cow::Borrowed("rmeta")),
300 CrateFlavor::Dylib => DiagArgValue::Str(Cow::Borrowed("dylib")),
301 CrateFlavor::SDylib => DiagArgValue::Str(Cow::Borrowed("sdylib")),
302 }
303 }
304}
305
306impl<'a> CrateLocator<'a> {
307 pub(crate) fn new(
308 sess: &'a Session,
309 metadata_loader: &'a dyn MetadataLoader,
310 crate_name: Symbol,
311 is_rlib: bool,
312 hash: Option<Svh>,
313 extra_filename: Option<&'a str>,
314 path_kind: PathKind,
315 ) -> CrateLocator<'a> {
316 let needs_object_code = sess.opts.output_types.should_codegen();
317 // If we're producing an rlib, then we don't need object code.
318 // Or, if we're not producing object code, then we don't need it either
319 // (e.g., if we're a cdylib but emitting just metadata).
320 let only_needs_metadata = is_rlib || !needs_object_code;
321
322 CrateLocator {
323 only_needs_metadata,
324 sysroot: &sess.sysroot,
325 metadata_loader,
326 cfg_version: sess.cfg_version,
327 crate_name,
328 exact_paths: if hash.is_none() {
329 sess.opts
330 .externs
331 .get(crate_name.as_str())
332 .into_iter()
333 .filter_map(|entry| entry.files())
334 .flatten()
335 .cloned()
336 .collect()
337 } else {
338 // SVH being specified means this is a transitive dependency,
339 // so `--extern` options do not apply.
340 Vec::new()
341 },
342 hash,
343 extra_filename,
344 target: &sess.target,
345 tuple: sess.opts.target_triple.clone(),
346 filesearch: sess.target_filesearch(),
347 path_kind,
348 is_proc_macro: false,
349 crate_rejections: CrateRejections::default(),
350 }
351 }
352
353 pub(crate) fn reset(&mut self) {
354 self.crate_rejections.via_hash.clear();
355 self.crate_rejections.via_triple.clear();
356 self.crate_rejections.via_kind.clear();
357 self.crate_rejections.via_version.clear();
358 self.crate_rejections.via_filename.clear();
359 self.crate_rejections.via_invalid.clear();
360 }
361
362 pub(crate) fn maybe_load_library_crate(&mut self) -> Result<Option<Library>, CrateError> {
363 if !self.exact_paths.is_empty() {
364 return self.find_commandline_library();
365 }
366 let mut seen_paths = FxHashSet::default();
367 if let Some(extra_filename) = self.extra_filename {
368 if let library @ Some(_) = self.find_library_crate(extra_filename, &mut seen_paths)? {
369 return Ok(library);
370 }
371 }
372 self.find_library_crate("", &mut seen_paths)
373 }
374
375 fn find_library_crate(
376 &mut self,
377 extra_prefix: &str,
378 seen_paths: &mut FxHashSet<PathBuf>,
379 ) -> Result<Option<Library>, CrateError> {
380 let rmeta_prefix = &format!("lib{}{}", self.crate_name, extra_prefix);
381 let rlib_prefix = rmeta_prefix;
382 let dylib_prefix =
383 &format!("{}{}{}", self.target.dll_prefix, self.crate_name, extra_prefix);
384 let staticlib_prefix =
385 &format!("{}{}{}", self.target.staticlib_prefix, self.crate_name, extra_prefix);
386 let interface_prefix = rmeta_prefix;
387
388 let rmeta_suffix = ".rmeta";
389 let rlib_suffix = ".rlib";
390 let dylib_suffix = &self.target.dll_suffix;
391 let staticlib_suffix = &self.target.staticlib_suffix;
392 let interface_suffix = ".rs";
393
394 let mut candidates: FxIndexMap<
395 _,
396 (FxIndexMap<_, _>, FxIndexMap<_, _>, FxIndexMap<_, _>, FxIndexMap<_, _>),
397 > = Default::default();
398
399 // First, find all possible candidate rlibs and dylibs purely based on
400 // the name of the files themselves. We're trying to match against an
401 // exact crate name and a possibly an exact hash.
402 //
403 // During this step, we can filter all found libraries based on the
404 // name and id found in the crate id (we ignore the path portion for
405 // filename matching), as well as the exact hash (if specified). If we
406 // end up having many candidates, we must look at the metadata to
407 // perform exact matches against hashes/crate ids. Note that opening up
408 // the metadata is where we do an exact match against the full contents
409 // of the crate id (path/name/id).
410 //
411 // The goal of this step is to look at as little metadata as possible.
412 // Unfortunately, the prefix-based matching sometimes is over-eager.
413 // E.g. if `rlib_suffix` is `libstd` it'll match the file
414 // `libstd_detect-8d6701fb958915ad.rlib` (incorrect) as well as
415 // `libstd-f3ab5b1dea981f17.rlib` (correct). But this is hard to avoid
416 // given that `extra_filename` comes from the `-C extra-filename`
417 // option and thus can be anything, and the incorrect match will be
418 // handled safely in `extract_one`.
419 for search_path in self.filesearch.search_paths(self.path_kind) {
420 debug!("searching {}", search_path.dir.display());
421 let spf = &search_path.files;
422
423 let mut should_check_staticlibs = true;
424 for (prefix, suffix, kind) in [
425 (rlib_prefix.as_str(), rlib_suffix, CrateFlavor::Rlib),
426 (rmeta_prefix.as_str(), rmeta_suffix, CrateFlavor::Rmeta),
427 (dylib_prefix, dylib_suffix, CrateFlavor::Dylib),
428 (interface_prefix, interface_suffix, CrateFlavor::SDylib),
429 ] {
430 if prefix == staticlib_prefix && suffix == staticlib_suffix {
431 should_check_staticlibs = false;
432 }
433 if let Some(matches) = spf.query(prefix, suffix) {
434 for (hash, spf) in matches {
435 info!("lib candidate: {}", spf.path.display());
436
437 let (rlibs, rmetas, dylibs, interfaces) =
438 candidates.entry(hash).or_default();
439 {
440 // As a perforamnce optimisation we canonicalize the path and skip
441 // ones we've already seeen. This allows us to ignore crates
442 // we know are exactual equal to ones we've already found.
443 // Going to the same crate through different symlinks does not change the result.
444 let path = try_canonicalize(&spf.path)
445 .unwrap_or_else(|_| spf.path.to_path_buf());
446 if seen_paths.contains(&path) {
447 continue;
448 };
449 seen_paths.insert(path);
450 }
451 // Use the original path (potentially with unresolved symlinks),
452 // filesystem code should not care, but this is nicer for diagnostics.
453 let path = spf.path.to_path_buf();
454 match kind {
455 CrateFlavor::Rlib => rlibs.insert(path, search_path.kind),
456 CrateFlavor::Rmeta => rmetas.insert(path, search_path.kind),
457 CrateFlavor::Dylib => dylibs.insert(path, search_path.kind),
458 CrateFlavor::SDylib => interfaces.insert(path, search_path.kind),
459 };
460 }
461 }
462 }
463 if let Some(static_matches) = should_check_staticlibs
464 .then(|| spf.query(staticlib_prefix, staticlib_suffix))
465 .flatten()
466 {
467 for (_, spf) in static_matches {
468 self.crate_rejections.via_kind.push(CrateMismatch {
469 path: spf.path.to_path_buf(),
470 got: "static".to_string(),
471 });
472 }
473 }
474 }
475
476 // We have now collected all known libraries into a set of candidates
477 // keyed of the filename hash listed. For each filename, we also have a
478 // list of rlibs/dylibs that apply. Here, we map each of these lists
479 // (per hash), to a Library candidate for returning.
480 //
481 // A Library candidate is created if the metadata for the set of
482 // libraries corresponds to the crate id and hash criteria that this
483 // search is being performed for.
484 let mut libraries = FxIndexMap::default();
485 for (_hash, (rlibs, rmetas, dylibs, interfaces)) in candidates {
486 if let Some((svh, lib)) = self.extract_lib(rlibs, rmetas, dylibs, interfaces)? {
487 libraries.insert(svh, lib);
488 }
489 }
490
491 // Having now translated all relevant found hashes into libraries, see
492 // what we've got and figure out if we found multiple candidates for
493 // libraries or not.
494 match libraries.len() {
495 0 => Ok(None),
496 1 => Ok(Some(libraries.into_iter().next().unwrap().1)),
497 _ => {
498 let mut libraries: Vec<_> = libraries.into_values().collect();
499
500 libraries.sort_by_cached_key(|lib| lib.source.paths().next().unwrap().clone());
501 let candidates = libraries
502 .iter()
503 .map(|lib| lib.source.paths().next().unwrap().clone())
504 .collect::<Vec<_>>();
505
506 Err(CrateError::MultipleCandidates(
507 self.crate_name,
508 // these are the same for all candidates
509 get_flavor_from_path(candidates.first().unwrap()),
510 candidates,
511 ))
512 }
513 }
514 }
515
516 fn extract_lib(
517 &mut self,
518 rlibs: FxIndexMap<PathBuf, PathKind>,
519 rmetas: FxIndexMap<PathBuf, PathKind>,
520 dylibs: FxIndexMap<PathBuf, PathKind>,
521 interfaces: FxIndexMap<PathBuf, PathKind>,
522 ) -> Result<Option<(Svh, Library)>, CrateError> {
523 let mut slot = None;
524 // Order here matters, rmeta should come first.
525 //
526 // Make sure there's at most one rlib and at most one dylib.
527 //
528 // See comment in `extract_one` below.
529 let rmeta = self.extract_one(rmetas, CrateFlavor::Rmeta, &mut slot)?;
530 let rlib = self.extract_one(rlibs, CrateFlavor::Rlib, &mut slot)?;
531 let sdylib_interface = self.extract_one(interfaces, CrateFlavor::SDylib, &mut slot)?;
532 let dylib = self.extract_one(dylibs, CrateFlavor::Dylib, &mut slot)?;
533
534 if sdylib_interface.is_some() && dylib.is_none() {
535 return Err(CrateError::FullMetadataNotFound(self.crate_name, CrateFlavor::SDylib));
536 }
537
538 let source = CrateSource { rmeta, rlib, dylib, sdylib_interface };
539 Ok(slot.map(|(svh, metadata, _, _)| (svh, Library { source, metadata })))
540 }
541
542 fn needs_crate_flavor(&self, flavor: CrateFlavor) -> bool {
543 if flavor == CrateFlavor::Dylib && self.is_proc_macro {
544 return true;
545 }
546
547 if self.only_needs_metadata {
548 flavor == CrateFlavor::Rmeta
549 } else {
550 // we need all flavors (perhaps not true, but what we do for now)
551 true
552 }
553 }
554
555 // Attempts to extract *one* library from the set `m`. If the set has no
556 // elements, `None` is returned. If the set has more than one element, then
557 // the errors and notes are emitted about the set of libraries.
558 //
559 // With only one library in the set, this function will extract it, and then
560 // read the metadata from it if `*slot` is `None`. If the metadata couldn't
561 // be read, it is assumed that the file isn't a valid rust library (no
562 // errors are emitted).
563 //
564 // The `PathBuf` in `slot` will only be used for diagnostic purposes.
565 fn extract_one(
566 &mut self,
567 m: FxIndexMap<PathBuf, PathKind>,
568 flavor: CrateFlavor,
569 slot: &mut Option<(Svh, MetadataBlob, PathBuf, CrateFlavor)>,
570 ) -> Result<Option<(PathBuf, PathKind)>, CrateError> {
571 // If we are producing an rlib, and we've already loaded metadata, then
572 // we should not attempt to discover further crate sources (unless we're
573 // locating a proc macro; exact logic is in needs_crate_flavor). This means
574 // that under -Zbinary-dep-depinfo we will not emit a dependency edge on
575 // the *unused* rlib, and by returning `None` here immediately we
576 // guarantee that we do indeed not use it.
577 //
578 // See also #68149 which provides more detail on why emitting the
579 // dependency on the rlib is a bad thing.
580 if slot.is_some() {
581 if m.is_empty() || !self.needs_crate_flavor(flavor) {
582 return Ok(None);
583 }
584 }
585
586 let mut ret: Option<(PathBuf, PathKind)> = None;
587 let mut err_data: Option<Vec<PathBuf>> = None;
588 for (lib, kind) in m {
589 info!("{} reading metadata from: {}", flavor, lib.display());
590 if flavor == CrateFlavor::Rmeta && lib.metadata().is_ok_and(|m| m.len() == 0) {
591 // Empty files will cause get_metadata_section to fail. Rmeta
592 // files can be empty, for example with binaries (which can
593 // often appear with `cargo check` when checking a library as
594 // a unittest). We don't want to emit a user-visible warning
595 // in this case as it is not a real problem.
596 debug!("skipping empty file");
597 continue;
598 }
599 let (hash, metadata) = match get_metadata_section(
600 self.target,
601 flavor,
602 &lib,
603 self.metadata_loader,
604 self.cfg_version,
605 Some(self.crate_name),
606 ) {
607 Ok(blob) => {
608 if let Some(h) = self.crate_matches(&blob, &lib) {
609 (h, blob)
610 } else {
611 info!("metadata mismatch");
612 continue;
613 }
614 }
615 Err(MetadataError::VersionMismatch { expected_version, found_version }) => {
616 // The file was present and created by the same compiler version, but we
617 // couldn't load it for some reason. Give a hard error instead of silently
618 // ignoring it, but only if we would have given an error anyway.
619 info!(
620 "Rejecting via version: expected {} got {}",
621 expected_version, found_version
622 );
623 self.crate_rejections
624 .via_version
625 .push(CrateMismatch { path: lib, got: found_version });
626 continue;
627 }
628 Err(MetadataError::LoadFailure(err)) => {
629 info!("no metadata found: {}", err);
630 // Metadata was loaded from interface file earlier.
631 if let Some((.., CrateFlavor::SDylib)) = slot {
632 ret = Some((lib, kind));
633 continue;
634 }
635 // The file was present and created by the same compiler version, but we
636 // couldn't load it for some reason. Give a hard error instead of silently
637 // ignoring it, but only if we would have given an error anyway.
638 self.crate_rejections.via_invalid.push(CrateMismatch { path: lib, got: err });
639 continue;
640 }
641 Err(err @ MetadataError::NotPresent(_)) => {
642 info!("no metadata found: {}", err);
643 continue;
644 }
645 };
646 // If we see multiple hashes, emit an error about duplicate candidates.
647 if slot.as_ref().is_some_and(|s| s.0 != hash) {
648 if let Some(candidates) = err_data {
649 return Err(CrateError::MultipleCandidates(
650 self.crate_name,
651 flavor,
652 candidates,
653 ));
654 }
655 err_data = Some(vec![slot.take().unwrap().2]);
656 }
657 if let Some(candidates) = &mut err_data {
658 candidates.push(lib);
659 continue;
660 }
661
662 // Ok so at this point we've determined that `(lib, kind)` above is
663 // a candidate crate to load, and that `slot` is either none (this
664 // is the first crate of its kind) or if some the previous path has
665 // the exact same hash (e.g., it's the exact same crate).
666 //
667 // In principle these two candidate crates are exactly the same so
668 // we can choose either of them to link. As a stupidly gross hack,
669 // however, we favor crate in the sysroot.
670 //
671 // You can find more info in rust-lang/rust#39518 and various linked
672 // issues, but the general gist is that during testing libstd the
673 // compilers has two candidates to choose from: one in the sysroot
674 // and one in the deps folder. These two crates are the exact same
675 // crate but if the compiler chooses the one in the deps folder
676 // it'll cause spurious errors on Windows.
677 //
678 // As a result, we favor the sysroot crate here. Note that the
679 // candidates are all canonicalized, so we canonicalize the sysroot
680 // as well.
681 if let Some((prev, _)) = &ret {
682 let sysroot = self.sysroot;
683 let sysroot = try_canonicalize(sysroot).unwrap_or_else(|_| sysroot.to_path_buf());
684 if prev.starts_with(&sysroot) {
685 continue;
686 }
687 }
688
689 // We error eagerly here. If we're locating a rlib, then in theory the full metadata
690 // could still be in a (later resolved) dylib. In practice, if the rlib and dylib
691 // were produced in a way where one has full metadata and the other hasn't, it would
692 // mean that they were compiled using different compiler flags and probably also have
693 // a different SVH value.
694 if metadata.get_header().is_stub {
695 // `is_stub` should never be true for .rmeta files.
696 assert_ne!(flavor, CrateFlavor::Rmeta);
697
698 // Because rmeta files are resolved before rlib/dylib files, if this is a stub and
699 // we haven't found a slot already, it means that the full metadata is missing.
700 if slot.is_none() {
701 return Err(CrateError::FullMetadataNotFound(self.crate_name, flavor));
702 }
703 } else {
704 *slot = Some((hash, metadata, lib.clone(), flavor));
705 }
706 ret = Some((lib, kind));
707 }
708
709 if let Some(candidates) = err_data {
710 Err(CrateError::MultipleCandidates(self.crate_name, flavor, candidates))
711 } else {
712 Ok(ret)
713 }
714 }
715
716 fn crate_matches(&mut self, metadata: &MetadataBlob, libpath: &Path) -> Option<Svh> {
717 let header = metadata.get_header();
718 if header.is_proc_macro_crate != self.is_proc_macro {
719 info!(
720 "Rejecting via proc macro: expected {} got {}",
721 self.is_proc_macro, header.is_proc_macro_crate,
722 );
723 return None;
724 }
725
726 if self.exact_paths.is_empty() && self.crate_name != header.name {
727 info!("Rejecting via crate name");
728 return None;
729 }
730
731 if header.triple != self.tuple {
732 info!("Rejecting via crate triple: expected {} got {}", self.tuple, header.triple);
733 self.crate_rejections.via_triple.push(CrateMismatch {
734 path: libpath.to_path_buf(),
735 got: header.triple.to_string(),
736 });
737 return None;
738 }
739
740 let hash = header.hash;
741 if let Some(expected_hash) = self.hash {
742 if hash != expected_hash {
743 info!("Rejecting via hash: expected {} got {}", expected_hash, hash);
744 self.crate_rejections
745 .via_hash
746 .push(CrateMismatch { path: libpath.to_path_buf(), got: hash.to_string() });
747 return None;
748 }
749 }
750
751 Some(hash)
752 }
753
754 fn find_commandline_library(&mut self) -> Result<Option<Library>, CrateError> {
755 // First, filter out all libraries that look suspicious. We only accept
756 // files which actually exist that have the correct naming scheme for
757 // rlibs/dylibs.
758 let mut rlibs = FxIndexMap::default();
759 let mut rmetas = FxIndexMap::default();
760 let mut dylibs = FxIndexMap::default();
761 let mut sdylib_interfaces = FxIndexMap::default();
762 for loc in &self.exact_paths {
763 let loc_canon = loc.canonicalized();
764 let loc_orig = loc.original();
765 if !loc_canon.exists() {
766 return Err(CrateError::ExternLocationNotExist(self.crate_name, loc_orig.clone()));
767 }
768 if !loc_orig.is_file() {
769 return Err(CrateError::ExternLocationNotFile(self.crate_name, loc_orig.clone()));
770 }
771 // Note to take care and match against the non-canonicalized name:
772 // some systems save build artifacts into content-addressed stores
773 // that do not preserve extensions, and then link to them using
774 // e.g. symbolic links. If we canonicalize too early, we resolve
775 // the symlink, the file type is lost and we might treat rlibs and
776 // rmetas as dylibs.
777 let Some(file) = loc_orig.file_name().and_then(|s| s.to_str()) else {
778 return Err(CrateError::ExternLocationNotFile(self.crate_name, loc_orig.clone()));
779 };
780 if file.starts_with("lib") {
781 if file.ends_with(".rlib") {
782 rlibs.insert(loc_canon.clone(), PathKind::ExternFlag);
783 continue;
784 }
785 if file.ends_with(".rmeta") {
786 rmetas.insert(loc_canon.clone(), PathKind::ExternFlag);
787 continue;
788 }
789 if file.ends_with(".rs") {
790 sdylib_interfaces.insert(loc_canon.clone(), PathKind::ExternFlag);
791 }
792 }
793 let dll_prefix = self.target.dll_prefix.as_ref();
794 let dll_suffix = self.target.dll_suffix.as_ref();
795 if file.starts_with(dll_prefix) && file.ends_with(dll_suffix) {
796 dylibs.insert(loc_canon.clone(), PathKind::ExternFlag);
797 continue;
798 }
799 self.crate_rejections
800 .via_filename
801 .push(CrateMismatch { path: loc_orig.clone(), got: String::new() });
802 }
803
804 // Extract the dylib/rlib/rmeta triple.
805 self.extract_lib(rlibs, rmetas, dylibs, sdylib_interfaces)
806 .map(|opt| opt.map(|(_, lib)| lib))
807 }
808
809 pub(crate) fn into_error(self, dep_root: Option<CratePaths>) -> CrateError {
810 CrateError::LocatorCombined(Box::new(CombinedLocatorError {
811 crate_name: self.crate_name,
812 dep_root,
813 triple: self.tuple,
814 dll_prefix: self.target.dll_prefix.to_string(),
815 dll_suffix: self.target.dll_suffix.to_string(),
816 crate_rejections: self.crate_rejections,
817 }))
818 }
819}
820
821fn get_metadata_section<'p>(
822 target: &Target,
823 flavor: CrateFlavor,
824 filename: &'p Path,
825 loader: &dyn MetadataLoader,
826 cfg_version: &'static str,
827 crate_name: Option<Symbol>,
828) -> Result<MetadataBlob, MetadataError<'p>> {
829 if !filename.exists() {
830 return Err(MetadataError::NotPresent(filename));
831 }
832 let raw_bytes = match flavor {
833 CrateFlavor::Rlib => {
834 loader.get_rlib_metadata(target, filename).map_err(MetadataError::LoadFailure)?
835 }
836 CrateFlavor::SDylib => {
837 let compiler = std::env::current_exe().map_err(|_err| {
838 MetadataError::LoadFailure(
839 "couldn't obtain current compiler binary when loading sdylib interface"
840 .to_string(),
841 )
842 })?;
843
844 let tmp_path = match TempFileBuilder::new().prefix("rustc").tempdir() {
845 Ok(tmp_path) => tmp_path,
846 Err(error) => {
847 return Err(MetadataError::LoadFailure(format!(
848 "couldn't create a temp dir: {}",
849 error
850 )));
851 }
852 };
853
854 let crate_name = crate_name.unwrap();
855 debug!("compiling {}", filename.display());
856 // FIXME: This will need to be done either within the current compiler session or
857 // as a separate compiler session in the same process.
858 let res = std::process::Command::new(compiler)
859 .arg(&filename)
860 .arg("--emit=metadata")
861 .arg(format!("--crate-name={}", crate_name))
862 .arg(format!("--out-dir={}", tmp_path.path().display()))
863 .arg("-Zbuild-sdylib-interface")
864 .output()
865 .map_err(|err| {
866 MetadataError::LoadFailure(format!("couldn't compile interface: {}", err))
867 })?;
868
869 if !res.status.success() {
870 return Err(MetadataError::LoadFailure(format!(
871 "couldn't compile interface: {}",
872 std::str::from_utf8(&res.stderr).unwrap_or_default()
873 )));
874 }
875
876 // Load interface metadata instead of crate metadata.
877 let interface_metadata_name = format!("lib{}.rmeta", crate_name);
878 let rmeta_file = tmp_path.path().join(interface_metadata_name);
879 debug!("loading interface metadata from {}", rmeta_file.display());
880 let rmeta = get_rmeta_metadata_section(&rmeta_file)?;
881 let _ = std::fs::remove_file(rmeta_file);
882
883 rmeta
884 }
885 CrateFlavor::Dylib => {
886 let buf =
887 loader.get_dylib_metadata(target, filename).map_err(MetadataError::LoadFailure)?;
888 let header_len = METADATA_HEADER.len();
889 // header + u64 length of data
890 let data_start = header_len + 8;
891
892 debug!("checking {} bytes of metadata-version stamp", header_len);
893 let header = &buf[..cmp::min(header_len, buf.len())];
894 if header != METADATA_HEADER {
895 return Err(MetadataError::LoadFailure(format!(
896 "invalid metadata version found: {}",
897 filename.display()
898 )));
899 }
900
901 // Length of the metadata - this allows linkers to pad the section if they want
902 let Ok(len_bytes) =
903 <[u8; 8]>::try_from(&buf[header_len..cmp::min(data_start, buf.len())])
904 else {
905 return Err(MetadataError::LoadFailure(
906 "invalid metadata length found".to_string(),
907 ));
908 };
909 let metadata_len = u64::from_le_bytes(len_bytes) as usize;
910
911 // Header is okay -> inflate the actual metadata
912 buf.slice(|buf| &buf[data_start..(data_start + metadata_len)])
913 }
914 CrateFlavor::Rmeta => get_rmeta_metadata_section(filename)?,
915 };
916 let Ok(blob) = MetadataBlob::new(raw_bytes) else {
917 return Err(MetadataError::LoadFailure(format!(
918 "corrupt metadata encountered in {}",
919 filename.display()
920 )));
921 };
922 match blob.check_compatibility(cfg_version) {
923 Ok(()) => {
924 debug!("metadata blob read okay");
925 Ok(blob)
926 }
927 Err(None) => Err(MetadataError::LoadFailure(format!(
928 "invalid metadata version found: {}",
929 filename.display()
930 ))),
931 Err(Some(found_version)) => {
932 return Err(MetadataError::VersionMismatch {
933 expected_version: rustc_version(cfg_version),
934 found_version,
935 });
936 }
937 }
938}
939
940fn get_rmeta_metadata_section<'a, 'p>(filename: &'p Path) -> Result<OwnedSlice, MetadataError<'a>> {
941 // mmap the file, because only a small fraction of it is read.
942 let file = std::fs::File::open(filename).map_err(|_| {
943 MetadataError::LoadFailure(format!(
944 "failed to open rmeta metadata: '{}'",
945 filename.display()
946 ))
947 })?;
948 let mmap = unsafe { Mmap::map(file) };
949 let mmap = mmap.map_err(|_| {
950 MetadataError::LoadFailure(format!(
951 "failed to mmap rmeta metadata: '{}'",
952 filename.display()
953 ))
954 })?;
955
956 Ok(slice_owned(mmap, Deref::deref))
957}
958
959/// A diagnostic function for dumping crate metadata to an output stream.
960pub fn list_file_metadata(
961 target: &Target,
962 path: &Path,
963 metadata_loader: &dyn MetadataLoader,
964 out: &mut dyn Write,
965 ls_kinds: &[String],
966 cfg_version: &'static str,
967) -> IoResult<()> {
968 let flavor = get_flavor_from_path(path);
969 match get_metadata_section(target, flavor, path, metadata_loader, cfg_version, None) {
970 Ok(metadata) => metadata.list_crate_metadata(out, ls_kinds),
971 Err(msg) => write!(out, "{msg}\n"),
972 }
973}
974
975fn get_flavor_from_path(path: &Path) -> CrateFlavor {
976 let filename = path.file_name().unwrap().to_str().unwrap();
977
978 if filename.ends_with(".rlib") {
979 CrateFlavor::Rlib
980 } else if filename.ends_with(".rmeta") {
981 CrateFlavor::Rmeta
982 } else {
983 CrateFlavor::Dylib
984 }
985}
986
987// ------------------------------------------ Error reporting -------------------------------------
988
989#[derive(Clone, Debug)]
990struct CrateMismatch {
991 path: PathBuf,
992 got: String,
993}
994
995#[derive(Clone, Debug, Default)]
996struct CrateRejections {
997 via_hash: Vec<CrateMismatch>,
998 via_triple: Vec<CrateMismatch>,
999 via_kind: Vec<CrateMismatch>,
1000 via_version: Vec<CrateMismatch>,
1001 via_filename: Vec<CrateMismatch>,
1002 via_invalid: Vec<CrateMismatch>,
1003}
1004
1005/// Candidate rejection reasons collected during crate search.
1006/// If no candidate is accepted, then these reasons are presented to the user,
1007/// otherwise they are ignored.
1008#[derive(Debug)]
1009pub(crate) struct CombinedLocatorError {
1010 crate_name: Symbol,
1011 dep_root: Option<CratePaths>,
1012 triple: TargetTuple,
1013 dll_prefix: String,
1014 dll_suffix: String,
1015 crate_rejections: CrateRejections,
1016}
1017
1018#[derive(Debug)]
1019pub(crate) enum CrateError {
1020 NonAsciiName(Symbol),
1021 ExternLocationNotExist(Symbol, PathBuf),
1022 ExternLocationNotFile(Symbol, PathBuf),
1023 MultipleCandidates(Symbol, CrateFlavor, Vec<PathBuf>),
1024 FullMetadataNotFound(Symbol, CrateFlavor),
1025 SymbolConflictsCurrent(Symbol),
1026 StableCrateIdCollision(Symbol, Symbol),
1027 DlOpen(String, String),
1028 DlSym(String, String),
1029 LocatorCombined(Box<CombinedLocatorError>),
1030 NotFound(Symbol),
1031}
1032
1033enum MetadataError<'a> {
1034 /// The file was missing.
1035 NotPresent(&'a Path),
1036 /// The file was present and invalid.
1037 LoadFailure(String),
1038 /// The file was present, but compiled with a different rustc version.
1039 VersionMismatch { expected_version: String, found_version: String },
1040}
1041
1042impl fmt::Display for MetadataError<'_> {
1043 fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
1044 match self {
1045 MetadataError::NotPresent(filename) => {
1046 f.write_str(&format!("no such file: '{}'", filename.display()))
1047 }
1048 MetadataError::LoadFailure(msg) => f.write_str(msg),
1049 MetadataError::VersionMismatch { expected_version, found_version } => {
1050 f.write_str(&format!(
1051 "rustc version mismatch. expected {}, found {}",
1052 expected_version, found_version,
1053 ))
1054 }
1055 }
1056 }
1057}
1058
1059impl CrateError {
1060 pub(crate) fn report(self, sess: &Session, span: Span, missing_core: bool) {
1061 let dcx = sess.dcx();
1062 match self {
1063 CrateError::NonAsciiName(crate_name) => {
1064 dcx.emit_err(errors::NonAsciiName { span, crate_name });
1065 }
1066 CrateError::ExternLocationNotExist(crate_name, loc) => {
1067 dcx.emit_err(errors::ExternLocationNotExist { span, crate_name, location: &loc });
1068 }
1069 CrateError::ExternLocationNotFile(crate_name, loc) => {
1070 dcx.emit_err(errors::ExternLocationNotFile { span, crate_name, location: &loc });
1071 }
1072 CrateError::MultipleCandidates(crate_name, flavor, candidates) => {
1073 dcx.emit_err(errors::MultipleCandidates { span, crate_name, flavor, candidates });
1074 }
1075 CrateError::FullMetadataNotFound(crate_name, flavor) => {
1076 dcx.emit_err(errors::FullMetadataNotFound { span, crate_name, flavor });
1077 }
1078 CrateError::SymbolConflictsCurrent(root_name) => {
1079 dcx.emit_err(errors::SymbolConflictsCurrent { span, crate_name: root_name });
1080 }
1081 CrateError::StableCrateIdCollision(crate_name0, crate_name1) => {
1082 dcx.emit_err(errors::StableCrateIdCollision { span, crate_name0, crate_name1 });
1083 }
1084 CrateError::DlOpen(path, err) | CrateError::DlSym(path, err) => {
1085 dcx.emit_err(errors::DlError { span, path, err });
1086 }
1087 CrateError::LocatorCombined(locator) => {
1088 let crate_name = locator.crate_name;
1089 let add_info = match &locator.dep_root {
1090 None => String::new(),
1091 Some(r) => format!(" which `{}` depends on", r.name),
1092 };
1093 if !locator.crate_rejections.via_filename.is_empty() {
1094 let mismatches = locator.crate_rejections.via_filename.iter();
1095 for CrateMismatch { path, .. } in mismatches {
1096 dcx.emit_err(errors::CrateLocationUnknownType { span, path, crate_name });
1097 dcx.emit_err(errors::LibFilenameForm {
1098 span,
1099 dll_prefix: &locator.dll_prefix,
1100 dll_suffix: &locator.dll_suffix,
1101 });
1102 }
1103 }
1104 let mut found_crates = String::new();
1105 if !locator.crate_rejections.via_hash.is_empty() {
1106 let mismatches = locator.crate_rejections.via_hash.iter();
1107 for CrateMismatch { path, .. } in mismatches {
1108 found_crates.push_str(&format!(
1109 "\ncrate `{}`: {}",
1110 crate_name,
1111 path.display()
1112 ));
1113 }
1114 if let Some(r) = locator.dep_root {
1115 for path in r.source.paths() {
1116 found_crates.push_str(&format!(
1117 "\ncrate `{}`: {}",
1118 r.name,
1119 path.display()
1120 ));
1121 }
1122 }
1123 dcx.emit_err(errors::NewerCrateVersion {
1124 span,
1125 crate_name,
1126 add_info,
1127 found_crates,
1128 });
1129 } else if !locator.crate_rejections.via_triple.is_empty() {
1130 let mismatches = locator.crate_rejections.via_triple.iter();
1131 for CrateMismatch { path, got } in mismatches {
1132 found_crates.push_str(&format!(
1133 "\ncrate `{}`, target triple {}: {}",
1134 crate_name,
1135 got,
1136 path.display(),
1137 ));
1138 }
1139 dcx.emit_err(errors::NoCrateWithTriple {
1140 span,
1141 crate_name,
1142 locator_triple: locator.triple.tuple(),
1143 add_info,
1144 found_crates,
1145 });
1146 } else if !locator.crate_rejections.via_kind.is_empty() {
1147 let mismatches = locator.crate_rejections.via_kind.iter();
1148 for CrateMismatch { path, .. } in mismatches {
1149 found_crates.push_str(&format!(
1150 "\ncrate `{}`: {}",
1151 crate_name,
1152 path.display()
1153 ));
1154 }
1155 dcx.emit_err(errors::FoundStaticlib {
1156 span,
1157 crate_name,
1158 add_info,
1159 found_crates,
1160 });
1161 } else if !locator.crate_rejections.via_version.is_empty() {
1162 let mismatches = locator.crate_rejections.via_version.iter();
1163 for CrateMismatch { path, got } in mismatches {
1164 found_crates.push_str(&format!(
1165 "\ncrate `{}` compiled by {}: {}",
1166 crate_name,
1167 got,
1168 path.display(),
1169 ));
1170 }
1171 dcx.emit_err(errors::IncompatibleRustc {
1172 span,
1173 crate_name,
1174 add_info,
1175 found_crates,
1176 rustc_version: rustc_version(sess.cfg_version),
1177 });
1178 } else if !locator.crate_rejections.via_invalid.is_empty() {
1179 let mut crate_rejections = Vec::new();
1180 for CrateMismatch { path: _, got } in locator.crate_rejections.via_invalid {
1181 crate_rejections.push(got);
1182 }
1183 dcx.emit_err(errors::InvalidMetadataFiles {
1184 span,
1185 crate_name,
1186 add_info,
1187 crate_rejections,
1188 });
1189 } else {
1190 let error = errors::CannotFindCrate {
1191 span,
1192 crate_name,
1193 add_info,
1194 missing_core,
1195 current_crate: sess
1196 .opts
1197 .crate_name
1198 .clone()
1199 .unwrap_or("<unknown>".to_string()),
1200 is_nightly_build: sess.is_nightly_build(),
1201 profiler_runtime: Symbol::intern(&sess.opts.unstable_opts.profiler_runtime),
1202 locator_triple: locator.triple,
1203 is_ui_testing: sess.opts.unstable_opts.ui_testing,
1204 };
1205 // The diagnostic for missing core is very good, but it is followed by a lot of
1206 // other diagnostics that do not add information.
1207 if missing_core {
1208 dcx.emit_fatal(error);
1209 } else {
1210 dcx.emit_err(error);
1211 }
1212 }
1213 }
1214 CrateError::NotFound(crate_name) => {
1215 let error = errors::CannotFindCrate {
1216 span,
1217 crate_name,
1218 add_info: String::new(),
1219 missing_core,
1220 current_crate: sess.opts.crate_name.clone().unwrap_or("<unknown>".to_string()),
1221 is_nightly_build: sess.is_nightly_build(),
1222 profiler_runtime: Symbol::intern(&sess.opts.unstable_opts.profiler_runtime),
1223 locator_triple: sess.opts.target_triple.clone(),
1224 is_ui_testing: sess.opts.unstable_opts.ui_testing,
1225 };
1226 // The diagnostic for missing core is very good, but it is followed by a lot of
1227 // other diagnostics that do not add information.
1228 if missing_core {
1229 dcx.emit_fatal(error);
1230 } else {
1231 dcx.emit_err(error);
1232 }
1233 }
1234 }
1235 }
1236}