Dollar Origins

At work, we usually have GCC and binutils installed to the same root. I’ve recently been working on a project that needs a more recent GCC so I’ve been building these fairly often. Since these days GDB gets bundled with binutils, I thought I’d benefit from a nice new shiny GDB.

Some things that are always problematic when building and installing to non-standard locations are the shared libraries. Many of the system libraries that get used will be the ones that ship with the OS. The C standard library, libc, is one such example. However, if you build with a C++ compiler other than the system one, the odds are that you will need to link with the C++ standard library that was built with that compiler. Linking is only half of the problem. Finding the library at runtime is the other. When you build GCC, it will tell you all about that at the end of the build. If the library is in a well-known location like /usr/lib64 then all is well. If you have multiple different versions of, say, libstdc++.so, then things are a bit more complicated.

The bad way to find libraries is to use LD_LIBRARY_PATH. The problem with this is that you can’t use it to choose different library versions. It’s a colon-listed set of directories, and the first directory that contains the library being sought gets used. That soon deteriorates to the point where every application needs a wrapper script to set its own LD_LIBRARY_PATH. There is an alternative. RPATH. RPATH is, in effect, LD_LIBRARY_PATH compiled into an exe. For more reasons to avoid LD_LIBRARY_PATH, see George Southoff’s blog [Southoff16].

Recently, I decided to rename one of my directories, since I’d been doing some work with GCCs 11 and 13. And that broke my GDB. The problem was that I’d used an absolute RPATH to build it. When I renamed the directory, the absolute path no longer matched and the link loader could no longer find a suitable libstdc++. That gave me lots of errors like

/path/to/gcc-13.1.0/bin/gdb: /lib64/libstdc++.so.6:
version 'GLIBCXX_3.4.20' not found (required by 
/path/to/gcc-13.1.0/bin/gdb)

There is a better way to set your RPATH. You can use $ORIGIN. $ORIGIN is a way of specifying relative paths. It is not an environment variable – it gets baked into the executable. The link loader will replace $ORIGIN with the directory containing the exe. So, for my installation of GDB, I just need to give it an RPATH of $ORIGIN/../lib64. Sounds easy? Wrong! Whilst the link loader doesn’t look for $ORIGIN in the environment, the shell thinks that it is a shell variable and make thinks that it is a make variable.

binutils/GDB uses autoconf and a configure script. I wrote a shell script to run configure for the project.

Starting with something naive:

  ../configure {various arguments} LDFLAGS="-Wl,
  -rpath,$ORIGIN/../lib64"

The -Wl,-rpath, bit is the parameter to tell g++ acting as the linker driver to pass -rpath to the link editor.

In the generated Makefile I get

  LDFLAGS = Wl,-rpath,/../lib64

That’s no good. My shell script has interpreted $ORIGIN as an environment variable and replaced it. OK, so I’ll escape the dollar in my script, making it \$ORIGIN. Now the Makefile contains

  LDFLAGS = Wl,-rpath,$ORIGIN/../lib64

That looks better, so I build GDB and ... same error. I can check what rpath (if any) has been built into gdb as follows:

  readelf -d gdb | grep rpath

that gives me

   0x000000000000000f (RPATH)
   Library rpath: [RIGIN/../lib64]

OK, I got something. The next problem is that binutils/GDB uses a hierarchical configuration. Running configure in my build directory just generates a top level Makefile. Running make reruns configure for each subdirectory. The gdb subdirectory Makefile contains

  LDFLAGS =  -Wl,-rpath,RIGIN/../lib64

That means that the recursive make has played the same trick on me, though this time it has interpreted $O as a make variable.

I could try to ‘escape the escape’ with \\$ORIGIN in my script. That won’t work as the first escape only protects the second escape, and the shell will still replace the environment variable. I could try a triple escape. That gives me -Wl,-rpath,\RIGIN/../lib64. The problem is that \ isn’t the escape character for Makefiles. In order to escape a $, you need a second $.

So, let’s try -Wl,-rpath,\$\$ORIGIN/../lib64 in my script. That gives me LDFLAGS = $$ORIGIN/../lib64 in the outer Makefile but just LDFLAGS = -Wl,-rpath,/../lib64 in the inner Makefile. That looks like a shell replacement when the outer make runs configure for the inner gdb directory. Those dollars need protecting in the outer Makefile. I didn’t think that it was the right thing, but I tried \$\$\$\$ORIGIN in my script. That gave me -Wl,-rpath,60598ORIGIN/../lib64 in the inner Makefile. Definitely shell replacement where $$ gets replaced by the PID. I think that’s enough trial and error. Let’s try to reason about it.

• To protect a shell dollar it needs to be preceded by a \.
• To protect a shell backslash it needs to be preceded by a \.
• To protect a make dollar it needs to be preceded by $.

I then spent a while looking at the flow from my script that runs configure to the final gdb binary. I wanted to understand that flow in terms of successive executions of shell and make, so that I could understand what replacements get done and what escaping is needed.

So there is:

1. The starting shell
2. Outer make
3. Outer config.status shell
4. Outer recursive make
5. Inner config.status shell
6. Inner make
7. Linker shell
8. Final gdb rpath

Along the way, there’s some awk self modification of the Makefiles, but thankfully that doesn’t need any escaping.

Working backwards and applying the 3 protection rules described above that means that the strings need to be

1. $ORIGIN
2. \$ORIGIN
3. \$$ORIGIN
4. \\\$\$ORIGIN
5. \\\$$\$$ORIGIN
6. \\\\\\\$\$\\\$\$ORIGIN
7. \\\\\\\$$\$$\\\$$\$$ORIGIN
8. \\\\\\\\\\\\\\\$\$\\$\$\\\\\\\$\$\\$\$ORIGIN

I’m glad that I didn’t persist with the trial-and-error approach to finding that. So the big question, does it work? Yes! As long as I keep the gdb binary at the same position relative to ../lib64, I can move the lower directories around to my heart’s content. One disadvantage is that this approach may not allow in-place execution of the binary.

Let’s hope that binutils/GDB never adds a third level of recursion. The backslashes grow by 2x+1 for every extra level of shell, so two more shells would mean that leading group would need 63 backslashes. That really would be a ‘fistful of backslashes’.

Reference

[Southoff16] George Southoff ‘LD_LIBRARY_PATH considered harmful’, posted on 22 Jul 2016 and accessed on 21 November 2023 at https://gms.tf/ld_library_path-considered-harmful.html

Paul Floyd has been writing software, mostly in C++ and C, for about 30 years. He lives near Grenoble, on the edge of the French Alps and works for Siemens EDA developing tools for analogue electronic circuit simulation. In his spare time, he maintains Valgrind.