Programatic sourcing of dependencies

Hi!

I am having a little trouble with a particular bash project. Typically each script sources its dependencies by looping though the files:

function source_dependencies() {
  for file in ../../utils/*; do
    source "${file}"
  done
}

In any script which does this, I cannot go to the defention of any function contained within these files.

For example if functionX is in one of those scripts and I have a script like:

#!/bin/bash

function source_dependencies() {
  for file in ../../utils/*; do
    source "${file}"
  done
}

source_dependencies
functionX

I cannot click through to the definition.

Would this be a valid feature request?

Hi @mashawan,
you’re right that this kind of project is still a bit hard to use with the current version.

Feature proposal

I’ve been thinking about this for a while now. Larger shell projects, e.g. bashdb or the letsencrypt tools, are also using this approach. I’m planning to improve this for 2.0, but haven’t decided yet about the best approach.
Possible solutions are:

  • A directive like #bashsupport source=../utils/*.bash in a script. You’d have to put this into every script, which uses declarations of utils/*.bash. This would pollute a lot of files, esp. if sourced files use the utils. Using # shellcheck source= isn’t possible, because it’s not supporting wildcards.
  • something like .bashsupportrc in the filetree. Such a file would define options for the directory and its subtree. If could be stored at the top-level of your project and could look like this:
    source: ./utils/*.bash
    
    or
    source-dir: ./utils
    

So far, I’m leaning towards the file-based solution. Are you aware of any other, common solution which could/should be used?

Workaround

Here’s a workaround for the current version, which might be possible for your project.
You could use multiple shellcheck source directive in a script. If put right after the shebang they’re valid for all of the file. This might be a bit hard to use for large scripts or lots of utility files.

#!/bin/bash
# shellcheck source=../utils/math.bash
# shellcheck source=../utils/message.bash

function source_dependencies() {
  for file in …/…/utils/*; do
    source “${file}”
  done
}

source_dependencies
functionX

Thanks for your reply. I am not really a bash expert so IU’m leaning on this plugin a lot to help me. I would vote for the file based solution too. But for now the shell check workaround gives me jump to declaration. Not sure if I can check these in as we are not all using IntelliJ + Bash Pro, but it gets me moving at least. Thanks!

Hi Joachim
Have you had any further thoughts on this topic?

I have several projects that use Bash extensively and have the situation where a script in project B may source a script from project A. (In my case, project A is library of standard logging and error handling functions that is used across a number of Bash projects.) In this situation, using “shellcheck source=” directives work but is not a pretty solution:

  • if I use absolute paths, they will contain my user-specific home directory so the directive breaks for any other user
  • if I use relative paths, the number of “…/” is very large in order to navigate back up the source-tree before heading down the tree of the other project

Cheers
Bruce

Hi @bus-error,
I do have made progress with this, but I’m not happy with the solution yet. I’m a bit slow with this, but didn’t want to publish a solution for wouldn’t be usuable for everyone.

Are project A and project B both in the same IntelliJ project? Or is one outside of the project files?

I’m leaning to avoid the filesystem-based configuration mentioned in an earlier post. That would clutter the filesystem, has issues with multiple files and overrides in the directory hierarchy and also easily introduces performance issues.

What I’m planning is a UI-based configuration for a project, i.e. it’s part of the project settings and not of the project filesystem itself. I attached a screenshot of the current UI. Things may still change a lot, though.

This is a library, which consists of sources and a set of files and directories, where these sources are made available (similar to a source library/*).
shellcheck source= is still available as before, of course.

Do you think that something like this would be useful for your use-case?

Thanks!
Joachim

Thank you for sharing your current prototype!

In my case, Project A and Project B are completely separate JetBrains projects. (They are also separate Git projects, if that relevant. Each project builds a deb package and project_b.deb has project_a.deb as a dependency.)

Looking at your design, yes, having the project store a “path” for where to locate sourced files could work. However, in my organisation we do not store the JetBrains project settings (i.e. the .idea directory) in code control with the code so that each developer who works on the project can have their own settings. The downside with this is that each developer would have to configure the path in their project settings themselves. This seems perfectly reasonable to me when the sourced file is outside of the current project but maybe a bit of burden when the sourced file and the ‘main’ file are part of the same JetBrains project. Maybe something more automatic could be done in this case? Maybe the current JetBrains project should always included in the path?

Best regards

Bruce

Thanks for the details on your setup. It’s not related to git repositories but more about the IntelliJ setup. IntelliJ only indexes files which belong to a project. It should be possible to extend this to files outside, i.e. for libraries, but I’ll have to check that.

Regarding the project settings: I don’t think that automatically adding all scripts of a project will be good. At least in my experience many scripts are self-contained or limited to a few calls to source. The shellcheck source= comments are great for that. Now, adding all the files of a project, would create a bunch of issues: possibly unresolved variables and functions at runtime when the file isn’t actually including all the files, too many unrelevant completions, unclear (re)definitions of variables, etc.

I assume that you and the other developers use some kind of build system when you’re working with other languages and that IntelliJ is syncing the build definition to the project setup.
For now, I’d like to avoid adding something like a propietary library setup file.

It might be possible to add a smart inspection, which analyzes a loop over calls to source and suggest sa suitable library setup. But I’ll have to think a bit about this :slight_smile:

At least for the initial version I expect that it’s not perfect for all, but with more feedback we’ll hopefully get to a point where it’s good for most.

Joachim

@bus-error Finally there’s some progress. I just published 2.0.0.beta8 which contains an initial implementation of this feature.

I’d be glad if you could test this with your setup and requirements.

Details:

  • setup is done via UI in the project settings, as shown in an earlier post
  • the settings define a list of “shell script libraries”. Each library defines a set of library sources (i.e. the first table in the UI) and a list of directories and files, where the library is used (i.e. the second table in the UI)
  • External library sources are supported, i.e. the library sources don’t have to be located inside the project or a module root
  • code completion, go to definition, etc. should all be working as with regular scripts
  • the order of the library sources is important. Entries at the top of the list are checked first. Re-definitions inside of library sources are not supported. That means if there are multiple definitions of the same variable or function in different library source files, then the file defined earlier in the list is the target for “go to definition”, etc.

Limitations:

  • ShellCheck doesn’t know about this configuration, of course. I’ll investigate if it’s somehow possible to pass it the additional files to look at. In the long run BashSupport Pro’s own inspections for “unresolved variable” and “unused variable” are going to replace ShellChecks similar checks. The inspections are still marked as beta and not yet mature enough, though
  • There may be edge-cases which I haven’t found yet, let me know if you notice anything odd or unexpected
  • The UI will be improved with the next updates.
  • This is supposed to also work on Windows, but I haven’t tested this yet…

Example: acme.sh

This example explains how to use this with the well-known acme.sh project

In this project all important definitions are in a single file acme.sh. Many files rely on these definitions.

  1. Clone and open the repository of https://github.com/acmesh-official/acme.sh
  2. Create a new shell script library “acme.sh” at Settings > Languages & Frameworks > BashSupport Pro > Shell Script Libraries:
    • in table “Library sources”, add a new file entry acme.sh
    • in table “Library usage”, add new entries for directories “deploy”, “dnsapi”, and “notify”
  3. Apply and close the settings
  4. Now all scripts under “deploy”, “dnsapi”, and “notify” show completions for the definitions found in the acme.sh file.

@mashawan Just in case that this is still relevant.
This is possible now with the latest beta of 2.0. This thread has setup instructions.

You’d have to setup a library with path/to/utils as “Library source” and as “Library usage” the directory containing the files with the “source_dependencies” calls.

Yeah, so I set up utils as described, then set the Library usage as the directory with the scripts that dynamically source these utils. Seemed to work. There were a few issues applying the changes, had to quit the dialog and try again a couple of times. But once set, it worked, so this approach looks good for my purposes.

Thanks!