Hey, I’m trying to create a pipeline that takes files from a flat directory source/ that contains files with names formatted as yyyy-mm-dd-some-file-name.md and processes them with the output being created in destination/yyyy/mm/dd/some-file-name.md. I managed to string together an ugly script that converts the source path to destination path, but the problem is that make says that there is no rule to make the target. Here is my code so far:

SOURCE := $(wildcard source/*.md)
DESTINATION := $(foreach f,SOURCE,destination/$(shell echo $(notdir $(f)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/'))

$(DESTINATION): $(SOURCE)
	@mkdir -p $(dir $(shell echo $(notdir $@) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3/'))
	/bin/bash ./myscript.sh $< > $@

I believe that make sees that there are no target directories yet and just aborts the rule, but I’m not knowledgeable enough and just starting out with that tool. I would appreciate some guidance if any of you folks know a bit of Make.

  • eleijeep@piefed.socialEnglish
    7·
    10 days ago

    Is this the whole Makefile? The default target is the first rule that appears in the file which in this case is the DESTINATION rule, so if you just type make then it should work. If make is telling you there’s no rule to make the target then you’re probably invoking something like make all and you don’t have an all target.

    When I run it your Makefile with no target, I get this error:

    $ make -f blamster.mk 
/bin/bash ./myscript.sh source/1234-56-78-some-file-name.md > destination/SOURCE  
/bin/sh: 1: cannot create destination/SOURCE: Directory nonexistent  
blamster.mk:5: recipe for target 'destination/SOURCE' failed  
make: *** [destination/SOURCE] Error 2

    The following fixed Makefile works for me:

    SOURCE := $(wildcard source/*.md)  
DESTINATION := $(foreach f,${SOURCE},destination/$(shell echo $(notdir $(f)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/'))  

all: ${DESTINATION}  

$(DESTINATION): $(SOURCE)  
	mkdir -p $(dir $@)  
	/bin/bash ./myscript.sh $< > $@

    The things I had to change were: In the foreach you need to reference SOURCE as a variable ${SOURCE}. In your recipe you can just create the directory directly from the output filename instead of doing another regex. Lastly, to create all output files you also need to create a default target (eg. all) that depends on all of the output files.

    • fruitcantfly@programming.dev
      4·
      9 days ago

      This breaks down when there are more than one SOURCE file, since each DESTINATION file depends on all source files. This means that $< will always be the first file in SOURCE. For example, if source contains the files 0001-01-01-some-file-name.md and 0002-02-02-some-file-name.md:

      $ make -n
mkdir -p destination/0001/01/01/  
/bin/bash ./myscript.sh source/0001-01-01-some-file-name.md > destination/0001/01/01/some-file-name.md  
mkdir -p destination/0002/02/02/  
/bin/bash ./myscript.sh source/0001-01-01-some-file-name.md > destination/0002/02/02/some-file-name.md

      OP is probably better off using a scripting language to automate this kind of thing

      EDIT: Fixed example filenames

      • eleijeep@piefed.socialEnglish
        3·
        9 days ago

        Well spotted, thanks. I replied further down the thread to OP with another option for doing this in Make, if you’d like to spot any bugs in that one too ;)

    • blamster19@programming.devOP
      2·
      9 days ago

      Oh right, that was not the whole Makefile, only the part that caused problems. As @fruitcantfly@programming.dev pointed out, it breaks down with more SOURCE files and populates the newly created directories with only the first SOURCE file. After some more poking and reading I found a method that works by using define:

      define rule
$(DESTINATION)/$(shell echo $(notdir $(1)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/'): $(1) | $(DESTINATION)/$(shell echo $(notdir $(1)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3/')
	/bin/bash ./myscript.sh $$< > $$@
endef

$(foreach f,$(SOURCE),$(eval $(call rule,$(f))))

      I feel like this is an abuse of Make, but it works. When is it a good idea to delegate the building process to other tools?

      • eleijeep@piefed.socialEnglish
        2·
        9 days ago

        Yes, I didn’t notice that, I should have tested it more thoroughly!

        Make works best when there’s a simple relation between target filename and the dependency filenames, and when the output filenames are known ahead of time (or are easily derived from a list of inputs).

        You can use Make to achieve what you want though, even when the relations are complex. The idiomatic pattern is to dynamically create your dependency lists and include them at runtime. Make supports this as a first-class feature and will handle re-evaluating any rules that are changed every time an included Makefile is (re-)generated.

        For example:

        SOURCE := $(wildcard source/*.md)
SDEPS := $(SOURCE:.md=.d)

include all.d
include ${SDEPS}

all.d: ${SDEPS}
	sed -i "s/^/all: /" all.d

${SDEPS}: %.d: %.md 
	DEST="destination/$$(basename $< | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.md$$/\1\/\2\/\3\/\4.md/')"; \
	printf "%s: $<\n" $${DEST} >$@; \
	printf "%s " $${DEST} >>all.d

%.md:
	mkdir -p $(dir $@)
	/bin/bash ./myscript.sh $< >$@

        (I’ve tested this one and it seems to work)

        But if you want to avoid any complex use of Make then you can push a lot of the logic into your script and then just create a timestamp file for each input as a dummy output to represent the fact that the script was run on each input. eg.

        SOURCE := $(wildcard source/*.md)
SOURCE_TS := $(SOURCE:.md=.ts)

all: ${SOURCE_TS}

%.ts: %.md
	/bin/bash ./myscript.sh $<
	touch $@

        This is a much simpler Makefile, but now you need to create the output filename, make the output directory and redirect the script output inside the script itself. The other drawback is that it’s harder to use the output file as an input to another rule. You can use the timestamp file as an input but you won’t have the actual filename available to send to a command without some more scripting.