ocamlbuild-ctools - An ocamlbuild plugin for building C/C++ source with variants

1
2
3 Building a mixed C/OCaml library using ocamlbuild ctools plugin:
4 ================================================================
5
6 (On Windows we also use the .a extension, it will be translated automatically,
7 we don't use ocamlmklib because it is uncertain how well it works cross platform,
8 since there are dll issues - we don't create dll's here).
9
10
11 Given source tree
12
13    src/ml/bar.ml
14    src/ml/mylib_wrapper.ml
15    src/mylib/a.c
16    src/mylib/b.c
17    src/mylib/mylib_ml.c
18
19 We assume the file mylib_wrapper.ml references functions in mylib_ml.c and that
20 mylib_ml.c includes standard "caml/something.h" files.
21
22 Here we give a standard approach using the ctools ocamlbuild plugin.
23
24
25 Copy myocamlbuild_config.ml from ocamlbuild/ctools to source tree root. Make
26 sure the cppinclude tool "cppi" is in the executable path - this is needed for
27 dependency scanning c and related files.
28
29 Create myocamlbuild.ml in source tree root:
30
31   contents of myocamlbuild.ml:
32
33   (* requires ocamlbuild/ctools/myocamlbuild_config.ml
34    normally adding to current directory during start of build process *)
35
36   open Ocamlbuild_plugin
37   open Myocamlbuild_config
38   open Myocamlbuild_config.Build
39
40   let my_project () =
41     ()
42   ;;
43
44   Build.build my_project;;
45
46
47 Next we need to create two libraries: a pure C library
48
49   src/mylib/libmylib.a
50
51 And a pure ocaml library, both .cma and .cmxa versions:
52
53   src/ml/mylib.cmxa
54   src/ml/mylib.cma
55
56
57 These libraries can be created as follows:
58
59   create the files:
60
61   src/mylib/libmylib.clib:
62     a.c
63     b.c
64     mylib_ml.c
65
66   src/ml/mylib.mllib:
67     bar.ml
68     mylib_wrapper.ml
69
70 Note: we could also use mylib.mlpack, but then we must also tag ml files with
71
72    "<src/ml/*.ml>: for-pack(Mylib)
73
74 in the ocamlbuild _tags file.
75
76 Note: If we only have one file mylib.ml, ocamlbuild can automatically create a
77 .cm{x}a file, but if we create a dependency on the c library, the library is included
78 as an ordinary archive member instead of an external c library sometimes
79 causing linker errors when the library is not in the search path.
80 If we use mylib.mllib containing just "Mylib", we avoid this problem. Therefore, always
81 create either an .mllib or .mlpack file.
82
83 The c and ocaml libraries do not work properly yet because .cm{x,}a does not reference
84 libmylib.a and mylip_wrapper.ml needs functions from this library, and
85 because the mylib_ml.c file includes caml headers it cannot find.
86
87 We can add the caml headers to the c library as follows:
88
89   In myocamlbuild.ml, update my_project:
90
91   let my_project () =
92     c_include ["include_ocaml"] (C.find_ocamllib())
93   ;;
94
95 Create the "_tags" file and add the line:
96
97   <src/mylib/mylib_ml.c>: include_ocaml
98
99 To get the .cm{x,}a file to link with clib we need to both declare the c
100 library so it creates a flag the works with use_libmylib, and we need to tag
101 the .cm{x,}a file with use_libmylib, finally we need to add a dependency
102 between .cm{x,}a and the c-library.
103
104 However, we will not do this, since this is done automatically by two ctools
105 functions: dist_clib and dist_ocaml_clib.
106
107 We add the following lines to my_project:
108
109   dist_clib [] "libmylib" "src/mylib/libmylib.a" "src/mylib" "dist/lib/ocaml";
110   dist_ocaml_clib "mylib" "libmylib" "src/ml" "dist/lib/ocaml"
111
112 The dist_clib function will build debug and release versions of the library. It
113 can also add other compile flags such as "ml_api" and create new build variants.
114 This is controlled by the variant list, which is simply [] in our case.
115
116 Likewise, dist_ocaml_clib can change "libmylib" to "libmylibd" or some other
117 variant if we have created it (libmylib.a and libmylibd.a are available by default).
118
119 The include flag for clib is defined as include_libmylib and the linker flag
120 is defined as use_libmylib and use_libmylibd.
121
122 dist_ocaml_clib adds a dependency to the c library, and adds "-cclib -lmylib"
123 to the ocaml linker step, and makes sure the libmylib.a file is copied next to
124 the mylib.cm{x,}a before linking, otherwise the mylib.cm{x,}a file may contain
125 a relative path that makes it difficult to find libmylib.a once we copy the
126 library elsewhere, it also adds the -custom tag to byte code .cma libraries.
127
128 Note: The c library does not have to be named "libmylib", but if it doesn't
129 have the "lib" prefix, it is unclear whether the library can be located
130 correctly. The lib prefix convention also works on Windows because the ocaml
131 compiler translates this internally before calling the c compiler/linker.
132
133 Finally, we need to create a component to actually build the library. Until now
134 we only have a lot of build rules defined.
135
136 In the source root, create the file
137
138   mylib.component:
139     dist/lib/ocaml/mylib.cmxa
140     dist/lib/ocaml/mylib.cma
141     dist/lib/ocaml/mylib.cmi
142     dist/lib/ocaml/mylib.a
143     dist/lib/ocaml/libmylib.a
144     (and optionally libmylibd.a and mylib.mli)
145
146 The dist rules have defined copy rules to make the files available in the dist
147 directory, but the component is required to actually force the files to build.
148
149 Now we can run:
150
151   ocamlbuild mylib.stamp
152
153 We can also create the component all.component:
154
155   all.component:
156     mylib.stamp
157
158 Now we can simply build using
159
160   ocamlbuild all
161
162 (because ctools understall the all target as all.stamp)
163
164 Additional notes:
165 =================
166
167 mlpack and include paths:
168
169 If we use mylib.mlpack, we need to tag each .cmx file with -for-pack:
170
171   _tags:
172     <src/ml/{bar,mylib_wrapper}.cmx>: for-pack(Mylib)
173
174 This is not needed for byte code libraries.
175
176 If the .ml source references source in other directories, we can extend the
177 include path either using
178
179   ocamlbuild -I src/other/ml all
180
181 or we can add an entry in my_project() of ocamlbuild.ml:
182
183   Pathname.define_context "src/ml" ["src/other/ml"];
184
185 If C files include source code in other directories, we need to add c_include
186 to my_project, for example:
187
188   c_include ["src/mylib"] "src/other/c_headers";
189
190 For the library we built, we can also use include_libmylib as a tag files need
191 the library include path.
192
193 Even when c_includes are set up correctly, we may need to use
194 Pathname.define_context because the C dependency scanner does not understand these
195 flags and need to know where to search, so we add:
196
197   Pathname.define_context "src/mylib" ["src/other/c_headers"]
198
199
200 Advanced library creation:
201 ==========================
202
203 If we have several ocaml libraries, some of which a pure OCaml and some mixed C/Ocaml,
204 and some using Pack, we can this as follows:
205
206 We assume all incoming libraries have either .mllib or .mlpack files and can
207 built on their own. We also assume that multiple libraries do not use the same
208 source files, if they are tagged with -for-pack (then the project should be
209 reorganized).
210
211 Lets assume we have the input libraries:
212
213   src/x/x.mllib
214   src/y/y.mllib
215   src/clibs/lib1.clib
216   src/clibs/lib2.clib
217   src/z/z.mlpack
218
219
220 We now create a new directory and an .mllib and a .clib file:
221
222   src/everything/everything.mllib
223   src/everything/libeverything.clib
224
225 In myocamlbuild.ml, we add the paths:
226
227   Pathname.define_context "src/everything" ["src/x"; "src/y"; "src/z"]
228
229 The everything.mllib file is created with the content:
230
231   X
232   Y
233   Z
234
235 The libeverything.clib is updated by copying the content of lib1.clib and lib2.clib into
236 libeverything.clib. Thus, we create a new library from the original source files. We do not
237 try to reuse the original libraries. (If we need to do this, some magic with ar, libtool or
238 whatever might do the trick, but it is simpler to avoid the problem altogether).
239
240 Notice that it doesn't really matter what libraries are using for-pack, links with c or are just
241 plain ocaml libraries. In either case we just add the top-level modules. Ocamlbuild will try
242 to build the X.cmxa, X.cma and so on. Later on we get linker error if c functions are missing, but
243 we cover that with our new library.
244
245 That is basically it, we create dist_clib and dist_ocaml_clib as for normal libraries.
246 Also, we still need the dist_clib and dist_ocaml_clib for any mixed ocaml libraries we include.
247
248 One tricky step remaining:
249
250 We need to create everything.component:
251
252   dist/lib/ocaml/everything.cmxa
253   dist/lib/ocaml/everything.cma
254   dist/lib/ocaml/everything.a
255   dist/lib/ocaml/libeverything.a
256   dist/lib/ocaml/x.cmi
257   dist/lib/ocaml/y.cmi
258   dist/lib/ocaml/z.cmi
259
260 Notice that since we did not add any source files to everything, there is no .cmi file.
261 In fact, there is no ocaml module named Everything. A .cmi file is required for every module
262 in the library.
263
264 The function dist_ocaml_clib ensures .cmi files are copied to the dist directory. For ocaml only libraries
265 we can either use dist_ocaml or simply create a copy rule for the .cmi file:
266
267   copy "src/x/x.cmi" "dist/lib/ocaml/x.cmi"
268
269 or
270
271   dist_ocaml_lib "x" "src/x" "dist/lib/ocaml"
272
273
274 Linking our library with test code:
275 ===================================
276
277 For example:
278
279   ocamlbuild -lflags I,dist/lib/ocaml -cflags I,dist/lib/ocaml -libs src/test/test.ml
280
281 We need to add the linker include path to our library.
282 We should also be able to tag test.ml with use_libmylib, but this has not been tested.
283
284 We could also use
285
286   ocamlbuild -I dist/lib/ocaml src/test/test.ml
287
288 But it doesn't work for paths outside the current source tree (which is
289 typically the case when using a collection of libraries).
290
291
292
293 ===================
294 Misc. design notes:
295 ===================
296
297
298 Variant build example:
299 ======================
300
301 variant build example:
302
303 given files
304     mywork.component (content: "program1.prog program2.prog")
305     program1.cprog (content: file1a.o file1b.o)
306     program2.cprog (content: file2a.o file2b.o)
307     file1a.c
308     file1b.c
309     file2a.c
310     file2b.c
311
312 Without variants the following build targets can be given to ocamlbuild:
313
314     ocamlbuild program1.prog (building file1a.o file1b.o)
315     ocamlbuild file1a.o
316     ocamlbuild mywork.stamp (building program1.prog and program2.prog through program1.cprog and program2.cprog)
317
318 We can now build a debug version of file1a.o:
319
320     ocamlbuild file1a.variant_debug.o (from file1a.c with debug flags (-g for gcc))
321
322 Likewise we can build a debug version of program1:
323
324     ocamlbuild program1.variant_debug.prog
325
326 and the entire mywork.component can be built in debug and release versions:
327
328     ocamlbuild mywork.variant_debug.stamp (building program1.variant_debug.prog, file1a.variant_debug.o etc.)
329     ocamlbuild mywork.variant_release.stamp (building program1.variant_release.prog etc.)
330
331 This works by detecting the variant name, here "debug" or "release" and translate input specification files
332 for example:
333     program1.prog in mywork.component becomes program1.variant_debug.prog before it reaches the build engine.
334     But when building program1.variant_debug.prog the rule tells it to look for program1.cprog as input.
335     When reading program1.cprog, the variant is again detected from the target name program1.variant_debug.prog
336     and therefore file1a.prog is translated to file1a.variant_debug.prog etc.
337
338 When linking and compiling, the variant name is automatically added as tag.
339 Therefore, flags can be added to specific variants.
340 "debug" and "release" are officially supported variants because flags have been added in the generic configuration:
341
342 flag ["c";   "compile"; "gnu";  "debug"]    (S[A"-g"]);
343 flag ["c";   "compile"; "gnu";  "release"] (S[A"-O2"]);
344
345 for the "gnu" tool chain, and similar for other tool chains like "msvc".
346 tools are hardcoded at runtime so they are not variants (but could be).
347
348
349 Custom variants:
350
351 Assume we want to build a variant of the mywork.component that
352 generates html output instead of ordinary text output from the program1 and program2:
353
354 We add the flags:
355   flag ["c"; "compile"; "gnu"; "htmlout"] (S[A"-DOUTPUT=HTML])
356   flag ["c"; "compile"; "msvc"; "htmlout"] (S[A"/DOUTPUT=HTML])
357
358 Now all c files will get the define OUTPUT=HTML if the files have been tagged htmlout.
359 One way (but not the only one) to define such a tag is to build a variant with this name:
360
361   ocamlbuild mywork.variant_htmlout.stamp
362
363 Another way to add the htmlout tag is to add a _tags file and write "<**/*.c> : htmlout"
364 but then we we will always get htmlout defined.
365
366 Multi-variants:
367
368 assume we want to build a debug version with html output:
369
370   ocamlbuild mywork.variant_debug,htmlout.stamp
371
372 This works by detecting by splitting the variant name with "," such that "debug,htmlout"
373 generates two tags: "debug" and "htmlout".
374
375 Likewise, program1.variant_debug,release.prog can be build to get debug information in
376 a release compilation of the program.
377
378
379 Dependency scanning:
380 ====================
381
382 - dependency scanning with dynamic discovery of generated dependencies
383
384 The general convention:
385
386 %.includes : files that % directly includes.
387 %.includes_acc : files that % directly or indirectly includes (an accumlated includes file).
388 %.depends : same as %.includes_acc but (usually) depends on .includes_acc being built
389   and allows for more advanced operation given that dynamically generated dependencies
390   have been discovered and built or copied from source tree.
391
392 %.includes files are by default created by a simple fast include file scanner.
393 It can be replaced entirely or for specific file extensions.
394 The default scanner responds to the pattern:
395
396   linestart spaces? '#include' spaces? ('<' filename '>' | '"' filename '"' ) anything lineend
397
398 The scanner writes the result of scanning % to the %.includes file. The idenfitied files
399 are not visited during this step. A seperate %.includes_acc step visits the .includes file.
400
401 When building includes_acc, first .includes is built as a dependency, then
402 all files in the .includes file are visited and their corresponding
403 .includes_acc are built. This in turn triggers %.includes file build.
404 This in turn triggers % file build which either causes a source file copy or triggers the build
405 of a dynamically generated dependency. If any part of this build breaks, the result is
406 ignored. This handles files included from outside the source tree which should not be build.
407 If it actually is a build failure, later stages of the build will presumably fail.
408
409 It would be obvious to have the accumalated include files be named ".depends" instead of
410 ".includes_acc", but this can become a problem:
411 Consider a custom .c -> .c.depends dependency scanner using a C preprocessor.
412 If a .c file is included in some other file and we simply used .depends as the accumlated list of
413 included files, then the include build logic would attempt to build .c.depends as part of the
414 recursive include file discovery. But because of the custom .depends rule we would start up
415 a preprocessor scanner for an included file. This can file both because the recursive discovery
416 will be broken, and because the custom dependency scanner will not operate on the correct context.
417 In other words, we need to separate recursive dynamic discovery from custom dependency rules.
418 Instead we build .includes_acc file during the discovery process and copies (by default) the result
419 from includes_acc to .depends for top level files only. This copy is triggered for example by having
420 .c -> .o depend on .c.depends.
421
422 A custom %.<myext>.depends rule need not require that <myext>.includes* files be created
423 - it can use any means to create the .depends file. This is controlled by having dependencies
424 .<myext>.depends -> .<myext>.includes_acc or not.
425 Usually it helps to depend on includes_acc because this at least ensure that dependent files will
426 be copied to the build tree before the custom step is activated.
427
428 %.includes and %.depends can be created manually in the source tree to override incorrect
429 automatic dependency generation.
430 A manual .includes will only fix the scanning of a single file but
431 recursive include logic is still applied.
432 A manual .depends file will complete block any further dependency calculations. This can
433 speed up the build, but is sensitive to underlying changes in the include patter of underlying
434 files.
435
436 When building % to output (e.g. .c to .o) the assumption is that a .depends
437 file contains all dependencies. Therefore the default rule simply copies
438 .includes_acc to .depends. This simplifies the normal build rule logic.
439
440 A rule like %.c -> %.o should have %.c and %.c.depends as dependency.
441 The .depends dependency ensures that that all the include file and dependency
442 machinery is running. But it will only ensure that the correct include files
443 are discovered, it will not actually make the build depend on these files in case
444 they later change. Therefore:
445 The rule should also dynamically (using the build function)  build the contents
446 of the .depends file. This ensures that the output file (e.g. .o) will discover
447 changes in all dependent files. Only depending on the .depends file will correctly
448 build all dependencies the first time, but changes will not be correctly discovered
449 during a second build unless the dynamic rule is applied.
450
451 It is important that .includes_acc actually contains all the accumulated filenames.
452 Normally it could work by just touching the file, or by copying %.includes to %.includes_acc,
453 but this could cause incremental builds to fail or produce incorrect results.
454 The same applies to the .depends file - or actually - it only applies to the .depends file when
455 the .depends file does not depend on the .includes_acc file.
456
457 It is also important that .includes_acc only contains the accumulated filenames that actually
458 succeeded in building. Otherwise missing files will not be detected during rebuilds. If a file
459 suddenly becomes missing, this will change the .includes_acc and .depends files and force a rebuild.
460 ->
461 Unfortunately, ocamlbuild does not mirror removed files. Therefore missing files will not be detected
462 even if the modified .depends file triggers a rebuild.
463
464
465 Conditional Targets with Tag Patterns:
466 ======================================
467
468 used in build spec files such as .cprog, .clib, .component
469
470 example:
471
472 myprogram.cprog:
473
474   file1.o file2.o
475   [~debug] output.o
476   [debug] trace.o logging.o
477   [debug,test] test_suite.o
478   [release,~nocopy] copyprotect.o
479   [win32] ../sys/win32io.o ../sys/winbasics.o
480   [win64] ../sys/win64io.o
481    ../sys/winbasics.o
482   [~platform:windows] ../sys/posix/sysio.o
483   [] file3.o file4.o
484
485 tag patterns are matched against the build target (myprogram.prog), not the input files.
486 Files are matched if the previous pattern is matched, or if there are no previous pattern.
487 Tags starting with "~" are exclusive. Such tags must specifically not be present or the following files
488 will not be included in the build.
489 If a pattern is missing at the beginning, it is assumed to be "[]".
490 "[]" will match anything (and can thus terminate a previous pattern).
491
492 If a tag is both inclusive and exclusive the pattern will never match: "[x,~x]".
493 Empty patterns are ignored: "[x,,y,~]" -> "[x,y]".
494 Linebreaks have no special meaning other than space.
495
496 Spaces are not allowed in patterns, but only because of weak parsing powers
497 - a pattern must be a single token.
498 [release, not nocopy] could also be valid with a better parser.
499
500 Spaces in filenames are not supported, but only due to limited parser.
501 "File with spaces.txt" should be allowed when quoted with "".
502
503 Build variants can affect the result:
504  ocamlbuild myprogram.variant_debug.prog
505  ocamlbuild myprogram.variant_debug,release.prog
506
507
508 Components can also use tag patterns, for example:
509
510 all.component:
511
512   dist/bin/program1.prog
513   [tools]
514   dist/bin/extratool.prog
515
516 ocamlbuild all.stamp
517   -> builds program1.prog
518
519 ocamlbuild all.stamp -tag tools
520   -> also builds extratool.prog
521
522
523
524 Includes.acc rules:
525 ===================
526
527 "%.includes_acc" is the transitive closure of all included files listed in "%.includes"
528 files:
529
530 If we build myfile.includes_acc, this will force the build of myfile (if absent)
531 then the build of myfile.includes, then for each file in myfile.includes, the corresponding
532 myfile.includes_acc will be built.
533
534 We can use this to create .depends files. And we can use this to auto-discover included
535 files such as .h files, so they get copied to the build dir.
536
537 We can also use this to build dynamically generated files, even if they are only listed as
538 included files. For this to work, the cooperation between "%", "%.includes" and "%.includes_acc"
539 is important, including their dependencies. We try not to build too much at a time because
540 we could risk that some included file would not have been built yet. This is a problem for typical
541 dependency generators.
542
543 %.includes can contain out-of-source files, for example <stdio.h>. The .includes_acc will be
544 stripped from any files that can not be built in the build dir, or copied from the source tree,
545 but will not generate any missing include file errors.
546
547 This usually works, but can fail in one special case: if an included file exists in the source tree,
548 and the build succeeds and the included file is later removed,
549 ocamlbuild (v.0.10) not delete the file in the build dir.
550 The build will discover a change in the dependency list,
551 force a rebuild (e.g. compile an .o file again), but since the c compiler finds the missing file
552 in the build dir, the build succeeds where it should break.
553
554 The above system is not limited to C/C++ source and header files but as an example:
555
556 If we have a rule for %.c -> %.o it should depend on %.c and %.c.depends. It should read
557 the contents of the %.c.depends file and dynamically build all the files listed using the build function.
558 It should not at all care about .includes files or .includes_acc files.
559
560 We can create another rule to build the %.c.depends file. We can use any means to do this including
561 providing a manual .depends file in the source tree.
562
563 The easiest way to create a .depends file is to create a rule that depends on %.c and on %.includes_acc
564 and then copy %.includes_acc into %.depends (possibly modifying fileformat if needed). The default setup
565 does not do this - instead it used the includes_acc rule logic but creates a .depends output file instead
566 of an includes_acc file at top-level. This reduces the number of build targets.
567
568 %.includes and %.includes_acc file format:
569
570 A space or newline simple list of file names. Unlike common .depends files, this format
571 is not mytarget.depends contains "mytarget: dependency1 dependency2 ..."
572 but mytarget.depends contains "dependency1 dependency2 ..."
573 If a file contains spaces it will be quoted like: " filename with spaces ".
574 This is not really tested: The default scanner tool 'cppi' will handle spaces,
575 but ocamlbuild might not (uses ocamlbuild string_list_of_file).
576
577 FAQ:
578
579 Why is %.includes_acc not simply called %.depends?
580 Because:
581 a) to seperate generic build rules from the include scanning logic and make it optional.
582 b) if an included file had a custom depends rule (say including a .c file in a .c file), the include
583   engine would accidentally run the custom rule instead of transitive closure logic it needs for itself.
584
585 Why is there both %.includes and %.includes_acc files?
586 Because:
587 a) we need a hook for custom scanning of immediate includes e.g. a new special rule for %.rl.includes
588 b) we cannot scan the transitive closure in one step - not if an included file needs to be built.
589   therefore we create dependencies between %, %.includes and %.includes_acc to let the build engine
590   drive the work in the correct order.
591
592 A stupid includes file scanner can include too many files including non-existing files, isn't that bad?
593 Generally no:
594 It is ok to include missing files - they will be stripped from includes_acc as if they were out of source.
595 It will not become a dependency and since it is not needed, no harm is done.
596 Of course, we risk creating a dependency on a valid file that could break the build - in this case
597 a manual .depends file, or an .includes file or even an .includes_acc file can be created in the source tree.
598
599 A stupid includes file scanner might not discover all files, isn't this a problem?
600 Yes, it does not happen that often, but it can happen. One solution is to include a source file
601 that helps the include system discover all files. Another is to create a manual .includes or .depends file.
602
603 But can't we use a proper C-preprocessor depedency scanner (or similar for other tools)?
604 Sometimes. But even in simple cases the .h header files must be copied to the build dir before such
605 a scanner will work. With generated files such as parser generators, it becomes even more problemetatic.
606 In many cases the simple scanner will work better and faster.
607
608 But if pre-processor scanner is really needed?
609 Create a rule for %.depends that depends on %includes_acc - this will build or pull all the needed files
610 to the build directory (with the limitations already mentioned). The preprocessor should now have all the
611 files it needs to do a thorough dependency scanning which can be used as input to the .c -> .o rule.
612
613