path: root/README (plain)
blob: d777dfb18484b71389f9f83b03946d3e56faacb0

1
2	Requirements:
3	=============
4	Symbiosis requires ocamlbuild from the OCaml 3.10.1 distribution.
5
6	End-users do not require ocamlbuild or OCaml after Symbiosis has been
7	compiled.
8
9	Tutorial:
10	=========
11	You also want to read the tutorial in
12
13	doc/tutorial.html
14
15	This README contains some more technical reference documentation at the end.
16
17
18	Update:
19	=======
20
21	Since this documented were last updated, git source control
22	support were added to Symbiosis. The tutorial covers git. This document uses
23	monotone for examples.
24
25
26	Introduction:
27	=============
28
29	Synopsis is a meta build engine: It will check out multiple source
30	repositories and build them in order. The term "build" is quite general - any
31	kind of action can be performed using dedicated "agents".
32
33	Symbiosis is designed to glue a large project together from a large number of
34	small projects, each in their own source code repository.
35
36	In particular, Symbiosis is designed to cross-glue many different medium to
37	large projects that each reuse the same smaller projects in slightly different
38	ways.
39
40	Symbiosis replaces the development model where all projects are copied to one
41	large source tree and checked in, and the other model where one source control
42	tool can include sub projects, but which is not flexible enough and ties all
43	the source to the specific source control tool.
44
45	The one big source model tends to explode when many smaller projects must all
46	be in the same source tree has problems with versioning and variant creation,
47	huge disk space consumption, long build times, fragile build configurations
48	and manual import of updated external source. This model may work when there
49	is only one big project going, but when many different projects reuse some of
50	the same sub projects, this becomes a nightmare.
51
52	Modern build tools such as SCons and ocambuild use signatures to only build
53	what really has changed. Symbiosis attempts to do the same at the meta-build
54	level. However, Symbiosis do not try to track every change in every
55	sub-project. It depends on installable "agents" to extract key signatures from
56	a local build and pass this on for change tracking. Thus, symbiosis will only
57	be as good as the underlying local build support.
58
59	Symbiosis makes no hard requirements on project local build tools and will
60	happily use make or shell scripts, although a powerful and friendly tool like
61	ocamlbuild is recommended (the name suggest OCaml only, but it really is a
62	fast, powerful, general purpose cross platform build tool - as of this writing
63	a C/C++ extension is being developed with fully automated dependency scanning
64	of include files, including dynamically generated files with non-standard file
65	extensions). Symbiosis originated as a Ruby build engine named "components"
66	but has now been ported ocamlbuild and further developed on this platform.
67
68	During software development (not build server time), developers will modify
69	source code directly where Symbiosis has checked out the source. This removes
70	the need for manually of tracking branches of multiple dependent projects or
71	writing checkout scripts for each project constellation. Once an change is
72	ready to be committed, this is done directly inside the Symbiosis controlled
73	workspaces.
74
75	For continuous integration, the same Symbiosis configuration can be reused on
76	build servers. For cross platform builds, each local project is supposed to be
77	platform aware, and build the proper variant.
78
79	The overall build should be under source control just as much as the specific
80	projects being built. Symbiosis interacts with source control tools for all
81	meta-data, including multiple host configuration families. Most of this is
82	done through simple branch naming conventions, but also through the way
83	meta-data has been partioned, for example system independent .proxy files and
84	family specific .volume files. Using this approach, source control tools can
85	easily deploy and update build configurations on different build servers.
86
87	Symbiosis is not tied any particular source control tool or build tool (other
88	than for its own internal logic where it uses ocamlbuild). Moreover, Symbiosis
89	makes it agreeable to gradually change tools over time or across systems at
90	the same time.
91
92	A core philosophy of Symbiosis is to avoid large centrally controlled build
93	configurations yet provide a very sharp reproducible version controlled build
94	environment. One future vision is to enable full signature tracking of every
95	local build including the exact revision used for building and provide a
96	script that can recreate this build: Normally a project is build as "latest
97	source on current branch", but at build time this is always a specific
98	revision which we might as well keep track of. In this way we can recreate an
99	exact build later on if a bug report is linked to a release signature of the
100	build log.
101
102	As few tools as possible should be system installed. Source control tools and
103	large compilers like gcc would be, but zlib and small parser tools might as
104	well be built from a source controlled version, now that we have the means to
105	easily do this. This both simplifies deployment of build platform and ensures
106	reproducibility and stability. A downside (and potential upside) is that we
107	need to track security patches in a different way than just using the
108	operating systems security patches.
109
110
111	Getting started:
112	================
113
114	Well, basically it is not possibly to start from quickly from ground zero
115	because several key files must be configured correctly including how to locate
116	source code repositories and how to locate and build projects. These files
117	will become part of the source controlled project.
118
119	Once the initial configuration is done and understood, it is fairly easy to
120	add new projects and to include existing projects in new projects.
121
122	Now, assuming we already have a working configuration:
123
124	1) Make sure we have the symbiosis.system configuration in the local path or
125	in ~/.symbiosis/symbiosis.system (strictly speaking, this file is optional, but
126	that would not make life any easier).
127
128	2) Make sure symbiosis is installed as an executable binary
129	(see Source code and INSTALL at end of this doc).
130
131	Ideally, step 1) and 2) are done by checking out a configuration and a binary
132	from source control on a branch matching the local system (see source control
133	at end of doc).
134
135	3) Make sure the local repositories directory contain relevant local
136	repositories according to the system configuration. For example
137	~/repositories/hg/myprojects.hg
138
139	This step might not be needed if we only use remote source control as per
140	system configuration. For distributed source control we would normally like to
141	have local staging repository.
142
143	4) Try to figure out if we have all key tools installed. Looking into
144	symbiosis.system tools section can give hint, notably source control tools.
145
146	The symbiosis binary installed at step 2) will likely have built in agents for
147	tool such as hg, git, mtn, cvs or svn, and system.symbiosis will likely have
148	a configuration to check out additional configuration information using one
149	such agent.
150
151	4) Find a nice empty working
152	directory, for example ~/work/test
153
154	5) run 'symbiosis init'
155
156	This will fail if required source control systems have not been installed - see step 4).
157
158	According to the symbiosis.system file, this will extract .proxy and .volume
159	files into the activities/workspaces subdirectory, and keep track of this in
160	activities/_oversight.
161
162	For simpler configurations, we may not check out .proxy and .volume files, but
163	simply point to pre-existing directories. In the simplest case we do not even
164	have a configuration file and simply use ./proxies and ./volumes directories.
165	In this case the "symbiosis init" step is not required.
166
167	6) Try to build a real project. Assuming we have "myproject.proxy" file
168	available in the proxies directory and a "myproject.synopsis" file available
169	inside the source code of myproject:
170
171	run 'symbiosis myproject.proxy.synopsis.build.resp'
172
173	The myproject.proxy file points to a source location. This in turn requires a
174	.volume file, as specified inside the .proxy. The volume file knows about our
175	source code repository. The .synopsis file knows how to execute "build" on the
176	source code.
177
178
179	7) Try to check out a source repository without building it:
180
181	run 'symbiosis another/project.proxy.volume.checkout.resp'
182
183	In step 6) the synopsis.build action automatically checks out the myproject
184	source. Here we do it manually. Internally Symbiosis figures out that any
185	synopsis action (like "synopsis.build) requires the source code to be
186	available, therefore it looks up the volume of the corresponding proxy and
187	performs a checkout operation.
188
189	Project names:
190	==============
191
192	Notice that the project we checked out in getting started step 7) was named
193	"another/project" and the earlier project in step 6) was named "myproject".
194
195	Project names are defined by the relative path from the "proxies" directory to
196	the proxy file, excluding the ".proxy" extension. The "proxies" directory
197	location is defined in the symbiosis.system file.
198
199	It is a good idea to name project with a sensible prefix like
200	"myorg/myproject" so we can have multiple collections of proxy files without
201	conflicts.
202
203	The symbiosis.system file can check out any number of "meta-volumes"
204	including several independent proxy volumes, all in a subdirectory of
205	the main "proxies" directory. This is driven by the "symbiosis init" step.
206
207
208	How it works:
209	=============
210
211	Symbiosis does not deal with components or projects. It has a concept of:
212
213	systems, tools, agents, repositories, actitivies,
214	volumes, proxies, branches, workspaces, and the "oversight".
215
216	A project is represented by a small .proxy file and a slightly larger
217	.synopsis file. A proxy is located in the "proxies" directory given by the
218	system configuration.
219
220	Each proxy can be used to locate a version controlled source tree and to
221	create a workspace copy (checkout) under the "activities/workspaces"
222	directory. Is also used to locate the synopsis file for the workspace.
223
224	A .synopsis file is only interesting after we have a workspace checked out. A
225	synopsis knows how to build the source tree. For example, if a source has a
226	Makefile with the targets all, doc, and clean, the synopsis file might have
227	the actions "build" mapping to "make all", "clean" mapping to "make clean" and
228	"document" mapping to "make doc".
229
230	The synopsis also knows about dependencies. A dependency is a proxy name and
231	a corresponding synopsis action. Before a synopsis action is executed, all
232	dependent actions will be executed, including the creation of workspaces
233	copies if needed.
234
235	It is often convenient to have a synopsis file close to the source tree. This makes
236	it possible to version it together with the source and to update the file when
237	internal build changes are made. The public interface through the synopsis remains
238	unchanged. For this reason, proxies and synopsis files are separated. A proxy can
239	also select between multiple synopsis files depending on settings such has host system.
240	Thus, a proxy provides a unique identifier that changes infrequently and a synopsis
241	provides hands-on information.
242
243	Sometimes a synopsis is better left outside the source tree. We may have
244	external source that we do not want to modify or we may simply want to quickly
245	add new source to symbiosis without modifying the source tree. In this case, a
246	synopsis file can be located next to the proxy file so both
247	"<myproject>.proxy" and "<myproject>.synopsis" exist in the same directory
248	under "proxies." Such a file takes precedence over any other .synopsis file.
249
250
251	How Symbiosis build:
252	====================
253
254	Symbiosis never starts a "build". It also does not have phases like fetch
255	build etc. Instead it executes an "action", typically a synopsis action such
256	as "build" for a given proxy. The dependencies of that action determines the
257	further of actions, and will naturally cause workspaces to be created as
258	needed.
259
260	Symbiosis does not have a concept of a project configuration. Any given
261	proxy is a project in this sense. So depending on the point of view, Symbiosis
262	either has a lot of projects, or none.
263
264	Symbiosis has a system configuration file named symbiosis.system, but this is
265	not specific to any project. It defines how to locate critical files on the
266	given system and where to create new files as a user preference.
267
268	If we want to create a special variant of a "project", we create a new proxy
269	that points to the same source tree and which may or may not reuse an existing
270	synopsis. Multiple such proxies can be used actively in the same build, for
271	example an http server library with and without open-ssl linked in for two
272	different subsystems. (The actual local project build could also create
273	multiple variants internally for a single proxy if that makes more sense).
274
275	Symbiosis is implemented as a plugin to ocamlbuild. A build is started by
276	giving a target file name to ocamlbuild. This target encodes the desired action,
277	for example:
278
279	symbiosis myproject.proxy.synopsis.build.resp
280
281	Here we ask Symbiosis to give us the condensed result (the .resp file) of
282	performing the synopsis action named "build" for the proxy named "myproject".
283
284	The response (.resp target) triggers the build of sequence of dependent request (.req)
285	and response files until the final request file is available as the sole input the requested
286	action. The action does what it does and the creates the .resp file to flag a successful completion.
287
288	Symbiosis uses the .req / .resp files to monitor progress inside the
289	"activities/_oversight" directory. Signatures are used to track changes so an
290	action will be reexecuted unless its dependencies have changed.
291
292	Actions are performed by "agents". Agents only understand request files and a
293	request file is formatted specifically for a given agent type. Briefly, agents
294	read information from the request, performs a sequence of shell commands (such
295	as calling "make" to perform a build or "mercurial" to check out some source)
296	and then, also via the shell, produces the .resp file.
297
298	A good action will put sensible information in the .resp output, such as a
299	signature of key build products and avoid putting timestamp sensitive
300	information into the file. In this way builds are partially updated when needed.
301	A sloppy action may just copy the request to the response - at least this will ensure
302	that products later in the chain will be updated if the input to the action has changed.
303
304	In particular, there are internal dependencies to ensure a workspace is
305	checked out before a synopsis action is executed. While rebuilding source is a
306	waste of time, repeated check out of source code is a problem. Likewise, the
307	.resp files can track immaterial events such as sending an email once a build
308	product completes, but only do it once.
309
310
311	How proxies locate source trees:
312	================================
313
314	A proxy located in the "proxies" directory identifies source code by
315	identifying a branch inside a "volume".
316
317	A volume located in the "volumes" directory is a small file with information
318	to identify a suitable "agent" and with text substitution powers to format a
319	suitable checkout request for the given agent based on the the proxy file
320	contents. The volume will also collect information from the system
321	configuration and possibly the environment to provide a repository paths.
322	(Some remote repositories will not need a local path, but might still need
323	local access credentials). Any non-trivial logic beyond this will be handled
324	by the agent.
325
326	The agent will identify a suitable tool path from the system configuration and
327	do the actual job including taking the final decision on workspace location
328	based on input from the request.
329
330	If a single volume is physically located in multiple repositories, depending
331	non-trivially on branch names etc., an "intelligent" agent can be created to
332	handle this additional logic. For most cases, a volume simple drops
333	information into a request for a standard source control agent, i.e. an agent
334	that understands the "checkout" action.
335
336	The anatomy of a proxy:
337	=======================
338
339	A proxy represents a source tree stored somewhere else, typically under source control.
340
341	The only job a proxy file has, is to give enough information to identify a
342	specific version of a source tree so it can be copied into a workspace, and to
343	identify a synopsis file which has more details about what can be done with
344	the source tree.
345
346	A proxy is essentially a system independent directory entry in a global name space.
347
348	Proxy files live in the "proxies" directory specified in the system
349	configuration. The proxies file defaults "proxies" in the current directory,
350	but often we keep the proxies under source control and check them out in directory named
351
352	"activities/meta/proxies"
353
354	Two proxies named
355
356	"myproxy" and
357	"org.example/experimental/x"
358
359	respectively would then be located in
360
361	"proxies/myproxy.proxy" and
362	"proxies/org.example/experimental/x.proxy"
363
364	or
365
366	"activities/meta/proxies/myproxy.proxy" and
367	"activities/meta/proxies/org.example/experimental/x.proxy"
368
369	or anywhere else under the current directory, depending on the proxies setting.
370
371	If the "proxies" setting is not correct, Symbiosis will generate a strange
372	error about not finding a build target, simply because it can't.
373
374	The content of a proxy is quite simple, for example:
375
376	{
377	"volume": "org.example.experiments",
378	"branch": "example.ver-0.1.2"
379	}
380
381	Later we will see how volumes can point to the exact source location and
382	enable a checkout action. The volume is essentially a pointer to the logical
383	owner of the source. Both the branch and the volume are abstractions that can
384	be translated into a physical storage location. Ideally, it is possible to
385	change the storage from raw files to source control to archives etc. without
386	ever having to change the proxy.
387
388	Here is an example of a Symbiosis request target to build the source code of a proxy:
389
390	symbiosis org.example/experimental/x.proxy.synopsis.build.resp
391
392	We see the proxy name "org.example/experimental/x", the request type
393	"proxy.synopsis", and the action "build". Finally we see the target type
394	"resp" meaning we want the response, or the result of this action.
395
396	The meaning of build and how to do that, is controlled by the synopsis file.
397
398	When we issue a proxy.synopsis request, Symbiosis knows that it needs a
399	functional workspace. Therefore, it will automatially issue the following request:
400
401	symbiosis org.example/experimental/x.proxy.volume.checkout.resp
402
403	Here we have a "proxy.volume" request type. The volume identified by the proxy
404	knows what "checkout" means.
405
406
407	A proxy can have the following attributes with defaults shown:
408
409	"branch":
410	An abstract branch that a source control agent can translate into a physical branch or revision.
411	Defaults to "".
412
413	"module":
414	The name of the source component as the volume recognizes it.
415	Defaults to "". Optional if the branch is sufficient to identify
416	the component.
417
418	"name":
419	Note: This is not a settable attribute. It is the relative path from the
420	"proxies" directory to the proxy file excluding the ".proxy" extension.
421	"name" is how the component is recognized by build system, unlike the
422	"module" which is targeted the volume and it's source control agent.
423
424	"path":
425	Relative path from workspace into source tree, if this project is not
426	located at the root. Relevant when build tool needs to change into a
427	subdirectory before building.
428	Defaults to ".".
429
430	"synopsis":
431	Name of synopsis file inside workspace. Symbiosis will search for a
432	synopsis file relative to the workspace. A synopsis file can also exist
433	as a file with extension .proxy replaced by .synopsis in the proxy
434	directory. Such a file takes priority and the synopsis value
435	then has no effect.
436	Defaults to "<path>/<name>.synopsis".
437
438
439	The minimal proxy file will have the content "{}" and still make a lot of
440	sense because the proxy name can be used to check out source from a default
441	repository. However, normally it is worthwhile to add slightly more information.
442
443
444	The anatomy of a volume:
445	========================
446
447	Volume files live in the "volumes" directory which defaults to "volumes" in
448	the current directory. Typically volumes are checked out from source control into the directory:
449
450	"activities/meta/volumes"
451
452	A volume is named according to the file name relative to the volumes
453	directory, excluding the ".volume" extension. This is completely analogous to proxies.
454
455	Volumes are never referenced directory. Volumes a discovered via "volume"
456	attributes in proxy files. If proxies do not set a "volume" attribute, a
457	volume named "default" should be created, for example:
458
459	"proxies/default.volume" or "activities/meta/volumes/default.volume"
460
461	A volume file is used to create a "proxy.volume" request. One important
462	proxy.volume request is the "checkout" request.
463
464	Here is an example volume file using the "monotone" source control agent:
465
466	{
467	"agent": "monotone",
468	"arguments": { "db": "mtn/org.example.<volume>.mtn" }
469	}
470
471	A lot more settings are possible, but we rely on sensible defaults here:
472	We assume an "agent" named "monotone" has been registered with the system.
473
474	See the proxy.volume request context below for values that can be used to feed
475	the agents.
476
477	Handling proxy.volume requests:
478	===============================
479
480	Let's consider the checkout example. We create a checkout request by issuing
481	the following target:
482
483	symbiosis org.example/experimental/x.proxy.volume.checkout.resp
484
485	The "monotone" agent supports several "actions", including the "checkout" action.
486	The monotone checkout action expects the following values:
487
488	"branch", "db", "repositories", "workspaces" and "workspace".
489
490	Symbiosis recognises a proxy.volume request and automatically provides
491	default values for "branch", "workspaces" and "workspace". "db" is specific
492	to the monotone agent and provided by the volume file.
493
494	The values provided by monotone is called "context" values. All "proxy.volume"
495	request have the same context variables, but the values depends on the
496	context, hence the name.
497
498	We can create new agents as a list of actions and register the agent with
499	Symbiosis. An agent gets access to values through an eval function. For example:
500
501	eval "mtn --db <repositories>/<db> --branch <branch>
502	checkout <workspaces>/<workspace>"
503
504	This is not exactly how it works, but close enough. In this way agents format
505	shell commands and execute external tools. The details are not important here,
506	only that we get the correct values passed on.
507
508	A volume must ensure that an agent receives all the values that it requires by
509	setting the appropriate values in the arguments object.
510
511	All Symbiosis files use the JSON syntax (Java Script Object Notation - see
512	json.org). The volume can store any valid JSON syntax inside the arguments,
513	and it must set the agent to a string value. What happens in the arguments is
514	strictly between the volume an it's agent.
515
516	When a request is generated, it has the format:
517
518	{
519	"agent": "<agent-name>",
520	"action": "<action-name>",
521	"arguments": "<from a volume or synopsis file, depending on request type>"
522	"context": "from symbiosis internals, depending on request type>"
523	}
524
525	It is possible to manually create such a request as follows:
526
527	symbiosis org.example/experimental/x.proxy.volume.checkout.req
528
529	A request (".req") is generated automatically when a response (".resp") is
530	given as target to Symbiosis.
531
532
533	Values are looked up in arguments first, then context. This makes it possible
534	for a volume to override context values. Context values can be referenced like
535	"<:context:branch>" or "<branch>". By using the scope prefix, conflicts with
536	identical argument names can be avoided.
537
538	In expression expansion, the standard xml entities are supported as escape values:
539
540	< ::= &lt;
541	> ::= &gt;
542	& ::= &amp;
543	" ::= &quot;
544	' ::= &apos;
545
546	Here is an example of a mercurial volume:
547
548	{
549	"agent": "mercurial",
550	"arguments":
551	{
552	"source" : "hg/<volume>.hg",
553	"target" : "<workspace>",
554	}
555	}
556
557	Here the volume name is used to directly name the .hg directory, but this depends on
558	the local configuration.
559
560	Note that volume files a specific to a family of system configurations that
561	share the same repository layout. Proxy files are more plentiful and more
562	general. We can maintain a small set of volume files for each family of system
563	configurations we work with, but keep the same proxy files across all systems.
564
565
566	A special case for the "checkout" action:
567
568	When a proxy.volume.checkout action is executed, Symbiosis will create the
569	following directory in advance:
570
571	"<workspaces>/<workspace>/.."
572
573	If the proxy is not in a namespace, this will simply be the "workspaces"
574	directory. In any case, it is expected the the checkout action creates the
575	actual workspace directory.
576
577
578	Context of proxy.volume requests:
579	=================================
580
581	The context provide variables available in volume arguments. Volumes can
582	reference these values and the agent action receiving the request will also
583	have access.
584
585	Here current directory means the directory in which symbiosis was started. At
586	execution time the actual current directory might have changed but the paths
587	will still be relative to the original directory.
588
589
590	"branch":
591	Branch name given in proxy, or "" by default.
592
593	"family":
594	Name of active configuration family. Mostly used for checkout out meta data
595	such as a "volumes" branch that matches the local system. Copied from system
596	configuration. Defaults to "common".
597
598	"module":
599	Name of a module inside a volume. For example when a volume is organized
600	as a tree of repositories, either on local disk or on a remote server.
601	Often appended to "repositories". Defaults to "".
602
603	"name":
604	The name of the proxy without the .proxy extension, a path relative to
605	the "proxies" directory.
606
607	"path":
608	A path relative to workspace copied from proxy, defaults to ".".
609
610	"repositories":
611	Relative path from current directory, or an absolute path.
612	Copied from system configuration. Defaults to "repositories".
613	Used by source control agents as a search path for local repositories.
614	Should be terminated by '/' or other protocol specific
615	url terminator such as ':'.
616
617	"site":
618	Similar to "repositories" but is a url prefix used by some source
619	control agents, for example "http://example.org/pub/download/".
620	"site" may be set equal to "<repositories>" when a local repository
621	tree is desired. Url should be terminated like "repositories".
622
623	"synopsis":
624	Relative path from <workspace> to synopsis file, defaults to
625	"<path>/<name>.synopsis".
626
627	"volume":
628	Name of the volume without the .volume extension, a path relative to
629	the "volumes" directory.
630
631	"workspace":
632	Path relative to "workspaces". Always set to "<name>" so a volume can
633	override this value if needed in the volume arguments.
634
635	"workspaces":
636	Relative path to workspaces from the current directory.
637
638	The above attributes can also be accessed using the ":context" scope, for
639	example "<:context.branch>". This is useful when a request argument has the
640	same name since the argument scope is default when evaluating request
641	expressions.
642
643
644	Attributes part of a request, but not strictly part of the context:
645
646	":agent":
647	Name of requested agent, for example "mercurial".
648
649	":action":
650	Name of requested action, for example "checkout".
651
652
653	The anatomy of a synopsis file:
654	===============================
655
656	A synopsis file is a list of actions that can be applied to a source tree that has
657	been checked out.
658
659	The process works much like volume files, but here agents operate on
660	workspaces, not repositories.
661
662	For example example:
663
664	content of experiments/gw.synopsis:
665
666	{
667	"agent": "build",
668	"arguments": { "tool": "make", "target": "all" },
669	"actions": [ "build" ]
670	}
671
672	The arguments are, as usual, specific to the given agent, and actually also
673	specific to the action.
674
675	In this example, the built-in build agent expects the argument "tool" to use
676	as a build tool. It also expects a target argument. Thus, when a request is
677	made to the "build" agent, it will first read the tool argument, then call
678	the Symbiosis "tool" function on the argument to locate the tool in the system
679	configuration (see "tools" configuration in system configuration anatomy
680	below). If the tool is found (i.e. it is installed on the system), the shell
681	command is created.
682
683	Assume that tool simply translated "make" to "make", we get the
684	following shell action executed:
685
686	make all
687
688	Actually, this is not entirely correct. We get the following command:
689
690	cd .../workspaces/experiments/gw && make all
691
692	The agent ensures that we change path to the current workspace before the
693	command is executed. For many build tools this is essential.
694
695	Actually, this is also not entirely correct. We really get the following command:
696
697	cd .../workspaces/experiments/gw && make all;
698	cp .../proxies/experiments/gw.proxy.synopsis.build.req
699	.../proxies/experiments/gw.proxy.synopsis.build.resp
700
701	The agent also copies the input request to the output to ensure some progress
702	in the symbiosis build system.
703
704
705	The agent and arguments in the .synopsis file are default values.
706
707	Each action can have its own agent and arguments, for example:
708
709	{
710	...
711	"actions":
712	[
713	{
714	"action": "build",
715	"agent": "build",
716	"arguments": { "tool": "ocamlbuild", "target": "all" }
717	"dependencies": [ {"action" : "test-build", "name": "experiment/k1"} ]
718	} ,
719	{
720	"action": "document",
721	"agent": "build",
722	"arguments": { "tool": "ocamlbuild", "target": "doc" },
723	"dependencies": [ "doc-tool" ]
724	},
725	"clean"
726	}
727	]
728	...
729	}
730
731	Notice that we can have both simple actions as strings (here "clean"), and
732	action objects (like "build"). The "clean" action could have been written
733	{"action": "clean"} instead of "clean" in the above example.
734
735	When an action object is missing values, like "agent" or "arguments", the
736	values are derived from the top-level agent and arguments. It is either one or
737	the other, arguments are not taken from both places.
738
739	In the example we also see actions can have dependencies. Again, they can be
740	simple strings where "doc-tool", { "name": "doc-tool" } and { "name":
741	"doc-tool", "action": "document"} are equivalent because of the default mechanism.
742
743	Any action that does not set dependencies will inherit dependencies from the
744	top-level dependencies, similar to how agent and arguments are inherited.
745
746	If an action does not specify a name, it is a self reference. Here we require
747	a depend action to be performed before the build action on the same project:
748
749	...
750	"actions": [{ "action": "build", "dependencies": [ {"action": "depend"} ] }]
751	...
752
753	Dependencies can be simplified using "name:action", "name" or ":action" as a
754	string value in dependencies. Internally this is mapped to an object with name
755	and action fields and defaults taken from curren action as usual, for example:
756
757	...
758	"actions": [{ "action": "build", "dependencies": [
759	":depend", "doc-tool:install", "templates" ] }]
760	...
761
762	This could also be written:
763
764	...
765	"actions": [{ "action": "build", "dependencies": [
766	{"action": "depend"}, {"name": "doc-tool", "action": "install"}, {"name": "templates"} ] }]
767	...
768
769	TODO: It would be possibly to add more information to a dependency, such as build
770	tags, but this has not been implemented.
771
772	Dependency name spaces:
773	=======================
774
775	A dependency is a proxy name without the ".proxy" extension. Proxies live in
776	name spaces as defined by the path relative to the "proxies" directory.
777
778	When one proxy depends upon another, they are assumed to exist in the same
779	name space. As an example, consider the proxy:
780
781	"org.example/experiments/blast"
782
783	The corresponding synopsis file appointed by the proxy lists "ignition" as a
784	dependency. This implies that the proxy:
785
786	"org.example/experiments/ignition"
787
788	must be valid.
789
790	It does not matter what the .synopsis file is named. Several proxies can point
791	to the same synopsis file from different name spaces. In all cases the
792	dependency is relative to the appointing proxy.
793
794	A dependency can also be absolute: "/org.non.gov/eco/parabolic/sun" in which
795	case the dependent proxy does not impact the name space.
796
797	A dependency can also use relative paths similar to file paths:
798
799	The dependency:
800
801	"../live/ignition"
802
803	for the proxy
804
805	"org.example/experiments/blast"
806
807	will then become:
808
809	"org.example/live/ignition"
810
811	It is a bad idea to use relative paths that reaches outside the "proxies"
812	root. Symbiosis will complain.
813
814	By using relative name spaces, it is possible to move proxy directory trees
815	without too much pain. For example when checking out a sub-tree of proxies
816	into a different sub-directory using the the meta-volumes feature in the
817	system configuration (see below).
818
819
820	Exporting values to synopsis actions:
821	=====================================
822
823	When one project needs content from another product, this is communicated
824	through exports and dependencies. Dependencies have already been mentioned.
825
826	A synopsis file may optionally contain "exports" either in the common section
827	or specific to an action. As usual the common section is used as a default
828	when "exports" are missing in a given action.
829
830	As usual: export values are taken either from action or common, they are not
831	merged.
832
833	example content of a synopsis file:
834
835	{
836	"agent": "build",
837	"arguments": { ... },
838	"exports": {
839	"lib": "<project>/_build/lib",
840	"includes": "<project>/src/includes"
841	},
842	"actions": [
843	"build",
844	{"agent": "docbuilder", "action": "doc", "exports": {"doc": "<project>/doc"}}
845	]
846	}
847
848	When an action is requested, Symbiosis will copy the export values relevant to
849	that action into the context of the corresponding proxy.synopsis request message.
850	This message will also contain information such as "project".
851
852	example content of a proxy.request message for the "doc" action of the above synopsis:
853
854	{
855	"agent": "docbuilder",
856	"action": "doc",
857	"arguments": { ... },
858	"contents": [
859	...
860	"exports": { "doc": "<project>/doc" },
861	...
862	"project": "activities/_work/myproject"
863	]
864	}
865
866	Any exported value can reference context values of the action request it
867	relates to. It is useful to prefix paths with "<project>" because a dependent
868	project will receive the expanded project path so it can find the full path
869	relative to the current directory.
870
871	Later on, another project depends on the doc action of the above project.
872
873	Symbiosis will generate a request rather similar to the above request, but for
874	some other action. This request may also contain exports for other
875	dependencies further up the food chain.
876
877	However, in this case we are interested in the dependencies, not further exports.
878
879	The synopsis will list dependencies, as have been documented elsewhere. This
880	will cause Symbiosis to perform the dependent actions including generating the
881	action request messages. Symbiosis will read the export section these
882	requests, evaluate the export expressions locally to each of these requests
883	(to expand "<project>" etc.), and insert the exported values in to the
884	context of the new request.
885
886	The collected and expanded export values are stored in the request context in the
887	"dependencies" value.
888
889	content of an proxy.synopsis action request with dependency exports:
890
891	{
892	...
893	"context": {
894	"dependencies": {
895	"myproject": { "doc" : "activities/_work/myproject/doc"},
896	"exports": { "export-name": "export-value"},
897	"project": "activities/_work/thisproject",
898	"base": "../../..",
899	...
900	},
901	}
902	}
903
904	The agent action for the above request can now locate the value "doc" exported
905	from "myproject":
906
907	"<dependencies:myproject:doc>"
908
909	The dependencies is inserted to the scope of variable expansions with priority below
910	the context and argument scope. Thus, we could also write:
911
912	"<myproject:doc>"
913
914	skipping the dependencies scope prefix.
915
916	If "myproject" was not "myproject", but some name spaced name like
917
918	"org.example/experiment/k2",
919
920	the expression would be:
921
922	"<dependencies:org.example/experiment/k2:doc>"
923
924	or
925
926	"<org.example.experiment/k2:doc>"
927
928	The agent can use exported values as command line options to the action, as an
929	example.
930
931	The agent could also read all the "dependencies" values and write them to a
932	name/value file that can be easily included in Make files or Ruby scripts etc., or
933	it could export the values to the environment before executing the action.
934
935	Symbiosis does not really care what the agent does with the exported values,
936	it only ensures that the appropriate values are made available to the agent.
937
938
939	Values may or may not refer to paths. This is why <project> should be used
940	for local paths. Symbiosis would not know which values to path translate.
941
942	Notice the value "base" in the context. This is the "inverse" project path. If
943	an action executes inside the project path (not all actions do), the base path
944	can be prefixed exported values:
945
946	"<base>/<dependencies:myproject:doc>"
947
948	Now we get a project relative path to some other project. In our example,
949	the above expression expands to:
950
951	"../../../activities/_work/myproject/doc"
952
953	If an agent blindly copies every exported value into some local build file,
954	it cannot insert the base value without knowing what values are paths.
955	A solution to this problem could be to also copy the base value and let
956	the local build handle the remaining path translations.
957
958	Note: There is a conflict when two actions from the same project both export
959	values to the same dependency. In this case the same dependency is listed
960	twice and the first occurrence will hide the second. Not likely to be a huge
961	problem though.
962
963	Exported values can also be accessed using "*" so we can refer to the exported
964	value without caring about the name of the dependency, for example:
965
966	"<base>/<dependencies:*:doc>"
967
968	instead of
969
970	"<base>/<dependencies:myproject:doc>"
971
972	However, if there are conflicts, such as multiple doc values, only the first
973	occurrence will be used. "*" can also be used to access values otherwise hidden
974	because two dependencies have the same name with different actions.
975
976
977	Writing an environment file:
978	============================
979
980	The built in agent "build" has an optional "echo" argument - here the arguments
981	for the build agent in some .synopsis file:
982
983	...
984	"agent": "build",
985	"actions": "build",
986	"arguments": {
987	"tool": "make",
988	"target": "all",
989	"echo": {
990	"contents": "name=<:name>\nlibs=<base>/<dependencies:myproject:lib>;<base>/<dependencies:otherproject:lib>",
991	"file": "<project>/_environment"
992	}
993	}
994
995	Or, taking advantage of dependencies being in the scope:
996
997	...
998	"agent": "build",
999	"actions": "build",
1000	"arguments": {
1001	"tool": "make",
1002	"target": "all",
1003	"echo": {
1004	"contents": "name=<:name>\nlibs=<base>/<myproject:lib>;<base>/<otherproject:lib>",
1005	"file": "<project>/_environment"
1006	}
1007	}
1008
1009
1010
1011	If the echo argument is present, the content and file argument must also be present.
1012
1013	The echo argument makes it possible to create customized environment files, for example name / value pairs,
1014	quotes name / value pairs, or JSON objects.
1015
1016	The filename is relative to the current directory (where symbiosis was stored) and should point to an existing
1017	directory. The file may or may not exist and is only considered relevant for the duration of the action.
1018
1019	We can also write the content as an array, since array's are converted to strings by appending a newline to
1020	each array element:
1021
1022	"arguments": {
1023	"tool": "make",
1024	"target": "all",
1025	"echo": {
1026	"contents": [
1027	"name=<:name>"
1028	"libs=<base>/<myproject:lib>;<base>/<otherproject:lib>"
1029	],
1030	"file": "<project>/_environment"
1031	}
1032	}
1033
1034
1035	Or, if we are happy with basic name=value\n syntax, we can simply create an object:
1036
1037	"arguments": {
1038	"tool": "make",
1039	"target": "all",
1040	"echo": {
1041	"contents": {
1042	"name": "<:name>"
1043	"libs": "<base>/<myproject:lib>;<base>/<otherproject:lib>"
1044	},
1045	"file": "<project>/_environment"
1046	}
1047	}
1048
1049	The content value is treated like any other value which is evaluated and
1050	converted to string. The array and object syntax behaves the same as any other
1051	value being evaluated.
1052
1053	In our example, we can now include "_environment" into the Makefile. We could
1054	also have source the file as a shell script, or changed the syntax and loaded
1055	the file as yaml in Ruby or Python for example.
1056
1057
1058	See also "Exporting values to synopsis actions".
1059
1060
1061	Handling proxy.synopsis requests:
1062	=================================
1063
1064	At proxy.synopsis request will receive the following runtime :context data
1065	when Symbiosis creates the request message, as an example:
1066
1067	"context":
1068	{
1069	"name", "experiments/gw"
1070	"workspace": "experiments/gw",
1071	"workspaces": "activities/_work/",
1072	"path": "."
1073	}
1074
1075	The context has more data than this, see the full list below. The context does
1076	not depend on the agent or the action only on the request type, the proxy, the
1077	volume and the system configuration.
1078
1079	The synopsis file is responsible for providing all relevant arguments to the
1080	given agent and action.
1081
1082	The workspace path is relative to where symbiosis was started, what
1083	we normally refer to as "relative to current directory". However, at runtime
1084	the current directory has changed to somewhere inside "activities/_oversight"
1085	according to ocamlbuild mechanics and symbios.system configuration. Hence it
1086	will be a slightly different relative path. This is only important for agent
1087	implementers, and there are helper functions to deal with this path
1088	translation. That being so, it is quite tricky for agents to get all paths
1089	exactly right. Fortunately this is not normally a concern when writing
1090	synopsis files.
1091
1092	Reaching outside the workspace during build is a big no-no. Even if we have
1093	dependencies (see below), we cannot guess the workspace location of these
1094	dependencies, or assume they will remain stable. For example, an indirect
1095	dependency to a build step might be skipped and replaced by a cache pull
1096	causing entire workspaces to never be created. Special brokers are available
1097	for this kind of work.
1098
1099	TODO: no such brokers are available nor fully designed at this point, but they
1100	are clearly needed. Actually - maybe official brokers are not needed: agents
1101	can access exported values in their action requests, and this could be
1102	sufficient. Beyond this, the agent could call upon other tools to read the
1103	request file, or generate a simpler name/value list for local builds to
1104	consume. In this way Symbiosis need not become too involved in local build
1105	processes. Still, a broker concept could make sense, but it would be further
1106	down the tool chain.
1107
1108
1109	Context of proxy.synopsis requests:
1110	===================================
1111
1112	"base":
1113	The inverse path of "projects", for example "../../../.." if projects
1114	is "activities/workspaces/myworkspace/mypath".
1115
1116	"dependencies":
1117	An object containing exported values of each dependency.
1118	Dependencies are derived from common or action specific dependencies in synopsis.
1119	For example:
1120
1121	{
1122	...
1123	"context": {
1124	"dependencies": { "mydependency": { "mydependency-export-value": "exported-value"}}
1125	"exports": { "myown-export-value": "some-value"}
1126	}
1127	...
1128	}
1129
1130	"exports":
1131	Derived from synopsis. Can contain expressions like
1132	"<project>/localpath". Evaluated in local context before being copied into
1133	"dependencies" of dependent action requests.
1134
1135	"name": Name of proxy, a path relative to the "proxies" directory without the
1136	".proxy" extension.
1137
1138	"path":
1139	Relative path from the workspace to project source. Copied from the
1140	proxy path attribute. Defaults to ".".
1141
1142	"project":
1143	Like path, but relative to the current directory. Similar to workspaces / workspace / path.
1144
1145	"workspace":
1146	Relative path from "workspaces" to the project workspace. Defaults to "<name>".
1147
1148	"workspaces": Relative path from current directory to workspaces.
1149
1150
1151
1152	Attributes part of a request, but not strictly part of the context:
1153
1154	":agent":
1155	Name of requested agent, for example "build".
1156
1157	":action":
1158	Name of requested action, for example "build".
1159
1160
1161	The anatomy of a system configuration:
1162	======================================
1163
1164	Symbiosis uses an optional system configuration file named symbiosis.system to
1165	locate paths to proxies, volumes, repositories etc.
1166
1167	Symbiosis will search the symbiosis.system file in two places:
1168
1169	./symbiosis.system
1170
1171	and
1172
1173	<env:HOME>/.symbiosis/symbiosis.system
1174	(env: is the Symbiosis scope prefix for shell environment variables).
1175
1176	in that order.
1177
1178	The settings are general as opposed to project specific. In fact, multiple
1179	hosts can often use the same configuration when they share path conventions.
1180
1181	Here is the default file (and default values for a user specified file)
1182
1183	{
1184	"configuration":
1185	{
1186	"activities": "activities",
1187	"artefacts": "<activities>/artefacts"
1188	"family": "common",
1189	"metaspaces": "<activities>/meta"
1190	"oversight": "<activities>/_oversight",
1191	"proxies": "proxies",
1192	"repositories": "repositories",
1193	"site": "<repositories>",
1194	"tools": "<activities>/tools",
1195	"volumes": "volumes",
1196	"workspaces": "<activities>/_work"
1197	},
1198	"meta-volumes": {},
1199	"tools" : {},
1200	"force-tools": false
1201	}
1202
1203	Note that :tools discussed below and :configuration:tools above are different
1204	settings. The configuration specifies a tools directory, and :tools lists
1205	known tools in the configuration file, possibly renaming tools, like cc ->
1206	gcc.
1207
1208	All paths above are absolute or relative to the current directory, that is,
1209	where symbiosis was started.
1210
1211	To override an installed configuration file in .symbiosis/symbiosis.system:
1212
1213	echo "{}" > symbiosis.system
1214
1215	in the working directory - now we get an empty configuration with pure
1216	defaults.
1217
1218
1219	Here follows a description of the ":configuration" settings:
1220
1221
1222	":configuration.activities":
1223
1224	Relative to current path. Has not significance in itself; it is only used as
1225	path prefix for other paths. Default root path for other dynamic content
1226	such as "oversight" and "workspaces". Also default root for new types of
1227	output that new agents might define, for example "distributions" or
1228	"publish". Other useful values:
1229	"activities": "."
1230	"activities": "myproject"
1231
1232
1233	":configuration.artefacts":
1234
1235	A directory for installing build products. This directory is automatically
1236	created by Symbiosis if it does not exists, typically in
1237	"<activities>/artefacts". It is up to individual agent actions to make any
1238	further use of this location.
1239
1240	Artefacts can for example be used to install libraries that will survive
1241	after a local workspace has been deleted, or severely modified during
1242	development, without breaking dependent builds. In this case the
1243	location would typically be exported by the installing action.
1244
1245	Please observe that American English spelling is "artifacts", but here we
1246	use the British form "artefacts".
1247
1248
1249	":configuration.family":
1250
1251	Name of a group of host systems that have compatible configurations. Mainly
1252	used in source control branch names to pull the correct configuration.
1253	Family may also be a single host name which could then later form the family
1254	name for child systems, that is, other hosts that copy the configuration of
1255	the original host.
1256	See also "meta-volumes" where "family" is likely to be used.
1257
1258
1259	":configuration.oversight":
1260
1261	Relative to current path, but is typically prefixed with "<activities>.
1262	Where all bookkeeping is stored. This is where the ocamlbuild engine
1263	operates. The path should be prefixed with an underscore '_' to prevent
1264	ocamlbuild from searching source files here. Likewise other paths should not
1265	be prefixed with an underscore since this may cause files to become
1266	invisible during build (files like .proxy, .volume and .synopis). ocamlbuild
1267	will create ./_build to build the symbiosis program, then proceed to operate
1268	inside the "oversight" directory. If symbiosis is pre-built and installed,
1269	the ./_build directory is not created.
1270
1271
1272	":configuration.proxies":
1273
1274	Relative to current path.
1275	Search path for <**/*.proxy> files.
1276	An alternative typical setting is
1277
1278	"proxies": "<workspaces>/proxies"
1279	(the exact path then depends on the checkout - see meta-volumes)
1280
1281	when proxies are checked out from source control.
1282
1283	"proxies": "<repositories>/proxies"
1284
1285	when proxies are stored near the source they represent
1286
1287	":configuration.repositories":
1288	The default location of local repositories.
1289
1290	":configuration.site":
1291	The default place for repository url's.
1292
1293	":configuration.tools":
1294
1295	A directory created by Symbiosis if it does not already exist; typically
1296	under <activities/tools>. The tools path is added to the executable path
1297	so any local build will find tools here. The agent "tool" function will
1298	look up tools here (if the :tool lookup fails (see :tool below)).
1299
1300	The experimental built-in "tools" agent has a simple "install" action that
1301	will copy a file from the <project> directory to the <tools> directory. More
1302	advanced agents can use a similar approach to make multi-file tools
1303	available in the executable path.
1304
1305
1306	":configuration.volumes":
1307
1308	Relative to current path.
1309	Search path for <**/*.volume> files.
1310	An alternative typical setting is
1311
1312	"volumes": "<workspaces>/volumes"
1313	(the exact path then depends on the checkout - see meta-volumes)
1314
1315	when volumes are checked out from source control, and
1316
1317	"volumes": "<repositories>/volumes"
1318
1319	when volumes are stored near the repository files they represent
1320
1321
1322	":configuration.workspaces":
1323
1324	Relative to current path. Repositoriesare checked out here, typically with
1325	the same path as proxy names. Optionally use an underscore as first
1326	character in the name to prevent ocamlbuild from wasting time indexing
1327	source code that is not relevant. In the future it could potentially be
1328	dangerous if ocamlbuild confused other local ocamlbuild projects with its
1329	own. For now, it seems to work either way.
1330	Defaults to "<activities>/_work".
1331
1332
1333	":configuration.metaspaces":
1334
1335	The path where meta-volumes are checked out, similar to workspaces.
1336	Metaspaces should _not_ have an underscore in its name since this prevents
1337	ocamlbuild from depending on files in this directory and will break proxy
1338	search if proxies are located here (according to "proxies").
1339	Defaults to "<activities>/meta".
1340
1341
1342	This ends the list of configuration variables.
1343
1344	The following list are top-level configuration settings:
1345
1346
1347	":tools":
1348
1349	Example:
1350
1351	"tools":
1352	{
1353	"make": "gmake",
1354	"mtn": "/usr/local/monotone-0.38",
1355	"ocamlbuild": "ocamlbuild",
1356	"cc": "gcc"
1357	}
1358
1359	We use this both to document required tools, and to use the proper tool
1360	version. This only covers tools that agents specifically ask for through the
1361	"tool" function and is therefore not an exhaustive list of required build
1362	tools. If no agent tool checking agent action is used, this setting is
1363	optional.
1364
1365	The tools function used by agents will also look in the :configuration:tools
1366	directory if the tools was not listed here.
1367
1368	See also "force-tools".
1369
1370
1371	":force-tools":
1372
1373	If "force-tools" is true, a tool looked up by an agent using the "tool"
1374	function must exist in the tools listing or an error is issued.
1375
1376	If "force-tools" is false and the tool was not found in "tools", the input
1377	value is returned unmodified, assuming the tool is somewhere in the
1378	executable path.
1379
1380	If the tools is present in the :configuration:tools directory, the tool
1381	function will not fail even if force-tools is true.
1382
1383	See also ":tools".
1384
1385
1386	":targets":
1387
1388	Adds user-friendly targets. This simplifying building complex target
1389	requests, and also helps remembering the most important projects among the
1390	potentially many available.
1391
1392	The targets will not be automatically executed automatically or grouped,
1393	they are simply aliases. For example:
1394
1395	"targets" :
1396	{
1397	"heat" : "org.example.environmental/warming.proxy.synopsis.build.resp",
1398	"co-data" : "mydata.proxy.volume.checkout.resp"
1399	}
1400
1401	Now we can type
1402
1403	"symbiosis heat" and "symbiosis co-data"
1404
1405	or simply
1406
1407	"symbiosis heat co-data"
1408
1409	instead of
1410
1411	"symbiosis org.example.environmental/warming.proxy.synopsis.build.resp" and
1412	"symbiosis mydata.proxy.volume.checkout.resp".
1413
1414
1415	":meta-volumes":
1416
1417	A list of internal volumes with a format similar to .volume files. The
1418	"symbiosis init" command will check out all volumes listed in meta-volumes.
1419
1420	Meta-volumes are used to check out repositories that are required before
1421	the rest of build process can progress. In particular volumes and proxies.
1422
1423	There are no meaningful proxy for a meta-volume. The "<name>" is set to
1424	the volume name.
1425
1426
1427	Meta-volumes are not in the volume search path, they are only intended
1428	to bootstrap access to other volume and proxy files. Meta volumes are
1429	invisible to proxies.
1430
1431	Example:
1432	{
1433	"force-tools": true,
1434	"tools": {"mtn": "mtn", "ocamlbuild": "ocamlbuild"},
1435
1436	"configuration":
1437	{
1438	"repositories": "<env:HOME>/repositories",
1439	"family": "server-cluster-K",
1440	"proxies": "<metaspaces>/proxies",
1441	"volumes": "<metaspaces>/volumes"
1442	},
1443	"meta-volumes":
1444	{
1445	"proxies":
1446	{
1447	"agent": "monotone",
1448	"arguments":
1449	{
1450	"db": "mtn/symbiosis.mtn"},
1451	"branch": "org.example.symbiosis.proxies",
1452	"workspace": "<volume>"
1453	}
1454	},
1455	"volumes":
1456	{
1457	"agent": "monotone",
1458	"arguments":
1459	{
1460	"db": "mtn/symbiosis.mtn",
1461	"branch": "org.example.symbiosis.family.<family>.volumes",
1462	"workspace": "<volume>"
1463	}
1464	},
1465	"system":
1466	{
1467	"agent": "monotone",
1468	"arguments":
1469	{
1470	"db": "mtn/symbiosis.mtn",
1471	"branch": "org.example.symbiosis.family.<family>.system",
1472	"workspace": "<volume>"
1473	}
1474	}
1475	}
1476	}
1477
1478
1479	Notice the family branch convention. This allows us to share configurations
1480	across multiple servers by keeping .volume files and .system files under a
1481	family specific branch while proxies are global to the organisation.
1482
1483	The "system" volume has no direct significance, it will simply check out a
1484	copy of our current configuration. However, this makes it easy for use to
1485	make changes copy the config file into the workspace/system directory and
1486	check in the change.
1487
1488	With the proper content already checked into the symbiosis.mtn repository,
1489	all a new user needs to do is to copy the symbiosis.mtn file to
1490	~/repositories/mtn/symbiosis.mtn, then system configuration file to
1491	~/.symbiosis/symbiosis.system. And finally install the symbiosis binary into
1492	the current path like /usr/local/bin or ~/bin. Then execute "symbiosis init"
1493	in any suitable directory to bootstrap the proxies and volumes files.
1494
1495	If a new volume is added to meta-volumes, "symbiosis init" will discover
1496	the new repositories and leave existing volumes unmodified - even if
1497	arguments were changed. The activities/workspaces/<myvolume> workspace and
1498	the corresponding activities/_oversight/meta-volume.<myvolume>.* files
1499	can be removed manually.
1500
1501
1502	("_build"):
1503
1504	Relative to current path. "_build" is a directory created by ocamlbuild when
1505	building symbiosis from source, but it is not a configuration setting. See
1506	"oversight". "_build" is not created when running the symbiosis binary - See
1507	INSTALL.
1508
1509
1510	Multiple proxy volumes and name spaces:
1511	=======================================
1512
1513	If we have a complex configuration with proxies from many indenpendent or
1514	loosely coupled project teams as well as external sources, we may want to
1515	maintain multiple proxy repositories and assign name spaces.
1516
1517	Namespaces are created by adding directories inside the "proxies" home
1518	directory. A simple way to include multiple proxy volumes is to add these in
1519	the meta-volumes configuration (see "meta-volumes" above):
1520
1521	...
1522	"meta-volumes":
1523	{
1524	"mozilla":
1525	{
1526	"agent": "monotone",
1527	"arguments":
1528	{
1529	"db": "mtn/symbiosis.mtn"},
1530	"branch": "org.example.symbiosis.proxies.<volume>",
1531	"workspace": "proxies/<volume>"
1532	},
1533	},
1534	"external":
1535	{
1536	"agent": "monotone",
1537	"arguments":
1538	{
1539	"db": "mtn/symbiosis.mtn"},
1540	"branch": "org.example.symbiosis.proxies.<volume>",
1541	"workspace": "proxies/<volume>"
1542	},
1543	},
1544	...
1545
1546	}
1547	...
1548
1549	In this way we can import proxy volumes from external providers - here
1550	assuming the Mozilla project chose to publish proxies, or alternatively that
1551	we chose to create them and organise them separately.
1552
1553	Using this approach, we can have extremely large code bases where even the
1554	number of proxies without source code would still be a large checkout. We just
1555	need select the name spaces that we actually care about. If we include to
1556	little, this will not affect the build in any other way than to report a
1557	missing proxy, which is then easy to act upon by updating the system
1558	configuration.
1559
1560	We have to be careful with dependencies when we start to rearrange name spaces
1561	because proxies will change names and thus break references.
1562
1563
1564	The anatomy of an agent:
1565	========================
1566
1567	An agent is a list of actions that all takes a request as input and produces a
1568	response. The request is a file. The request file is already processed when
1569	given to the agent. Thus arguments can be accessed by evaluating strings like
1570	"<target>" or "<target>.<name>"
1571
1572	The agent _must_ create the response file given to it as input, otherwise it
1573	will terminate the build with an error. For special cases this might be
1574	useful, but it is not the intended operation. If the response file is a
1575	signature of the build products, this will help Symbiosis and the underlying
1576	ocamlbuild engine to build incrementally with best possible precision.
1577
1578	The response file should not contain time sensitive information, unless we
1579	really want to force rebuilds - like adding time in hours ti do a
1580	rebuild every hour, otherwise build incrementally.
1581
1582	An agent is created as an ocaml module in the myocamlbuild.ml file which is
1583	part of the source code for Symbiosis. Ocamlbuild will automatically build the
1584	agents in this file. Using this approach requires some knowledge of ocamlbuild
1585	plugins, but it is not difficult. See monotone example in source distribution.
1586
1587	The Symbiosis.agent module provide path relative functions and workspace
1588	access functions. This module is located in ocamlbuild_config.ml.
1589
1590	The agent is registered with an agent name and a list of name, action pairs
1591	where the action is a function the receives an eval function and a request
1592	file name and a response file name. The eval function expands strings like
1593	"<name>", rather similar to the ocamlbuild 'env' function, although it is a
1594	completely different underlying mechanism.
1595
1596	Here is the build in "build" agent, it only has one action, but agents can have
1597	any number of actions:
1598
1599	./myocamlbuild.ml
1600
1601	open Myocamlbuild_config
1602
1603	module Monotone = struct
1604	open Symbiosis.Agent
1605
1606	let mtn x = [A (tool "mtn"); S x]
1607	let rev ws resp = respond_cmd_in ws resp (mtn [A "automate"; A "get_current_revision_id"])
1608	let ws eval = subdir (eval "<workspaces>") (eval "<workspace>")
1609	let wsdb eval = ws eval, eval "<repositories>/<db>"
1610
1611	(* "relcwd path" ensures that we get a path relative to the current working directory
1612	instead of a path relative to the site. The current working directory is normally
1613	inside the symbiosis activities/_oversight directory. *)
1614
1615	let checkout_latest msg eval req resp =
1616	let ws, db = wsdb eval in
1617	Seq [
1618	cmd (mtn [A "--db"; A db; A "--branch"; A (eval "<branch>"); A "checkout"; Px ws]);
1619	rev ws resp]
1620	end (* end of abridged Monotone module *)
1621
1622	module Build = struct
1623	open Symbiosis.Agent
1624
1625	let build msg eval req resp =
1626	Seq [cmd_in (eval "<project>") [A (tool (eval "<tool>")); A (eval "<target>")]; cp req resp]
1627
1628	let agent =
1629	"build", ["build", build]
1630	end (* end of Build module *)
1631
1632	(* register agents *)
1633	let () = Symbiosis.initialize [Monotone.agent; Build.agent]
1634
1635
1636	The agent also receiveds the msg argument. The eval function already
1637	understands how to access the msg structure, but msg can be used with additional support functions such as
1638
1639	Agent.has_argument msg "arg-name"
1640
1641	in order to handle optional arguments. Evaluating a missing argument directly would result in an error.
1642
1643	Agent.has_variable msg "var-name"
1644
1645	is similar to has_arguments but covers a wider scope, for example "name" is
1646	not an argument, but it is a variable available in the message evaluation scope.
1647
1648	The eval function also support "*" patterns. For example:
1649
1650	eval "<dependencies:*:zlib>"
1651
1652	or just
1653
1654	eval "<*:zlib>"
1655
1656
1657	Suppose there are more ways to create the library zlib using different
1658	proxies, we can use '*' to pick the exported value zlib. "*" should be used
1659	carefully though, since it might match the wrong value, and could result in
1660	circular references.
1661
1662	Built-in Agents:
1663	================
1664
1665	This is experimental and subject to change.
1666
1667	"tool" agent:
1668
1669	Only supports the "install" action and requires the argument "file". The
1670	file is expected relative to the project directory and the action will copy
1671	the file to the tool directory. A tool can be built and installed using some
1672	convenient build action and another install action that depends on the build
1673	action. The install action would use the tool agent to install the tool. The
1674	"tools" directory (activities/tools) is in the executable path of all
1675	actions.
1676
1677	Tools can also install themselves into the tools path, and this action
1678	could easily be implemented using the shell agent - see below.
1679	Thus, this actin is not strictly necessary.
1680
1681	"shell" agent:
1682
1683	Shell supports the _missing_ action and hence any action.
1684	Arguments (by example)
1685	"arguments": {
1686	"dir": "<project>"
1687	"script": [
1688	["#", "this is a comment"],
1689	["mkdir_p", "testdir"]
1690	["cp", "<base>/<dependencies:*:somefile>", "testdir/"]]
1691	}
1692
1693	The dir argument is optional and defaults to current directory where we
1694	started, or <base>. Currenlty "cp" and "mv" and handled internally by
1695	ocamlbuild, while other arguments are passed through the "tool" function. If
1696	force_tool is true in the system configuration, tools must be listed there, or
1697	present in the "tools" directory.
1698
1699	mkdir_p is not supported internally, but we could create a tool in
1700	"tools/mkdir_p" to provide a cross platform mkdir_p utility and create a
1701	dependency on this tool.
1702
1703	"build" agent:
1704
1705	The build agent only has the 'build' action. The action expects two arguments
1706	"tool" and "target" an will execute tool with the actin target. For example:
1707
1708	"arguments": { "tool": "make", "target": "all" }
1709
1710	or
1711
1712	"arguments": { "tool": "make", "targets": ["doc", "index"] }
1713
1714	The tool argument is translated and verified with the "tool" function, thus
1715	we could use "cc" as a build action to compile simple c programs and have the
1716	tool funciton translate cc to gcc, cl, cc or whatever the platform compiler is
1717	named.
1718
1719	The "target" and "targets" arguments are optional. If both are present, target
1720	becomes the first argument to the tool. If none are present, the tool receives
1721	no arguments.
1722
1723	The build action optionally accepts the additional "<echo>" argument. <echo>
1724	generates a file before the build action is executed. This can be used to
1725	export environment variables to the build script, or even to create a build
1726	script or Makefile containing environment variables.
1727
1728	If <echo> is present, it _must_ be an object with at least the following values:
1729
1730	<echo:file> the filename relative to <base>.
1731	<echo:contents> an array or an object.
1732	If contents is an object, each value is exported as "name=value" on a separate
1733	line. As an array, each element is exported as a separate line.
1734
1735	For example:
1736
1737	"arguments": { "tool": "make", "target": "all",
1738	"echo": { "file": "_environment",
1739	"contents": {
1740	"base": "<base>", "libs": "<*:zlib>;<*:mylib>"
1741	}
1742	}
1743	}
1744
1745	or
1746
1747	"arguments": { "tool": "make", "target": "all",
1748	"echo": { "file": "_environment",
1749	"contents": [
1750	"base = '<base>'", "libs = '<*:zlib>', '<*:mylib>'"
1751	]
1752	}
1753	}
1754
1755	Thus making the format controllable.
1756
1757
1758	"monotone" agent:
1759
1760	A source control agent for creating workspaces from proxies and volumes.
1761
1762	Supports the "checkout" action and untested the "checkout-revision" action.
1763	Intended for "proxy.volume" requests.
1764
1765	"checkout" action requires the "<db>" argument and uses
1766	"<repositories>", "<workspaces>", "<branch>" and "<workspace>" arguments from the request context.
1767
1768	The experimental "checkout-revision" requires the "<revision>" argument, but does not use the
1769	<branch> argument.
1770
1771
1772	Symbiosis source code:
1773	======================
1774
1775	Two files:
1776
1777	myocamlbuild_config.ml
1778	myocamlbuild.ml
1779
1780	myocamlbuild_config.ml is the symbiosis implementation.
1781	myocamlbuild.ml is for agents and other possible extensions.
1782
1783	(Ocamlbuild hardcodes these filenames in its current version).
1784
1785
1786	BUILD:
1787	======
1788
1789	Put files in empty directory, then execute:
1790
1791	ocamlbuild
1792	(pre-installed with OCaml 3.10.1 distribution)
1793
1794	then pick up the result and rename it to symbiosis:
1795
1796	cp _build/myocamlbuild ./symbiosis
1797
1798
1799	INSTALL:
1800	========
1801
1802	install the binary in an accessible path like
1803
1804	cp ./symbiosis /usr/local/bin
1805
1806	or
1807
1808	cp ./symbiosis ~/bin
1809	<edit .bash_login or whatever to add ~/bin to path>
1810
1811	or just keep the source in current dir and use "ocamlbuild" as command instead
1812	of "symbiosis". It will create the _build directory, build the source and then
1813	and execute symbiosis as if it was installed according to the above.
1814
1815	Source control:
1816	===============
1817
1818	Please see the anatomy of a system configuration section under
1819	meta-volumes. That section discusses source control branch conventions for
1820	system specific files.
1821
1822	We will continue this convention by adding source and binary to source control
1823	for easy distribution:
1824
1825	To provide a common convention
1826
1827	using the <env:HOME>/repositories/mtn/symbiosis.mtn repository
1828	(or whatever, but use a standard)
1829
1830	Check in the source at branch:
1831
1832	org.example.symbiosis.source
1833
1834	or, if we have system specific agent implementations:
1835
1836	org.example.symbiosis.family.server-cluster-K.source
1837	(replace "server-cluster-K" with your favourite host configuration)
1838
1839	Likewise, check in the binary at
1840
1841	org.example.symbiosis.family.server-cluster-K.bin
1842	(replace "server-cluster-K" with your favourite host configuration)
1843
1844	We definitely are system dependent here.
1845
1846
1847	Hacking the oversight and checking out workspaces again:
1848	========================================================
1849
1850	If a checked out workspace needs to be checked out again, it is possible to
1851	manually the delete the workspace (after assuring any relevant changes have
1852	been saved somewhere). Symbiosis will detect this an remove a log file.
1853	However, it may not be able to update it's own cache at runtime, thus asking
1854	you to try again. This should cause the workspace to be checked out.
1855
1856	The files controlling this are Symbiosis request/response file in a shadow
1857	directory of the proxies directory inside the _oversight directory. By
1858	deleting the file:
1859
1860	activities/_oversight/.../myproject.proxy.volume.checkout.resp
1861
1862	Symbiosis forgets that it earlier succeeded in checking out the
1863	workspace. These files can also be deleted manually, but without removing the
1864	workspace, the checkout agent will hopefully complain about a conflict.
1865
1866	We can run "symbiosis myproject.proxy.volume.checkout.resp" to check out the
1867	workspace, although normally this would happen automatically when giving a
1868	higher level build action as target.
1869
1870	In general, it is possible to delete .resp files to make Symbiosis forget an
1871	agent action was executed. .req files are normally updated according to
1872	dependencies and should not require deletion to force anything.
1873
1874
1875	JSON and YAML:
1876	==============
1877
1878	All Symbiosis files except the myocamlbuild* source files are in JSON syntax.
1879
1880	JSON is subset of YAML syntax, except for the following:
1881
1882	{ "name": "value"} is valid in JSON and YAML
1883
1884	but
1885
1886	{ "name":"value"} is only valid in JSON
1887
1888	YAML requires a space after ':' or it thinks it means something else.
1889
1890	By always inserting space after ':' YAML tools can read the configuration
1891	files directly. Ruby, Python and Perl happily consumes JSON files when this is
1892	taken care of.
1893
1894
1895