Page Content
abstract
A note about !plug's terminology and semantics.
1: WHAT EXACTLY IS A !PLUG ?
1.1. The core reference node
1.2. What a !plug is not
2: !PLUG COMPUTING MODELS
2.1. One class, several computing models
2.1.1 Dependencies (links)
2.1.2 Synchronisation & messaging (pipes)
2.1.3 Storage (containers)
2.1.4 Computed Storage (linked or plugged containers)
2.1.5 Cyclic, associative data model
3: THE !PLUG IMPLEMENTATION IN DETAIL
3.1. The shared context model
3.2. The !plug instance properties:
3.3. The !plug class (valve) attributes and methods:
abstract
This document is a reference of the core !plug reference node and is not meant as a learning tool in order to understand liquid itself or dataflow programming in general. This document should be used as a guide when you need to understand the specifics of a particular process within the !plug node. A prior understanding of either liquid or dataflow in general is suggested to fully grasp the content of the document.
version of !liquid this document applies to: alpha 0.7.0 ( beta Candidate)"
A note about !plug's terminology and semantics.
Although I will try to use academia in reference to the !plug's internal semantics within explanations, the reader should understand that !plug is not a pure implementation of any textbook case and so will present its own computing method and idiosyncracies. The reason is twofold:
- 1) remove the assumption about any academia term, which might be used in contradiction to a pure implementation.
- 2) make the concepts more approchable for neophytes, using an obvious metaphor, which complements many terms used in computing. The brain
works by association and this is a very natural way to understand the system.
1: WHAT EXACTLY IS A !PLUG ? | |
| |
1.1. The core reference node | |
| |
!plug is a reference implementation which allows you to create networks of collaborating computation using dataflow techniques.
Derive !plug to make your own custom plugs, following the guidelines for each function. As long as your node can properly handle the basic public methods (link, instigate, propagate, etc) you can intermingle your own nodes with !plugs.
It is even possible to create mission critical !plugs which only adopt part of the spec, if and only if your node controler will only allow your subnodes to interact with others using the part of the spec your nodes implement.
A textbook dg, dag, node.
Managed. There are currently no specs for the management of the plugs themselves. you are completely free to wrap their use within any kind of management. (graph, dialect, visual ed, functions, etc.
Out-of-the-box artificial neural net toolkit.
2: !PLUG COMPUTING MODELS | |
| |
2.1. One class, several computing models | |
| |
One of primary goals of liquid has always been to allow, several computing models to co-exist in the same network of nodes.
A note about the !plug implementation:
One of the most advanced features of !plug is the way in which an allocated and connected node can change processing model on the fly. This is because the same core object is used as the basis for the following models.
This is the result of a lot of prototyping and several research versions. Each had completely different apis and metrics.
2.1.1 Dependencies (links)
When people think of dataflow, a system which handles implicit data dependecies comes to mind first and foremost. In such a computing model, changes in input cause changes in outputs, automatically, implicitely.
This is a close relative to imperative or procedural programming paradigms since its a processing model which dynamically describes procedures to apply to input data, which gets fed to the next procedure in line.
<p>The !plug is designed to be a pull processing kernel. This is the most computationally efficient model, since it easily allows Lazy computation, a method which will not cause any processing until data is actually needed. </p>
<p>By setting the stainless? attribute to true, the node actually starts operating in a non-lazy push mode, since it will never remain dirty, automatically and explicitely processing any change in any input right away. Although necessary in some circumstances, this can be very intesive since any change in input might force the computation of several outputs for which no data is actually needed. This method, though, allows different systems to stay in sync in real time, and can be a used for signaling purposes.</p>
<title order of evaluation>
<p>By default, processing occurs top to bottom and left to right, just like normal imperative ordering of processes. Some of this can be manipulated by making plugs with custom processing algorithms.</p>
<P class="subject" id="subject2.1.2">2.1.2 Synchronisation & messaging (pipes)</P>
<p>Liquid allows you to easily make several nodes symmetric by allowing them to share a common pipe of data. Basically, everyone receives an event whenever data has changed in the pipe. This allows many things, like having different manipulators and views of the same data. Pipes are often used as data sources for dependent nodes. This has the advantage of allowing many external data sources to supply input as if they where the only data source.</p>
<P class="subject" id="subject2.1.3">2.1.3 Storage (containers)</P>
<p>Containers are simple data store, allowing you to set and update some static data.</p>
<p>Since its part of the engine, the other plugs can use its data as an input.</p>
<P class="subject" id="subject2.1.4">2.1.4 Computed Storage (linked or plugged containers)</P>
<p>this is a very lean combination of links and storage. In this mode, your processing can use static data and then use dynamic inputs to alter it.</p>
<p>This is very effective at simplifying networks which require static data aleviating the need for an extra container node in many cases.</p>
<p>What happens here is that you set your static data, but when asked for its value, it will have been altered automatically by the plug's process.</p>
<p>A Newer variant of this model, is to use the commit method instead of the fill method. This allow you actually fake inputs by storing data directly within the inputs, labeled or not.</p>
<P class="subject" id="subject2.1.5">2.1.5 Cyclic, associative data model</P>
<p>In this model, nodes do not process anything by default, but rather they serve as collections of associated data.</p>
<p>Data cycles are permitted so more steps are needed to ensure the network does not go haywire.</p>
<p>At this point, this type of use of !plugs is still embryonary. The plug supports being linked in cyclic setups, but there is still no management of cyclic processing.</p>
<p>Be carefull as you can very easily end up with cpu infinite loops and deadlocks.</p>
<BR><BR><BR><center><table cellspacing=0 cellpadding=0 class="chapter">
<TR><TD><P class="chapter" id="chapter3">3: THE !PLUG IMPLEMENTATION IN DETAIL</P></TD><TD class="drop-shadow-right"></TD></TR>
<TR><TD class="drop-shadow-bottom"></TD><TD class="drop-shadow-corner"></TD></TR>
</table></center>
<table cellspacing=0 cellpadding=0 class="topic">
<TR><TD><P class="topic" id="topic3.3.1">3.1. The shared context model</P></TD><TD class="drop-shadow-right"></TD></TR>
<TR><TD class="drop-shadow-bottom"></TD><TD class="drop-shadow-corner"></TD></TR>
</table>
<p>!Plugs share all methods amongst all peers of a specific plug type. This is a very different approach to object useage within REBOL.</p>
<p>This means the !plug object itself actually adopts a real Object oriented paradigm.</p>
<p class="title">Advantages</p>
<ul><li>RAM efficiency: this is the main advantage, because all nodes actually share the same class methods, there is no need to reallocate a new context to store locally bound methods. <font color="blue"> <B><I>In the current implementation we end up using 100 times less memory!</I></B></font></li>
<li>Allocation Speed: The fact that we do not have to bind all of those methods, means A tremendous amount of processing is ignored. This is compounded by the fact that memory requirements are so low, that the GC will not be struggling to handle all of that redundent RAM, a non negligible time consumer in its own right. <font color="blue"> <B><I>In the current implementation we end up being 20-100 times faster, depending on size of network!</I></B></font></li>
</ul>
<p class="title">Disadvantages</p>
<ul><li>slight impact in processing: Because we must supply the instance to any method as its first argument, there is a very small cpu hit, as the code must always resolve the path to valve, whenever it wants to call one of its methods.</li>
<li>code look and feel is a bit hampered, but its not extreme. You can build any kind of management within funcs or a dialect which allow you to wrap and hide this extra bit of code (an example is the liquid stubs to link(), fill(), and content()) .</li></ul>
<table cellspacing=0 cellpadding=0 class="topic">
<TR><TD><P class="topic" id="topic3.3.2">3.2. The !plug instance properties:</P></TD><TD class="drop-shadow-right"></TD></TR>
<TR><TD class="drop-shadow-bottom"></TD><TD class="drop-shadow-corner"></TD></TR>
</table>
<pre class="code">;------------------------------------------------------
;- VALUES
;------------------------------------------------------
;-----------------------------------------
;- sid:
;-----------------------------------------
; a unique serial number which will never change
;-----------------------------------------
sid: 0 ; (integer!)
;-----------------------------------------
;- observers:
;-----------------------------------------
; who is using ME (none or a block)
;-----------------------------------------
observers: none
;-----------------------------------------
;- subordinates:
;-----------------------------------------
; who am I using (none or a block)
;-----------------------------------------
subordinates: none
;-----------------------------------------
;- dirty?:
;-----------------------------------------
; has any item above me in the chain changed?
; some systems will always have this set to false,
; when they process at each change instead of deffering eval.
;
; when plugs are new, they are obviously dirty.
dirty?: True
;-----------------------------------------
;- stainless?:
;-----------------------------------------
; This forces the container to automatically regenerate content when set to dirty!
; thus it can never be dirty...
;
; used sparingly this is very powerfull, cause it allows the end of a procedural
; tree to be made "live". its automatically refreshed, without your intervention :-)
;-----------------------------------------
stainless?: False
;-----------------------------------------
;- pipe?:
;-----------------------------------------
;
; pipe? is used to either determine if this liquid IS a pipe or if it is connected to one.
;
; v0.5.1 change: now support 'simple as an alternative plug mode, which allows us to fill data
; in a plug without actually generating/handling a pipe connection. The value is simply dumped
; in plug/liquid and purify may adapt it as usual
;
; This new behaviour is called containing, and like piping, allows liquids to store data
; values instead of only depending on external inputs.
;
; This property will also change how various functions react, so be sure not to play around
; with this, unless you are sure of what you are doing.
;
; by setting pipe? to True, you will tell liquid that this IS a pipe plug. This means that the plug
; is responsible for notifying all the subordinates that its value has changed.
; it uses standard liquid procedures to alert piped plugs that its dirty, and whatnot.
;
; By setting this to a plug object, you are telling liquid that you are CONNECTED to a
; pipe, thus, our fill method will send the data to IT.
;
; note that you can call fill directly on a pipe, in which case it will fill its pipe clients
; normally.
;
; also, because the pipe IS a plug, you can theoretically link it to another plug, but this
; creates issues which are not designed yet, so this useage is not encouraged, further
; versions might specifically handle this situation by design.
;
;-----------------------------------------
pipe?: none
;-----------------------------------------
;- mud:
;-----------------------------------------
; stores manually filled values
mud: none
;-----------------------------------------
;- liquid:
;-----------------------------------------
; stores processing results (cached process) makes all the network lightyears faster.
; this is set by the content method using the return value of process
liquid: none
;-----------------------------------------
;- shared-states:
;-----------------------------------------
; this is a very special block, which must NOT be replaced arbitrarily.
; basically, it allows ALL nodes in a liquid session to extremely
; efficiently share states.
;
; typical use is to prevent processing in specific network states
; where we know processing is useless, like on network init.
;
; add 'init to the shared-states block to prevent propagation whenever you are
; creating massive amounts of nodes. then remove the init and call dirty on all input nodes
;
; the instigate func and clear func still work. they are just
; not called in some circumstances.
;
; making new !plugs with alternate shared-states blocks, means you can separate your
; networks, even if they share the same base clases.
;-----------------------------------------
shared-states: []
;-----------------------------------------
;- linked-container?:
;-----------------------------------------
; if set to true, this tells the processing mechanisms that you wish to have both
; the linked data AND mud, which will be filtered, processed and whatever
; the mud will always be the last item on the list.
linked-container: linked-container?: false
</pre>
<table cellspacing=0 cellpadding=0 class="topic">
<TR><TD><P class="topic" id="topic3.3.3">3.3. The !plug class (valve) attributes and methods:</P></TD><TD class="drop-shadow-right"></TD></TR>
<TR><TD class="drop-shadow-bottom"></TD><TD class="drop-shadow-corner"></TD></TR>
</table>
<pre class="code">;------------------------------------------------------
;- VALVE (class)
;------------------------------------------------------
valve: make object! [
;--------------
; class name (should be word)
;- type:
type: '!plug
;--------------
; used to classify types of liquid nodes.
;- category:
category: '!plug
;- miscelaneous methods
;---------------------
;- cycle?()
;---------------------
; check if a plug is part of any of its subordinates
;---------------------
cycle?: func [
"checks if a plug is part of its potential subordinates, and returns true if a link cycle was detected. ^/^-^-If you wish to detect a cycle BEFORE a connection is made, supply observer as ref plug and subordinate as plug."
plug "the plug to start looking for in tree of subordinates" [object!]
/with "supply reference plug directly"
refplug "the plug which will be passed along to all the subordinates, to compare. If not set, will be set to plug" [block!]
/debug "step by step traversal of tree for debug purposes, should only be used when prototyping code"
]
;---------------------
;- stats()
;---------------------
stats: func [
"standardized function which print data about a plug"
plug "plug to display stats about" [object!]
]
;- construction methods
;---------------------
;- init()
;---------------------
; called on every new !plug, of any type.
;
; See also: SETUP, CLEANSE, DESTROY
;---------------------
init: func [
plug "plug to initialize" [object!]
]
;---------------------
;- setup()
;---------------------
; IT IS ILLEGAL TO CALL SETUP IN YOUR CODE.
;
; called on every NEW plug of THIS class when plug is created.
; for any recyclable attributes, implement them in cleanse.
; This function is called by valve/init directly.
;
; At this point (just before calling setup), the object is valid
; wrt liquid, so we can already call valve methods on the plug
; (link, for example)
;
; See also: INIT, CLEANSE, DESTROY
;---------------------
setup: func [
plug [object!]
]
;---------------------
;- cleanse()
;---------------------
; use this to reset the plug to a neutral and default value. could also be called reset.
; this should be filled appropriately for plugs which contain other plugs, in such a case,
; you should cleanse each of those members if appropriate.
;
; This is the complement to the setup function, except that it can be called manually
; by the user within his code, whenever he wishes to reset the plug.
;
; init calls cleanse just after setup, so you can put setup code here too or instead.
; remember that cleanse can be called at any moment whereas setup will only
; ever be called ONCE.
;
; optionally, you might want to unlink the plug or its members.
;
; See also: SETUP, INIT, DESTROY
;---------------------
cleanse: func [
plug [object!]
]
;------------------------
;- destroy()
;------------------------
; use this whenever you must destroy a plug.
; destroy is mainly used to ensure that any internal liquid is unreferenced in order for the garbage collector
; to be able to properly recuperate any latent liquid.
;
; after using destroy, the plug is UNUSABLE. it is completely broken and nothing is expected to be usable within.
; nothing short of calling init back on the plug is expected to work (usually completely rebuilding it from scratch) .
;
; See also: INIT SETUP CLEANSE
;------------------------
destroy: func [
plug [object!]
]
;- plug connection methods
;---------------------
;- link?()
;---------------------
; validate if plug about to be linked is valid.
; default method simply refuses if its already in our subordinates block.
;---------------------
link?: func [
observer [object!] "plug about to perform link"
subordinate [object!] "plug which wants to be linked to"
]
;---------------------
;- link()
;---------------------
; link to a plug (of any type)
;
; v0.5
; if subordinate is a pipe, we only do one side of the link. This is because
; the oberver connects to its pipe (subordinate) via the pipe? attribute.
;
; v0.5.4
; we now explicitely allow labeled links and even allow them to be orphaned.
; so we support 0-n number of links per label
;---------------------
link: func [
observer [object!] "plug which depends on another plug, expecting liquid"
subordinate [object! none! block!] "plug which is providing the liquid. none is only supported if label is specified. block! links to all plugs in block"
/label lbl [word! string!] "the label you wish to use if needing to reference plugs by name. labels are always exclusive, meaning you can only have any label only once within your subordinates."
/exclusive "Is the connection exclusive (will disconnect already linked plugs), cooperates with /label refinement, note that specifying a block and /explicit will result in the last item of the block being linked ONLY."
/limit max [integer!] limit-mode [word!] "<FIXME> NOT IMPLEMENTED YET !!! maximum number of connections to perform and how to react when limit is breached"
]
;---------------------
;- linked?
;---------------------
; is a plug observing another plug? (is it dependent on a plug? other)
;
;---------------------
linked?: func [
plug "plug to verify" [object!]
]
;---------------------
;- sub()
;---------------------
; returns specific links from our subordinates
;---------------------
sub: func [
plug [object! block!]
/labeled label [word!]
]
;---------------------
;- links()
;---------------------
; a generalized link querying method. supports different modes based on refinements.
;
; returns the number of plugs we are observing
;---------------------
links: func [
plug [object!] "the plug you wish to scan"
/labeled lbl [word!] "return only the number of links for specified label"
/labels "returns linked plug labels instead of link count"
]
;---------------------
;- unlink
;---------------------
; unlink myself
; by default, we will be unlinked from ALL our subordinates.
;
; note that as of v5.4 we support orphaned labels. This means we can have labels
; with a count of 0 as their number of plugs. This is in order to keep evaluation
; and plug ordering intact even when replacing plugs. many tools will need this,
; since order of connections can influence processing order in some setups.
;
; v.0.5.5 now returns the plugs it unlinked, makes it easy to unlink data, filter plugs
; and reconnect those you really wanted to keep.
;---------------------
unlink: func [
plug [object!]
/only oplug [object! integer! word!] "unlink a specifc plug... not all of them. Specifying a word! will switch to label mode!"
/part amount [integer!] "Unlink these many plugs, default is all. /part is only handled along with /only, the /only acts as a start point (if object! or integer!) or bounds (when word! label is given)."
/label "actually delete the label itself if /only 'label is selected and we end up removing all plugs."
]
;---------------------
;- disregard
;---------------------
; this is a complement to unlink. we ask the engine to remove the observer from
; the subordinate's observer list, if its present.
;
; as an added feature, if the supplied subordinate is within a block, we
; remove it from that block.
;---------------------
disregard: func [
observer [object!]
subordinates [object! block!]
/part amount [integer!] "Only if subordinate is a block!, if amount is 0, nothing happends."
]
;- piping methods
;---------------------
;- new-pipe()
;---------------------
; create a new pipe plug.
; This is a method, simply because we can easily change what kind of
; plug is generated in derived liquid classes.
;---------------------
new-pipe: func [
plug [object!]
]
;---------------------
;- pipe()
;---------------------
;---------------------
pipe: func [
"return pipe which should be filled (if any)"
plug [object!] ; plug to get pipe plug from
/always "Creates a pipe plug if we are not connected to one"
]
;---------------------
;- attach()
;---------------------
; this is the complement to link, but specifically for piping.
;---------------------
attach: func [
""
observer [object!] "The plug we wish to start being piped, can be currently piped or not"
subordinate [object!] "The plug which will be providing the pipe. If it currently has one, it will be asked to create one, per its current pipe callback"
]
;---------------------
;- detach()
;---------------------
;--------------------
detach: func [
"Unlink ourself from a pipe, causing it to stop messaging us (propagating)."
plug [object!]
]
;---------------------
;- fill()
;---------------------
; If plug is not linked to a pipe, then it
; automatically connects itself to a new pipe.
;---------------------
fill: func [
"Fills a plug with liquid directly. (stored as mud until it gets cleaned.)"
plug [object!]
mud ; data you wish to fill within plug's pipe
/pipe "tells the engine to make sure this is a pipe, only needs to be called once."
]
;- computing methods
;---------------------
;- dirty
;---------------------
; react to our link being set to dirty.
; if the special shared-state contains init, no propagation occurs.
;---------------------
dirty: func [
plug "plug to set dirty" [object!]
/always "do not follow stainless? as dirty is being called within a processing operation. prevent double process, deadlocks"
]
;---------------------
;- instigate
;---------------------
; following method does not cause subordinate processing if they are clean :-)
; very computationaly eFishAnt (tm)Steve Shireman
;---------------------
instigate: func [
"Force each subordinate to clean itself, return block of values of all connections or pipe."
plug [object!]
]
;---------------------
;- propagate
;---------------------
; cause observers to become dirty
;---------------------
propagate: func [
plug [object!]
]
;---------------------
;- filter
;---------------------
; this is a very handy function which influences how a plug processes.
;
; basically, the filter analyses any expectations about input connection(s). by looking at the
; instigated values block it receives.
;
; it will then return a block of values, if expectations are met. Otherwise,
; it returns none and computing does not occur afterwards.
;
; note that you are allowed to change the content of the block, by adding values, removing,
; changing them, whatever. The only requirement is that process must use the filtered values as-is.
;
; note that if a plug is piped, this function is never called.
;
; in the case of linked-container?s? the function is now called, and so you can fix it as normal.
;
; eventually, returning none might force purify to propagate the stale state to all dependent plugs
;---------------------
filter: func [
plug [object!] "plug we wish to handle."
values [block!] "values we wish to filter."
]
;---------------------
;- process
;---------------------
; process the plug's liquid
;---------------------
process: func [
plug [object!]
values [block!] "filtered and ultimately valid data"
]
;---------------------
;- purify
;---------------------
; purify is a handy way to fix the filled mud, piped data, or recover from a failed process.
; basically, this is the equivalent to a filter, but AFTER all processing occurs.
;
; we can expect plug/liquid to be processed or in an error state, if anything failed.
;
; when the plug is a pipe server, then its a chance to stabilise the value before propagating it
; to the pipe clients. This way you can even nullify the fill and reset yourself
; to the previous (or any other value).
;
; eventually, purify will propagate the stale status to all dependent plugs if it is not
; able to recover from an error, like an unfiltered node or erronous piped value for this plug.
;
; Note that the stale state can be generated within purify if its not happy with the current value
; of liquid, even if it was called without the /stale refinement.
;
; we RETURN if this plug can be considered dirty or not at this point.
;---------------------
purify: func [
plug [object!]
/stale "Tells the purify method that the current liquid is stale and must be recovered or an error propagated"
]
;---------------------
;- cleanup
;---------------------
; processing manager, instigates our subjects to clean themselves and causes a process
; ONLY if we are dirty. no point in reprocessing our liquid, if we are already clean.
;---------------------
cleanup: func [
plug [object!]
]
;---------------------
;- content
;---------------------
; method to get plug's processed value, just a more logical semantic value
; when accessing a liquid from the outside.
;
; liquid-using code should always use content, whereas the liquid code itself
; should always use cleanup.
;
; optionally you could redefine the function to make internal/external
; plug access explicit... maybe for data hidding purposes, for example.
;---------------------
content: :cleanup
]
]
</pre>
<br>
<br>
<hr>
<center><P class="note"><b><i>page last updated: 27-Dec-2006</i></b></p>
<A HREF="http://www.rebol.com"><IMG SRC="../../images/powered-by-logo.jpg"></A>
<A HREF="../../index.html"><IMG SRC="../../images/built-with-remark.jpg"></A>
</center>
<br>
</body>
</html>
