Call/relay/broadcast within process or at the end

A few days have gone and still no Eclipse project in the new environment. Did I underestimate the power of the human factor (hard for me to start)? Or...

What I try to do now is to create the kernel features. The absolute minimum that my system must provide to an external developer for using just anything in the actual, dust-based environment. I have hit a serious question about control. When communicating with other components, I need to support the following tasks:

• direct call to another component, then continue the actual operation using the response (like evaluating an expression or alike);
• broadcast to a set of components (either through an iterated member field, or a result of a query), then either wait for the results, or continue operation, either synchronous or asynchronous way;
• set one or more relay components, either parallel or serial way (like in the interpreter, junction or sequence), either waiting for the return (and continue, like processing the closing block in a stream) or just let it go.

This can be done, my problem is with the "wait or not" question. If I wait, I need kernel functions to do this. If it is at the end, I need a special return constant to inform the kernel that it has to do so. I hate parallel stuff, and not straightforward solutions. If I support calls only, the kernel does not know if it has to do anything after the call, if it is at the end of the caller function. If I support return type only, the code becomes ugly and fragmented. None of them are acceptable.

Do I look in the wrong direction? A weird, but possible answer is: yes.

I keep thinking about providing a programming language interface, API for using dust, with generated codes, etc. At the same time, I think about an interpreter that allows fully functional dust programming above the language layer. What if I turn it upside down? 

I can say that dust has no programming interface. I have to "implement" my services using the declarative tools, without using any generated code. The only codes appear at the edges: when a component brings a functionality from the external environment (runtime, like JRE, threading, stream, etc.; or external toolkit, like a database connector, GUI library, etc.) So, I only need

• structure constants that the real codes can use to access data stored in their own connector data objects;
• connector constructs that help the loader to connect the functions/objects to the aspect message processors;
• and perhaps some generic tool functions that I always implement in all language environments to speed up development (in Java, StringBuilder manipulations, arrays, ...).

However, there is no need to support programmatic connection between custom codes. This is fundamentally modifies the role and core features of the shared kernel codes. That is:

• no entity evocation, 
• no message sending, call management, relay, etc - they go out to the core feature set available from the algorithm configurations...

This is most strange because I generally hate when configuration hacks replace source codes. On the other hand, I expose the structure to the framework and that looks great - I can do the same with the algorithms as well, that may be beneficial. My experience, and the currently growing number of interpreted script languages show that the common runtime performance allows this approach (the algorithm tree is exactly the same as a parsed script code). And there is still the option to generate source code or compiled binary from this tree... It is so unfamiliar because almost all codes that I have written so far or imagined in dust just disappeared.

The new approach also modifies the kernel implementation approach. So far I thought of the dust kernel as just another collection of components that talk with each other. Now it seems that the kernel components can be more interconnected as a Java implementation of this runtime, because there is no such thing as "dust code".

The implementation order:

• the shared parts (public include, shared Java toolkit project);
• kernel (with its modules perhaps in separate projects, but they can be interconnected on Java layer)
• external tools for the kernel (all for which normal dust communication is enough, like stream, log, etc)
• behavior (control execution, and expression evaluation)

Fundamental projects

Separate the Dust API from the Kernel. The first contains the communication layer and base classes, essential to communicate between components. The Kernel is the implementation of the API, it contains the main function and the initialization process. Important: it also must contain layers, because there can be various startup environments (for Java: desktop, servlet and GWT client are currently considered).

Start liking GitHub (server and client). Nice "for dummies" environment, perfect for me as I want to waste minimal amount of time on the environment, but make it safe. I use a new repository and also work with the wiki, both are better than enough for me.

On the other hand, it is hard to set up the toolchain. It's great having a mindmap on my tablet, but that is connected to Dropbox, not to GitHub. I type things here that I should add to wiki. In some cases I already feel the limitations of the wiki, I should use UML instead, but where - on the tablet or desktop? Better here because the UML plans should go directly to GitHub...

Dust restart

Besides logging my rantings on various forums, I decided to use this blog as it should be: a public web log. More precisely: keeping notes on my progress, so there is a place to go back when I go to a dead end.

So, I am restarting Dust (after countless previous experiments). The new start is motivated by the current achievements (and the further directions that are not reasonable to chase) at Continental. I managed to create a programming framework, that is seriously modular, mostly configuration-based (including the application structure like Spring and the GUI like XAML), mostly message oriented. A really big surprise: the JRE is very fast, although the fundamental comm layer is terrible (HashMap<String, Variant> value access, reflection-based function call), and that is still "good enough", even when this Java code runs in the browser (GWT client application).

So far so good, but the next step is upgrading the core with generated accessor codes, while that is an extremely complex stuff, and the framework is "good enough". I would not recommend this way to my bosses, though I would like to do it. This means coming home, and relaunch dust development, with the current experiences, but broader (aka: insane) scope.

Meta layer brought up another question.
I managed to separate the app structure from the language, not only the executor component hierarchy, but also the predefined messages, so sending them can also appear in configuration. I also integrated a script engine to the framework, so the "algorithm" became configuration, and (due to unpleasant reasons) I implemented a quick and dirty Excel formula parser and executor.
I am also sure that all source code is a serialized form of an algorithm, and the algorithm consists of only a very few tasks. This is why the source code can be compiled to machine code, and this is why a so simple language as C can be generally used to solve any task. (Not to mix the features of the programming language with the external tools and the runtime).

So, I want to implement the algorithm layer as well, and generate the language source from it. Finally, I want to use plain C, because that focuses on the control structure; it is harder to manage Java/C# sources, where the language and the runtime is merged.

Serialization brings grammars into view. I chose ABNF for it looks simpler than EBNF to me. Checking the elements that are required to build a parser, I found a close match with the control structures (block, sequence, repetition, junction). I want only one, and that should be the control components; grammar should be a user of control components.

I also found that relay comes into view. When parsing a stream, a junction (or the end of a repetition) means the rest of the stream can continue in multiple paths. My parser is already a SAX analyzer, the stream has the control. This means the junction root does not "call" the optional processors, no "trackback", but it should register multiple relays, and should get the control back when they finish (and in normal case, only one succeeds). In this way, the relay path represents the call stack (easier to analyze).

I have problems with separating the serialization layers: the grammar (control structures that can do stream i/o), the grammar of grammars (which is able to generate control structures from a grammar definition), reading arbitrary content (by a simple language), serialization (the same data language with smart content processors and some semantic rules), and code (control structures in a real programming language).

I can't see the separation, so I have to implement a working code to see the process. Today I am a Java programmer, so I will create this part in Java for the first time. I need a generic Dust runtime, and process ABNF, JSON and C. The required components (separate projects):

• API: base classes, fundamental services; standards for pseudo-generated codes
• Data: cloud, Entity/Aspect separation, aspect access, data management
• Process: call, relay (later intercept)
• Util: some of the generic util stuff
• Stream: read and write files; processor types
• Behavior: the Control and Expression components

Nice list.

Business logic - brain dump

From the aspect type, a full „bean-like” interface source can be generated, and so this instance will be returned on request. The type restrictions can fully apply here. Behind the interface, a generated mapper code exists that can 1: using the generated constants it can call the generic data access/message sending

Messages: messageID is mandatory; additional parameter type optional; if present, providing a message entity with that aspect is required. Question: id is globally unique OR I can use the same id for multiple entry points with different type parameter?

From the user perspective, the message ID identifies the required service – and that is independent of the actual parameter; and anyways, practically ANY instance can be thrown on that message if it contains the required aspect. On the other hand, the message processor should be different for the different content, because the incoming parameter data has to be handled differently, though there were shared codes for the actual function.

Result: there can be multiple incoming parameter declarations for the same message IDs. The caller may send any of those parameters to the same service – actually the caller does not see the entity, only the aspect with the required type. The message processor can also be declared with the same interface…

So: the proxy implements both interfaces: provide a bean-like typed access to the local data; the referred linked entities (DATA), and the functions that can be called (SERVICE). On the service side, the user code has to implement the SERVICE interface, and has access to the wrapped DATA instance, handled by the framework.

The proxy object itself is an “aspect” instance, with direct access into the access implementation code, that is the actual executor of the proxy wrapping calls (local, no more dispatching). The same applies to the message service interface: that is linked to the dispatcher implementation. This instance is also passed to the working code; it can send message to itself through the proxy object… There is a third component: the actual data area containing the instance data? NO: that is the entity instance that contains a “map” of aspect references connected to data content.
Preliminary idea - The dispatcher contains a tree: implementation / type / loaded business logic class. The lowest level is a Method ID. This ID is returned back to the caller in a map when resolving the message IDs; on actual calls these IDs are referred to, and the actual methods are called after the proper administration. REMEMBER: this dispatcher is STATIC, able only to resolve the known type calls. Another level is responsible for loading new implementation packages, and yet another for parallel thread management. Perhaps additional aspects of the same entity… There is one problem: different implementations can coexist for the same type. The aspect statically refers to the type, but the instance (inside the Entity) may refer to other implementations. So when calling the method, two separate levels are required: the Entity for the actual implementation, and the aspect with the actual type. In this way, the Entity contains ONE ref ID to the implementation, and the Aspect knows the type IDs for the actual message identifier on the type level. Both IDs are required for calling an actual method. This looks good.

The other side does not look this complex: the memory management seems to be clear. The aspect knows its required fields. It passes to the memory manager that returns the offset for each member, and a reference in its config area. When creating an instance, this reference is used, the memory is allocated, and a block ID is returned. On data access, this block ID and the offset is required, the memory content is copied out or loaded into the memory manager area. NO direct access is allowed in the mem manager area. Here we have the locking issue, and also the configuration change features which are other aspects of the memory manager. By default it is only able to serve those instances that it already knows. The initialization is a bi-directional communication between the mem manager and the aspect instance.

Meta - architecture

Meta, because now I don't think about the architecture of the Dust framework (contexts, entities, aspects, messages), not even the connection to the running environment (wrap the hardware drivers into aspects, and thus remove the dependency to the running environment: OS or JRE), but the connection between the Dust ideas to a compiler that is able to transfer them to a computer-executable form.

I do this because I must have a vision of the core functionality without language dependency, and provide running solution to those questions in multiple languages to prove the independence - even if those solutions are fundamentally different because of the language. But first to clarify: for me, "language" means a syntax of a broadly used compiler/interpreter, with which I can reach many environments. "I" mean that I can only use my time, so something that I know or think I can learn in an acceptable time. Therefore my current focus is Java because I know this the best, and if I can use Dust in my work as well, that should be Java. I also have to provide solution to the same requirements in plain C for the contrast. For fun, I keep Objective-C for iOS, Android for other tablets, GWT for generic web interface, and maybe C# for current Windows development. From this, I found Objective-C and C# pretty straightforward environment for my concepts; GWT is mainly Java without reflection for user interface subset (and so a funny mix of Java and C concepts). I know nothing about Android, but I would be surprised if I can't make use of it in months.

I know that there are other paradigms and languages, and it is free for anyone to do a Haskell, Matlab, Prolog; Perl, PHP, Javascript, Ruby or Brainf*ck implementation for the Dust kernel. From my point of view that would be equal waste of time: I don't know these languages, and I don't think that there are environments that can only be used with them and not by the above mentioned ones. To be honest: I don't see too much real value in them either, but that is my private opinion.

"No Dust"

I have concluded to the statement of "there is no Dust Framework". Of course this is not about the existence but the visibility. Yes, the framework exists, a Dust programmer has to know about its structure and yes, the solution will be dependent on the other components one uses, including the framework components. But Dust kernel components and services do not appear on programming environment level, and if you want to provide different implementation, or even different architecture for the Dust kernel services, you are free to do so. Of course that solution will not be compatible with the standard Dust environment, and as the drivers and tools use the kernel too, you would have to reimplement them as well to make your solution run. Or, you can even create a "virtual machine" for your altering kernel inside a standard Dust runtime. But I got far from the original aim.

So, Dust kernel itself is (and should be) invisible on the language level, because we use the language only to provide functionality (behavior) behind the aspect declarations (made either by you or someone else) that can be used in a running Dust environment. So we have the following requirements:

• access data and send messages
• let the runtime identify and call my function on an incoming message
• the code should be able to utilize elements of the (language independent, Dust based) type declarations with the most compiler support (type safety, compile-time warnings or errors)

These functions can be fulfilled very differently in an OO environment (Java) and in plain C of course, but having them provided means that Dust components (even the kernel itself) can be implemented in that language. For a side note: yes, it gives a transparent break through language barrier: you don't have to know that the actual runtime was implemented in C and runs on native level; if it has a Java loader, it can use your components implemented in Java - or vice versa.

Access data / send messages

When the code talks to the environment, this means that it refers to some components around it through identifiers, and the environment should resolve those identifiers to actual component data or service calls. The "context" is (quite poorly though) handled in OO languages with the existence of "this"-like built in language construct through which the code can access its "own" data, so in theory Dust can utilize this - but this has limitations:

• of course there is no such support in non-OO environments;
• even in Java-like environment, if the object provides the gateway to the kernel, it should have to be passed all the way round when calling other services. Not very nice.

For these reasons, and because I want to promote a kind of language-independent Dust coding style, I vote on similar solution, and this means adhering to the limitations of the weaker environment, C. Dust service appears as a final static global object for Java, and as the "send", "get", "set" functions in C. One, totally generic header file included in the code, and it can use Dust - and through this gateway all the functions of all the referred components (including the kernel of course). 

Any code can use these functions, the Dust kernel on the other side has to know the actual context - and yes: you can't play tricks with it. For an OO programmer, this should feel bad: using a super language like a script. On the other hand: the "superness" of that language is not the syntax itself, but the huge set of service components built into the running environment (GUI, persistence, ...), the way that language supports building the data and application structures (classes, reflection, ...) and environment (session information, connection pools, threading). But hey: these things belong to the abstract, language and environment independent architecture of the service, and in fact they should not be burned into a specific running environment.

OR... Convenience

I plan to generate code for the identifiers from Dust declarations in the target language. Why not generate wrapper classes for the types? That would hide the strange behaviors of Dust, would support more type safety, compilation errors when using the wrong identifier on accessing a component. It would also make Dust even more hidden from the programmer, and would make Dust programming very similar to the "normal" way.

Well, at this point I can't decide if I should implement this layer or not. The above reasons are very strong:

• having compile-time errors on access and type issues instead of runtime exceptions is very important;
• when I write implementation and use another component, I really "lock" myself to it. Dust declaration means a strict separation of public interface and private implementation; the provider can change the code, or I can switch to another provider - the interface must be consistent;
• "esoteric": this way Dust can really disappear.

What does it mean to C codes? In each implementation source you can have typed accessor functions to the declared data and message senders (including to "this" with a special reference constant). This might result that a code that does not use direct casting and does not play with void* and can be compiled will be valid as well.

The result is a totally "normal" implementation code. The only difference is that the "connection" of that code (constants, internal and public interfaces) are generated from the declarations and not typed manually - which of course guarantees following the standards with them. When you use a unit, you also get the generated public interface, and can access the methods "normally" - even though that may go through the Dust kernel, sometimes to another core or a separate running node.

I have to admit that this may worth the extra effort, even though I am very far from the point when the required sources can be generated, and I would need running Dust subset to generate them. Or should I type them in? Is it really the point where my absolute favorite KISS rule should be broken?

Or: this is actually the absolute KISS rule: the framework with a minimal footprint on the application level code. In fact, that footprint is actually none. I like that, so even though looks tough, I should follow this direction until it breaks.

Integrating my logic to Dust

TBD

Identifiers

TBD

Dust is a journey

"Life is a journey, not a destination"

Although the origin is not that clear, I have heard this sentence in Amazing from Aerosmith - remember the time when pop music was not trapped in "love me baby" / "I hate the world" / "life sucks" triangle, but actually had lyrics that is worth reading and thinking about... :-)

Finally I have realized why I write a blog about Dust development instead of a wiki specification.

Of course it is a question if the whole idea has meaning, the fundamental question is good and I am looking for the answer in the right direction. But this is a question I can't answer, because this is my faith and passion that this has to exist, has to be written in this way, and I can write it. I can be wrong of course, but what a great way of making mistake this is! And anyway, being an employee now I plan to get a lot of money on using experiences that it gave me; and even if it is finally wrong, you can also find some useful ideas in it.

But what if it is right? I think it should turn the industry upside down. I think it can let us use the full power that we already have in the smallest gadgets (I started programming on machines that had less power than a ... I can't even think of any nowadays smart toys of that weak performance: 640k RAM? 4.77 Mhz (but had Turbo button)? 20MB HDD?). I think it means that what we call crap today can connect huge amount of less fortunate people to our technically global, but economically very limited world. I think the epic failure of a Java processor can be tried again: the Dust kernel is a better choice to be implemented in silicon; the hardware can know all applications internally, and adapt to them, optimize the behavior; Dust is inherently parallel, not by letting the programmer control threads but this is a fundamental element of the software component design.
All in all, I think huge mass of people would benefit from it - and this makes it worth the time and effort for that very little chance. But they will not understand the whole thing at all, which is not a problem.

There should be lots of adopters, who would understand a specification and can use it. For them, Dust will mean truly reusable designs (for more forethinking of course), having the housekeeping done by previous standard components in really any environment, etc. They will give the muscles to Dust, but sorry, I don't really write this blog for them.

There must be some people who will follow the way I think about Dust. They must take over this obsession of something that must be brought to life, to this world. To always take the longest, most weird path through a swamp if they feel that they must go that way - even if there is no light on the other end. And to understand why I do this, what experience and vision forces me doing this now. I publish this blog for that few dozen people out in the world now, or at any later time. They should take this concept further than I could, because of my limited knowledge and time.

So for them: yes, it's ridiculous how I am linked to Esperanto for both any programmer (why not use English, bro?) or for an Esperantist (you don't know the language at all, man!). But I have a very strong vision that Esperanto is an existing, full language; its grammar can be formalized and used to express any state of the world or exact orders without ambiguities; even the speech to text conversion is easier because of the fixed position of the accentuated syllable. Finally, I want to be able to express actually anything in Esperanto, and that should be understandable and executable orders to the Dust environment.

I had the same strong vision before, like "use message object only instead of call parameters", "there is no difference between service and data objects" - the result is a fully declarative software architecture; "the OO paradigm and the Operating System itself is a very useful but fatal failure" - and I could switch to C# in weeks for a POC system; I could design and implement a production system in Objective-C. I feel the same now.

Self containing framework

And here I am again at the root question: how does a Dust application start up? With the addition of the new idea that "there is no Dust framework"... The latter means that there is no static entry point, no framework functions; all such thing are done by the kondutos of the Dust core terms. The only interesting thing is that the "main" function has to set up those items and connect them to each other: this core component network is able to serve the basic functions: message passing and data/reference access requests. Later, when I can add other unuos like serialization, binary loading, GUI, etc, then the framework can stand on its feet.

But first of all, I have to start up the self containing core service network. To do this, I have to create a kunteksto, and put the root components into it in the root initialization call. This code will be generated later from the system deployment information, but now of course I have to write this code myself.

The kunteksto, as this will go later, can be modified in a transaction. I can send ento initialization information (not actual instances, because the objects must be created by the kunteksto - on the other hand: how can the kunteksto create external, not yet added types? hmm...) The ento contains one or more aspektos with initial data and references to other entos. The references are resolved in a lazy creation fashion, so the referred ones get placeholders in the kunteksto, and later when their actual content arrives, are initialized. At the end, the transaction is committed - at this point this means the kunteksto runs a consistency check if all the references are resolved. The kunteksto ento is added to itself (because it is also referred from the entos inside).

At this point I will have a kunteksto from which I can access different functions, but naturally it would not be "Dust-like", so instead of this, I also have to add a test "Runnable" ento, and do those actions in its run message processor. Later on, drivers, external message handlers, etc. will be "Runnable" and started in this way.

Identifiers

This is where the nice idea seems to fail. The identifier (identigilo) is a string constant; the instance should represent its own name. The instance should be

• accessible for the user codes (in generated sources the coder will use them);
• should not be mixed with external objects (cannot be created by the user code, and the messages should contain only these constant values)

It is so nice that I keep forgetting what I already thought over... All components, including this initial framework core is inside a realigo: it has a connection interface to the framework - to itself. The identifiers are generated source codes: enums; the access functions use these enums. On the other hand, the framework resolves the names (the enum name strings) to actual identifier instances, which are lazy created. Now I have to type the eums, but later on that code will also be generated from the identigilos in the unuo. The enum strings are loaded into the global identifier table, and also attached to the enum order number table contained in the realigo. So the client code refers to the table index through the enum, and the reference is resolved by the framework by the enum name at the first call.

The only difference is that during the initialization, this local index table is pushed directly into the kunteksto from the source code.

What to add?

... to be continued...

Dust core terms

Now trying to collect the first terms in the Dust vocabulary.

Meta-meta layer

• identigilo: Generic variable type for identifying core data elements: attributes, messages, references. Atomic information, general implementation is an ASCII string identifier (small and upper case English characters, numbers, underscore). Has NO functionality, works like generic constant variable for declaration-level codes (like a GUI display or expression); high level business logic components "see" the identifiers through generated source constants (enum and alike). Not to be confused with the unique identifiers of entities (although it works as that for the core elements, but only within the closest context).
• variablodeklaro: The declaration information of one variable, either a local temporal value in a certain context or a member value inside a type. This connects an identigilo to a usage information structure: data type, life cycle (constant, final, modifiable, etc), ... contained within this component. The identifier must be unique in its actual context, as the value is referred to by it. The type may either be a generic type or a (single or multi-value) reference to other aspects. Handling references is different from generic data, but the identification, passing and resolving should "look similar", especially in expressions and language constructs, this is why they both covered by this component. Side note: "value set" for option selector is handled by a set reference to other entos in this type: the elements of that enumeration are entos themselves. This is different from 
• tipo: This is similar to the normal "type" or "class", contains 0-n variablodeklaro elements forming the "data context", and a list of other tipos that it can accept. All kondutos (business logic) are implemented as being inside the context of an aspekto ("object instance of tipo "class"), that gives the "member variables and references". All logic entry points are reactions to an incoming "message": ento with the primary aspekto of the referred tipo. Furthermore, each tipo contains 0-n "required" tipo elements, which are also required to exist in the same ento. The business logic can refer to these other aspektos of those tipos as present, this feature provides behaviors similar to inherintance without the forces tree structure.
• datumo (generic type) wrapper type for generic types handled by Dust: boolean, integer, etc. The common set is extended by identigilo, which is also a "generic type" in Dust. Funny question: I started reducing this set by saying: no, string is not generic, float is not generic; what remains? Integers with different byte size, like in assembly, and the special identigilo. Am I surprised?

Meta layer

• aspekto: This is similar to the normal "type" or "class", although in Dust you have no independent "Object instances" by an aspekto. It contains 0-n variablodeklaro elements forming the "data context", and a list of other aspektos that it can accept. All business logic are implemented as being inside the context of an aspekto, that gives the "member variables and references" of the business logic. All logic entry points are reactions to an incoming aspekto "message". Furthermore, each aspekto contains 0-n "required" aspekto elements, which are also required to exist in the same ento. The business logic can refer to these other aspektos as present, this feature provides behaviors similar to inherintance without the forces tree structure.
• ento: Entity, a programming object instance inside Dust representing a "being" either in the outer world  (like a person, a booked resource, ...) or inside the virtual world of the running environment (like a GUI element, an event message or even an exception). Each ento has at least one aspekto, and always have a primaraaspekto, which the ento "is".
• kunteksto: container of entos and their relationships. Responsible for storing, resolving ento instances and also for their relationships (which is not stored inside the entos). The kuntekstos form a tree, the root is the first, meta-kunteksto. The connection is invisible for the entos inside, any resolution request not found in the current kunteksto is passed up to the parent until the root. The root kunteksto contains itself as a normal ento, and (as far as I think now) all the other kunteksto instances as well. However, a child kunteksto may contain internal kuntekstos acting as a "virtual machine".

Term context

• vendanto (vendor): the responsible owner of terms and behaviors, responsible for the coherence and the unique identification of the provided element tree.
• bieno (domain, field): the higher area of connected terms. From the application point of view, bieno is the highest level: "all" terms that are handled by Dust has to be distributed into disjunctive, coherent subsets: these domains. However, this is life, different owners can have altering ideas of these domains, their separation and content, there can be multiple, competing alternatives. So, Dust does not try to enforce its own structure as "the" solution, the framework (as meta layer) is designed to handle multiple approaches from different vendors (Dust is only one of them). This is why bieno is under vendanto.
• unuo: a "working unit" inside a bieno. Unuos contain a set of tipos; and an optional list of referred unuos. The tipos can only refer to other tipos from the registered unuos. The Dust running environment ("an application") consists of several unuos, where these references are all resolved within the set, so all used tipos have their declaration and behavior loaded. 

Service layer

• konduto (behavior): the application logic assigned to a tipo. The konduto itself is stateless, exists in one instance, has no "member variables" in any programming language. However, is can be stateful by injecting a workarea object into the aspekto that it handles, so when needed, it can analyze the data (publicly available and handled by Dust), but store local information to it to speed up processing. However, such entos will be harder to serialize (for example).
• realigo: an unuo has a single declaration, but possibly multiple implementations for different actual running environments, like drivers, GUI display frameworks, etc. A vendor an provide multiple realigos for the same unuo, even for unuos from other vendors. Entos representing an actual hardware have to be bound to their own realigo. The realigo contains kondutos for some or all of the tipos in the unuo. Question: what happens when a specialized konduto for an ento refers to a shared ento that has different kondutos in the available realigos?
• sendito (behavior container): in the kunteksto, this component contains the kondutos of the knows tipos. Responsible for calling the proper method with the target aspekto and the received message ento.

Identifiers and aspect references

How does a business logic object access other aspects? The connection can be dynamic and static.

Dynamic connection means something that the code accesses "right now" and it can change. Like in the messenger application, get a list of messages and select one. The selected and displayed mail is a temporal connection to a message, you can select another message and switch the display to it.

A static connection is something that the actual code can't change, only uses them, it creates the environment where the code is running. There are known examples for this in "standard programming": this is the Runtime for a static application, or the Servlet Context / Session for a web service; the start up parameters and the Environment variables, content of a configuration file or servlet / context xml.

In Dust this goes further. In standard programming, the code contains the function and member variable names, static / global variables. They are resolved by the compiler and linker to memory offsets and jump table indexes, burnt into the binary (and causing unwanted component dependency network, version compatibility issues and the  "dll hell").

Here you have aspect references only. Simple put: you have dynamic aspect references to object that you want to work with; and static references for telling what you want to do. When you declare that you want to use a certain type, field or shared instance, you actually issue Dust to generate a new constant: an index in a reference table through which you want to access that field or instance. The signature of your implementation contains this list with the referred identifiers, and when your code is active, this list is resolved (either immediately or in lazy-fashion: when first referred to) to actual type and field references - which are Aspect instances again. When you refer to a field, you actually use an index in your reference table, and behind it you get the aspect instance defining that field (and actually knowing not only the memory offset of that data, but all other settings like type, life cycle, access rights, etc.) The code there is responsible for giving you the content of that field from the actual Entity instance (which is then required to contain the memory for all the aspects with proper offsets again).

This finally answers the oldest question of mine: what is an "initial" Dust application and how it starts? The minimum Dust is the meta entity (Type, Aspect, ...) instances in a self-containing Context, with all their static references resolved to each other in code level (this code is generated from the core configuration). The core configuration can be extended to the required types, and we have a static-typed application (no type or binary loading, upgrading, like a GWT presenter); or even to all the entity instances (and we have an embedded application with no dynamic memory management). Of course, memory management and type / binary loading is the "next ring" of Dust kernel, above the absolute necessary core.

I still have issues with the Reference management. It now seems that Aspect instances can only contain generic variables; references to other Aspects require special care, using Containers and / or a dedicated Reference manager. This will be the next run.

There is no "Dust Framework"

The newest finding. As I was looking for the absolute minimum requirements from a framework (more precisely: an external framework runtime environment) it seems to disappear.

When I have all business logic implemented as business logic components derived from a common base class, it seems possible that the proper network of these elements (the grains of Dust) can do all the required operations. The "application" itself is not more but a component entity that has a Context, which is totally similar to any other "Context holders", like a GUI window or a serializer. The only difference is the initial boot process, where the application started on the machine, has a generated code segment that puts some preconfigured entities (like the meta-level type and service holder instances) into the Context before giving the control to the "launch" message handler.

It will take some time to process this idea.

Runtime 1

The runtime is the heart of Dust - not a surprise. The most important: I must believe that it is also totally simple. I must not think about speed right now, only one rule: KISS. Later on, when I have a working implementation based on the bones, there will be time to add muscles. But I should not start with them.

What is the ultimate minimum for this environment?

There is no type management, or array, map of types. I only need type by its id on meta level (reading a serialized content, or generating a GUI), otherwise the type is referred directly from the aspect.
No need of context management, the Context instances are also in one Context: the root context of the application.
There is no static Dust API. All codes run as event processors, so they have the same base class, and the Dust API is the functions of that base class. Even Dust kernel, which is only another instance in its own Context.

Esperanto programming

This is a strange one. I feel that I should use the Esperanto language for creating the terms in Dust. The reasons are the following.

Internationalization

Information systems are theoretically global. A person is a person, a chair is a chair, a calendar is a calendar everywhere on the Earth, at least on declaration level. Of course, we put different holidays into the calendar in Hungary and China, we write texts in different languages, and we want to see the user interface or the notification messages from our calendar in our chosen language - but the types and the business logic is the same.

However, in general we use English language to write source codes and comments - that is: we declare the elements in a natural language that we assume the other programmers will understand. So, in fact our systems are English by default, and then translated to other natural languages. However, the terms of our task domain should not be English, that is another "implementation", not the abstract definition. Luckily enough, we have an artificial language with the simplest possible (artificial) grammar, and a full word set for daily and scientific use. Luckily enough: I myself don't know Esperanto! so I will have an immediate need of anything I do to be translated to English, which I, and my fellow programmers can understand. It is a fundamental design decision of mine to separate identifiers from text information, where identifiers are hidden, programmatic information that can never get out of the system to the user, only through a translator - but now this will be an inevitable requirement because I (and my colleagues) would not understand the user interface or logs, if it is not translated to a language we know.

Esoteric

The esoteric reason behind the decision is that with Dust I start to create a parallel world. For the first time of course, I create the meta-meta layer: the definitions of "Type", "Aspect", "Entity", etc.; instances of these types will contain "themselves": the Type, Aspect, Entity, ... type definitions. Then other type definitions come which describe a Person, a Chair, etc. and then instances of persons, chairs, etc. So finally I will have an information system that can represent entities and relations in the external world. So far there is nothing new, all software does the same, yet the programmers don't think about it, and uses the constructs of the programming language for the meta layers and start with practical types. But here I talk about a system where for declarations I don't use a programming language construct, but an independent, external utility.

The structure of that knowledge, the meaning of the terms are the "words", the types: a "Person" is a structure of information that we have described in the Person type. Not a human perception of a Person, but an informatic representation of a person. A natural language word represents a human understanding - the artificial language word represents a type definition. A human knowledge can, and has to be simplified and organized to what the actual type definition can hold; and a human perception can be built by a human brain after browsing complex information networks available in the information system. I also think that all system knowledge can be translated to syntactically valid sentences; but to understand (parse) sentences, it is a fundamental need that the generation rules are clean, simple and obvious.

I accept Noam Chomsky's sentence that Esperanto is not a language. A natural language expresses and also affects the unique structure and complexity of the human brain, with all parallelism, analogies, different meanings, the very nature of human existence. This must be transformed when dealing with mathematics, natural sciences or computers to an environment that is locked to strict rules and only one, perfectly defined meaning. This can only be done on the human side, and I think all the attempts for automatic translation and machine understanding are doomed. The machine understanding will always fail on this in a certain percentage, and the more important it is that the machine does exactly what I want, the less likely that I will ever trust a speech-to-text engine and would want a button to press instead. Or: I have to learn how the machine will understand my sentences, and talk to it using an ultimately simplified "Language-for-my-coffee-maker"... But why?

We do have Esperanto, an artificial generic language. If we have the type definitions (the exact "technical meaning") of each word in a transparent form (again in Esperanto), we have a perfectly clean interface to the system: we can know what attributes, commands, relations we can refer to when talking to the coffee machine. We can use a simple but generic (and so globally available) language that is easy to learn, and so enter a world where we can actually talk to any entity in an unambiguous way, and we don't have to learn a locally crippled natural language for each new type we meet (as we do today for each new software, forget the text-to-speech part, any user interface does just this... not to mention the famous example of the new Office package).

I think it is possible that with Esperanto we can actually do anything in this information system. We can take a totally new component, understand its features, query its state and use its function, all without any additional user interface, just by using the fact that we know Esperanto (the "human script language of Dust"). The essential simplification and clarification between our thoughts and the machine understandable, unambiguous commands - and back: building ideas from the received unambiguous information is done in our brain, using our own, internal natural/artificial language translation. In this scenario, all parties do what they are the best at.

The last sentence: the business logic is now implemented in a programming language - but as I have already mentioned: a source code is nothing else but a more human readable form of a forest (mathematical graph term) of construct nodes of that language (declaration, value transfer, decision, call, repeat, ...). This is to what the compiler/interpreter transforms the text files, and then executes it (interpreter) or further transforms it to machine instruction code arrays. Thinking in this way, the source code itself is equivalent to an entity hierarchy, and any entity hierarchy can be transformed to Esperanto sentences, so perhaps this is the end of programming languages as well: we only need the declaration of the construct nodes to express just any action using them. (Of course this is not among the first targets, just a strange idea.)

Entities 2

References

The fundamental difference of Dust is that it considers the software (all of them without exceptions) a state machine. The software is built up from components having many attributes and connections to each other, but both the attributes and the connections are managed and kept inside the framework runtime environment, can be accessed and managed by framework API calls. The software code on the other hand is responsible only for handling events, and not for representing the system state. That is: no "main" function, no waiting cycles and polling, and no final and static variables, local buffers for storing system state information.

Whenever a code (an internal event handler of a component instance) wants to access another component, it can do so only through dust API calls, and only by a component (Aspect) reference that it has received from the incoming message, its own existing component links or by searching for entities using API calls. The code has a very short life: runs only to respond a message (and not to represent a "listening state" for example), and has no need to "remember" any component or state (except for storing such information in its own or other components' attributes).

This means that the Dust runtime is free to handle the actual entity instances behind the API wall as it wants. It is totally irrelevant if the instance is purged from memory and reloaded, replaced with another instance ("dll hell"), exists or temporarily moved to another computer - as long as its attributes are reachable and messages can be sent to it using their references.

Entities 1

Entity versus Aspect

Separation of these terms is fundamental in Dust. The Entity represents the "existence", be it a message, a software component, a real life object or person. The Entity has a life cycle independent from the computer environment, and its computer representation must be synchronized to reflect those changes - this is why we have "information systems". Behind all operations Dust manages entity instances in the background; all information packages come and go in Dust in Entity instances.

Aspects are "objects" of declared and managed types having attributes and implementing business logic (that is: processing and responding to incoming messages). An Entity instance has a collection of Aspects, which collection may change along the life of the Entity - so the Entity itself has a history of changing Aspects, while the Aspects have history of changing Variants, receiving and sending messages.

In any business logic (software code) we can only see Aspects. We use and communicate with these feature collections, not the Entities themselves. Every Aspect instance, where my business logic is integrated, have the list of required service components, with which I have to communicate to do my job. These links are defined in my type, declared when the application configuration is created (deployed to the actual running environment), and resolved when my entity is loaded. In fact, most business logic does not have to get and store entities dynamically, variable content come with the messages that it has to process.

Knowing this, I assume that the business logic code should never access entities, nor aspects directly, through a memory reference or object instance. The Dust kernel API therefore can totally separate the "user" business logic from internal memory and data management, and this is a fundamental step towards security and reliability.

On the other hand, there are special components that have to see through this layer: the generic services like expressions; automatic GUIs that are generated to an Entity by listing its Aspects; or serialization. The idea: i should create Entity, Aspect, Field aspect instances! The component can request them from an incoming Entity instance, and through them it can manage those generic operations. With this trick, there is no programmatic or API difference between kernel and user level code, no new "protected" kernel functions. Of course, the operations must be protected, but again with the same tools as all other Dust security features will be implemented, with the same declaration features.

Working with data 3

Collections again

It is very hard to get rid of old habits and choose a more complex-looking solution (which is complex only until we peek under the hood of the programming language where the problems are hidden).

I thought (and programmed in this way many times) that collections are generic part of data management. Again: they are not. Collections are aspects themselves, they either are part of the owner entity (like: subpanels of a window, scheduled commands of the scheduler, etc.) or referred internal entity members. Yes, the last time I have said that I did not need a collection directly, but only the iteration/search function - but forgot to mention the term "temporary". It is true that I don't need temporary collections, but the object themselves do contain many collections. If they are hidden under the "variant" layer, I have returned to the same problem of temporary collections, but under the hood with expressions for example.

So: Variants do not have multi-value content. If there is a need of it, it is a reference to a Collection aspect or entity. When expression evaluation encounters such a reference, that is again an accumulate / broadcast / search action (and reintroduce the parallel execution on this very low level).

Serialization

When different nodes communicate, they need to serialize their content. This component is responsible for working with references (independent of the actual implementation or the syntax of the stream). The generic solution is that each serialization action contains an ID map, where the unique id of the entity instance and a local, action-unique id is connected. The local id is used all the time, while the content is pushed into the stream only on the first call, when the item is registered into the map. In this way both sides have the simplest way to write and read the self referring structures properly.

Very important: the mapping must be set when starting the serialization: the referred entities may contain back references to the entity being serialized. They must receive the local id only, not start a recursive serialization process. On the other hand: anytime a local id based reference is resolved, the instance itself should be considered "final", but may be incomplete, and its content should not be used at this time. The read process should end with an independent initialization action after all referred components are read.

Content in serialization

From the content point of view, we have static and dynamic serialization.

Static is when I want to store and "remember" something as it is, like a configuration file: I want to keep some settings... NO! The config files are actually the primary source of the stored instances! Hmm... This means that for persistent entities, I have to serialize the content only if that serialization is actually the primary storage for those instances. All the others must only be stored by their unique identifiers (global type Id plus the identifier there). Of course, static serialization cannot contain temporal instances? NO: in some cases, like logging, anything, including temporal instances, must be stored persistently.

Dynamic serialization occurs when I send some content to another active node, which can ask back for additional information as well. In this case, the content of the stream should be optimized by the number of other request that the target has to make when processing the stream content. Some of the referred entities it may already have; others it can miss and ask back for. To cut it short: the sender may add the content of any entity instances to a dynamic serialization, even though the target is not the primary source of that information. This happens if the aim of the communication is in fact to get that instance (like requesting a Person record from a node that owns them), or that they are required to understand the response (sending Address, or Medical records as well with the Person).

It is possible that the sender keeps a sort of log about which objects is had sent to the requester, because it is responsible for keeping them in sync with the actual state, or even add event communication to them (later, this is screen sharing and active teamwork), or at least to optimize the serialization content: send only if the information is not available on the requester side. Of course, this is just an optimization: the requester is allowed to reload all content again (the client may be restarted or lose any content independently from what the server knows about it).

Format and type negotiation

When thinking about connecting to other nodes, we have to keep in mind that the endpoints may have different feature set. The most important is when the client is limited, like it has static type management, so it is simply unable to load a new type to understand the content of the serialization stream.

Level zero is to be able to build a stream connection and be able to transfer bytes through that line reliably. This is a lower level requirement that has to be solved with implementing the proper connector components; as long as I work with Java or other higher level languages, this should be a no issue. However, it also means that I must pack the stream operations into units as well, otherwise I might surprise myself when moving to a more limited environment.

The first level is that both ends must be able to parse and write the stream, that is: be able to handle the actual syntax. This means having Serializer component implementation, and the language elements for that syntax. This is a reference to an existing declarative information package, the EBNF-like declaration of the JSON language, which is actually used to parse and generate JSON streams (and in the hand of the Serializer, a valid JSON serialization) works just like the type declaration themselves. They are persistent entity instances, can be referred to, etc.

So, the limited client has the JSON language definition compiled into its codebase, and the client introduction contains the reference to that instance along with the statement that it is a static client, and cannot handle additional stream formats. When a stream connection is made, this information is used, and the more dynamic node can adapt to this requirement, optionally download the required language definition.

The second level is the content type set. Again, a static client may have final set of types, perhaps encoded again. This is a more generic limitation affecting any environment that is unable to load codes and object dynamically. For example the GWT environment can adapt to additional stream syntax, because if it has the generic EBNF code set, a new syntax is just another structure made from them. However, the GUI declaration types are final, a "Table" type is mapped to one specific code; it is not able to understand new, derived Table types. On a really static client this is also not a question: it is not even able to download and integrate new Type definitions.

So, a limited client introduction can also contain that it has a final type set, and the identifiers of the accepted types. The dynamic sender must "downcast" its content to the required types, and skip aspects that are of types unknown to the client. Naturally, this may result errors because of the lack of required references - the sender must handle these situations. But at least, they are visible...

Working with data 2

Some more meditation on working with data... not surprisingly, they are restrictions.

Floating point

Floating point values are nice, we have the normal mathematical operations with them and also tons of functions like trigonometry, root, etc. As I have just learned, some CPU families don't support floating point operations, and the math library provided with them actually emulates the functions with complex library calls. In a real time system this is not an advantage. Thinking on it further, even a current PC processor should not run floating point calculations because it has a way stronger neighbor there: the GPU...

So, floating point variables should not be considered "generic", but wrapped into a unit, and when deploying an application on a PC, we can consider that as a heterogeneous parallel processing environment, and let the GPU do the calculations. (I don't know how to do this yet, but I am sure that this would be a real advantage). Also some real time systems where floating point calculations are available but not hardware supported (therefore not recommended) can simply declare that they are not compatible with floating point using units.

Collections

Now, collections are wrapped into intelligent Variants that helps message communication, locking, etc., but there is one more thing. The collection should not be available directly, as an independent object or unit. In several years of programming I have found that I actually never use the collection as an object; I iterate the members (sometimes filtered) to collect some information or initiate actions on them. Therefore, I should support the real requirement: the basic functions: isEmpty, count, contains, insert, remove, clear, ... and the iteration itself, either by requiring callbacks with the items or broadcast the same message to them (or some of them based on a filter).

Without this temptation I will not waste memory on creating (sometimes huge) temporal collections just because I want to iterate the members. This also is a kind of "inversion of control": it is not me iterating a collection, but an iteration runs somewhere and I get a callback with each item. If the environment supports it, this can be a parallel operation either for fetching the data, broadcasting a message to them or calling me back with them.

Strings

Strings are basic language elements in most environments, but that is completely wrong. I have to write many string manipulation codes (like searching in them, building them, etc.), but if I look closer, I just repeat features that are available already: in regular expressions/parsers, template engines. Apart from this ad hoc transformation, I just take strings from one place and move them to another - even I take them from source code (or some language provided container like the Java properties file), and write it to the screen, to a log, database (or System.out...). I introduce problems like encoding, language dependency, different formatting standards, etc.

No. Strings (and its big brother, formatted documents) must have their own unit with proper (and black boxed) support for encoding, I/O, templating, etc. and the code must have only one way to deal with them: through their Units, as entity instances.

There is one exception: the Identifier, which is an array of plain ASCII (1 byte long) characters, used for identify components. They have only basic functions (equals, concatenation, find, ...) and all hardware environment can support them on hardware level. The Identifier gets a separate Unit apart from String, used by dust core as well.

Running environment capabilities

With a nice declarative framework one heavy-weight problem is when it hits the hardware and must run on it, and somehow meet the promise of being independent from it. Well, it does cause some problems, and it is again with data.

One is the actual coding. We either have a runtime environment, where the runtime declaration contains that "for Java language, the long is 64 bit". Or 128. It has native support. Or not. Depending on the actual hardware and the runtime version, and perhaps the runtime implementation: someone can make a Java runtime with different features: it actually runs Java code, but the code behaves differently. I don't say that this is an issue of a generic programmer - as long as (s)he is working in one environment and comfortably far from its limitations. But when thinking about designing and coding a Unit which is able to do its job in a Dust environment on practically any platform, this is a tough one.

When working with generic data, we must precisely declare the minimum storage requirement, and so the valid value range (of course by using the common sense terms for it, but perhaps going down to the good old assembly notations);and when generating or writing actual code, we must keep the names, but use the generic data types with proper (perhaps larger) capacity. We might also include range checks, and throw runtime exceptions on overusing the variable: that code works in the current environment, but it does not comply with the type declaration. Of course we can't check the temporal local variables in the running code that will fail on a more limited environment, but at least we can do the check when setting a Variant.

The problem arises again when we have to communicate with a running code, exchanging actual data. This happens on persistent data I/O and on sending messages to entities residing on a different node. The channel (either a shared memory area, network, serial, ...) must have portals on all sides, where the node capabilities are available (endianness, encoding support, etc). The negoriation must take place when configuring a static channel (at deployment time) or when building a dynamic channel (like accessing persistent instances on the specified server node). In dynamic case the node performance must also be considered, like a real time embedded node is not able to negotiate on the channel content, the stronger node must format its content as the weaker can handle it.

Working with data

The keyword here is reusability. I want an environment with totally independent parts, the same components for expression evaluation, interpreting an algorithm or accessing object member values from code; I want both string based access and compile time / run time support for type and nae checking. See how this goes.

Identifier

All starts with this. The Identifier has a short, limited ASCII "name", which must be unique in its context; and a reference to this context, which can either be a type declaration or an actual context (like a code block for interpreter environment).
The Identifier instance can be requested from its context by referring to its name, so it is unique, and adds the context information to the string name, therefore name collisions can be avoided.

Variant

The Variant is the core data element; a wrapper that holds the actual data value. It supports

• generic values like boolean, integer, floating point values and fixed length ASCII strings;
• single object reference to other objects;
• collection of generic values / references. The collection has flags of sorted: if the elements have a fixed enumeration order, and single: one element can appear only once in the collection.

The Variant also has a VariantDeclaration reference, where all meta information of that variant instance is stored. This means that the Variant itself is repeated for all actual values "anywhere"; they can travel through expressions, contexts, etc., but they always hold a reference to a data structure that can define them and the actual value can be understood by them.

Another idea: it holds a reference to an DataObject in which it lives AND a definition index - because in many cases I must go "up" from the Variant to the owner DataObject. And the final blow: the Variant has direct access and modification interface (to be used in expressions and interpreters), but is must invoke the change listeners of the owner DataObject, if exists. This requires DataObject reference in the Variant.

VariantDeclaration

This contains the variant identifier and all other information (type, access, life cycle, etc.). The declaration may come from a type or direct request in an interpreter. The important part is that the declaration is generally repeated for several Variant instances, so it is lifted from them.

The life cycle information (like "already set" or "final") may belong to an object instance - this enforces that the Variant should refer to the DataObject with a declaration index and not the declaration directly.

DataObject

The DataObject contains an array of VariantDeclarations and an array of Variants; it acts as an Aspect instance inside an Entity, but also the context for a running system, or the code block stack in an interpreted algorithm.

To mix flexibility with memory optimization, the most used DataObjects (Aspects) start with their declaration array initialized with the array from their type, which is immutable shared array instance, and sufficient for most cases. However, this instance may be referred from other aspects, resulting a reverse reference to the referring aspects of alien VariantDeclaration. When this thing happens, the array is copied to a mutable declaration array, and the new VariantDeclaration is added to this DataObject together with the new Variant. 

The same applies to the generic DataObjects acting as context: they have no initial declaration array, but extended on each variant declaration request. The only difference: the declaration objects are themselves can be local strings, in this case they must be unique in the actual context. However, types may contain "shared" member declaration, which means they are not contained by the Aspect instance but the identified Context (this feature implements the "static member" feature of the programming language, and extends it by enabling different Context levels like runtime, session, user, ...)

Field

Field is used when we access the Variant from the DataObject; to do so we need the object itself, and an Identifier - which is not just a name, but a context as well. It is also important that the Field itself is more locked to the Identifier than to the object: it is fine to access the same Variant in multiple objects through the same Field instance, but it is not good to switch a Field to access another member Variant of the same object. Consequentially, the Identifier is final, while the object is mutable - although you can get a Field directly from an object by providing the requested identifier, but the field can be reused for other objects that have the same member.

The Field inside is responsible for removing the identifier based string access. It has an object reference, and a VariantDeclaration index inside. When (and only when) you call setObject with a new reference, the identifier is searched in the declaration array of that object, and the index is stored. Whenever you access the content, it is just an indexed access of the variant array of the object behind the field.

Fields are mostly used in declarative "toolkit" components, where the actual context "knows" the required type, the field is identified by the declaration name within the type, and you can manipulate the content through the Field instances. Most of such environments work with multiple object instances of the same type (like a GUI panel, a string template, a table, etc.) for which the Field's final lock to a specific declaration is also a good feature.

FieldSet

A helper component for dynamic access environments: you provide a type and an array of strings, they are internally transferred to an array of identifiers and Field instances, and so you have an indexed access to the values, where the index is in sync with the index of the field name in the string array.

TypeWrapper

The heavy-weight wrapper for an aspect. This is a generated code from the type declaration in the target programming language. It contains a FieldSet by the type, and a reference to an actual Aspect instance, typed get/set functions with type casts by the variant names. The TypeWrapper actually looks like a "normal object" for the programmer. For each Unit declaration, the wrappers can be generated, and used by anyone who wants to use the objects from their native code. The important feature here is that the actual data binding between the wrapper and the data object instance is done runtime, by the declaration and identifiers, not any fompile time fixed tables. This really allows extending the types without breaking the caller code.

It also contains static accessor functions, which are translated to getting and casting the referred items from the declared context (like "Logger.getLog" returns the log object in the closest context: transaction, session, system). The wrapper also incorporates message sending, but this is another story. When working and writing code on this level, it should feel almost like coding without Dust in the background.

Dynamic type management and the application

The fundamental structure of any application is the deployment configuration. A GUI panel that is used to communicate with the user; the runtime environment that transfers hardware events (mouse movement, keyboard keys, etc.) to GUI control actions, then data modification - they are just the same kind of "data" and attached business logic components and implementation as the "Person" record displayed on the screen.

So if the application is anything at all, it is a deployment configuration about what components should be initiated and how they should be wired together with message channels to provide the required functionality. How should this deployment look like? (Of course, the kernel part of this deployment should exist in the form of compiled source code to make the kernel work and be able to load the other components.)

Application types

From structural viewpoint, an application can be static, type dynamic and binary dynamic.

• A static application is one that has all the types and binary codes available compilation time. This application will be able to deal with a fixed set of data and components - like most of current software do. 
• A type dynamic application can deal with an open set of data, like a generic view that is able to load and display any document in a document management system, without knowing all the document types at compilation time. This can be achieved by a sort of configuration-based data management, and template engines. 
• A binary dynamic application can integrate software components at run time that were not known at compilation time. This is a plugin based environment, a typical example is the Eclipse plugin framework.

As the other types are limited subsets of binary dynamic, I plan that one; the others are created by replacing dynamic service components with placeholders providing the configured set of data.

There are also types from memory management point of view (instance-based, pooled and free), but that is out of scope here.

Vendors

The fundamental question is: where the types (and the associated business logic) come from?
They come from the vendors. A vendor is a software designer or implementer organization, who analyzes different problems, offers data structures and attached business logic codes bundled into units. These units can be used as building blocks for providing a specific service to a user. Therefore, all deployment configuration starts with a vendor list. The list contains the unique vendor ID, to which later the units refer to; they provide those types and binaries that this application is built upon at compile time.

However, dynamic applications must extend the list of types or even get new unit binaries as they receive corresponding data items. So, exactly one vendor must be identified as vendor provider, and provide a way to access that vendor server, because it will be used to get more vendors (data can refer to vendors unknown to the application at deployment time), and get connection information to them (to reach them for type declarations and perhaps attached business logic binaries).

In this way, the application can start with a single root vendor (in a generic case, a public server of the creator of that software), with a limited set of units enough to boot, and perhaps to provide the basic functionality. The generic usage of any application is browsing and managing a data hierarchy (be it a word processor, a mailer, a company information system, etc.); the root vendor provides the additional components and the referred data, controls the access to the external links, etc.

A generic dust application refers to the public dust server which returns any registered vendor and allows applications to be built on various components created by them. The actual vendor may choose to allow the application link to dust directly, catching only local references; or act as relay to dust and other vendors to control external access from the application. It is important to know and handle the security risk of getting types and more importantly, binaries from external providers - on the other hand, knowing about and generalizing this access may lead to more stable environments.

Units

Units are the building blocks of any service, the smallest coherent component with the type declarations and the attached business logic. The Vendor can work on multiple areas, which here are called Domain, and a domain can contain multiple units. The unit focuses on solving one and only one task, perhaps in collaboration with other units. If the unit has multiple areas of interest, it has to be split by them, because some other service will later require only some of the areas, not the whole package - so the unit is better when smaller. Featuritis is not welcome here.

Having small units, it is easy to extend the system with new functions. If we have a proper minimal raw byte stream implementation with all higher level features (parsing, event management, caching, etc.) implemented in other units, it is very easy to create new stream units that wrap different network protocols, serial I/O, ... audio, ... interfaces.

The application deployment contains a Unit dictionary with a unique identifier and the actual unit information: the Vendor.Domain.Unit access path, and the actual unit version. The rest of the application declaration can refer only to the types available in the dictionary, using their local IDs. In fact, all persistent data collection, from an application configuration to a relational database behind an information system must contain this unit dictionary to be usable. When creating such collections (a saved document, a configuration file), the unit dictionary must exist in it, that allows later reading and resolution of the content.

All Vendors must be able to resolve its own path+version information to a unit declaration containing the external unit references ("imports") and the type declarations, which must be complete, all references resolved either locally or through the import. Of course, the user of the unit must also resolve the referred units, but not limited to the list that the creator used (the unit requires "a binary stream" which is resolved to a console in one and to serial I/O in another case).

The Vendor may provide business logic to the Unit. For Java, this means Java classes that implement the message processing of an Aspect of the specified Type; compiled and packed into the unit jar for the required JRE version. Right now I will only support J2SE6, but for fun or when required, it can be done for other versions. Later on I should also turn to C or other runtime environments; and a fully declarative algorithmic approach. Not right now, and not for production use.

Types

Dust types are created to extract the design knowledge from a particular programming language to a higher declaration level. "Objects" in Dust are split to the hierarchy of one single Entity (which reflects the mere existence of the represented "thing") and a collection of Aspect instances (what is that thing, what service groups it belongs to). The actual data structures and behaviors are associated to the Aspect instances; each Aspect has one Type.

The type is responsible for controlling the memory containing the data of the Aspect, the serialization, the access, rights and life cycle management (protected, final, etc. fields, relations). Therefore, any type is fully functional when the framework has its declaration, with one limitation: it has "variant" interface. The advantage of putting the programming language away (and using Maps and other objects in Java for content management) has this drawback. I should check Java annotation feature if I can get some of this support back, and have compile time type checking for example upon a generic access function.

On the other hand, this generic access allows creating a type dynamic system in environments where binary dynamic solutions are not available. For example, if I would support types through generated Java sources, they can only be accessed by extending the class loader, would need Class.forName() functions and reflection for generic access of the members. These are heavy-weight requirements, and not supported for example in GWT (which is the planned reference web environment for Dust). This is why I focus on "plain" declarative access and generic data container objects. In previous works (LogMon) I could easily avoid using string based maps for attribute access (having index-based mappers instead), with acceptable performance.

---

These features seem to be enough to initiate a Dust based environment for my current tasks.

Accessing data (and the betrayal of informatics)

What we do is called informatics, we are supposed to build information systems. The tools we use for that are the computers, which can process and transform the data, and the way we instruct the computers to process some data is programming. So we are programmers. And all that we have to do our task are tools for programming: different (and numerous) programming languages, different (and devastating amount of) tools to process a certain type of data in a certain way. Finally, we implement software: a collection of algorithms that can be executed in a specific environment: hardware, a certain set of operating system version, or higher level runtime, using a specific set of tools of a specific set of versions. All that we have investigated and found out of data is burnt and hidden into this implementation and execution environment. When it changes, and for all the other, simultaneously existing environments, we have to do the same thing again and again.

If we look at it from the business viewpoint ("work to get money"), this is good. But from any other aspect, ("work to solve a problem in a way that it also remains solved for a considerable amount of time"), this is totally, fundamentally wrong! To correct it, we should return to the basics.

We are supposed to build information systems. The key is the piece of information itself, not how, where, when we process it. The information system is a parallel universe that lives synchronously to the material world; the information system should hold a "shadow" of existing things - exactly one, unique instance that can follow that material (like a person or a car), or conceptual (like a calendar) entity. Dust Framework aims to support building this information system by extracting the concepts, types, pieces of information from the actual implementation environment to a declaration level. It is not dependent on any external framework, only upon itself: the meta-language of declaration (basic terms and their relations) itself is just a declaration, can be corrected or upgraded on need. The actual runtime services are atomized and therefore are wrapped into several black boxes. The representation of this content is also irrelevant: the same entity hierarchy can be stored in a database, LDAP server, in the structure of a file system, or in files of XML, JSON or whatever format.

Data access

The concept is nice, but has very complex consequences. For all execution environments, "data" is a tagged location in the system memory (temporal data storage area), and all we actually do is instructing a piece of hardware, the processor, to do something with the information at that location (read, test, change).

However (except for local, temporal working variables), "data" is not just a location in the system memory: it represent a state of an object in the external world. That object might be visible in other execution environments in this very moment, it might change as I watch it, and it may live longer than my environment (may have existed before I switched my machine on, and will exist when I switch it off). Accessing this piece of information is definitely not that easy as a processor instruction. It is essential how I get, lock and commit (or rollback) this data.

Get data

Lock

Commit / Rollback

Entity meta-types

With the declarative tools I have to go to the end. Finally, the business logic, which can be implemented in a programming language (right now C is the preferred choice), can only do the following:

• access connected other instances: the connection itself is done through a local ID mapped to the target instance in the deployment
• access values through a variant interface
• send messages to the connected instances (targeted messages)
• send messages to the component's own broadcast channels.

Question: do I need the programming language at all, if its use is reduced to this level, and the above actions can also be used in declarative environments as "action nodes"? I could create the tree of action nodes directly, the same structure that the compiler actually creates when it parses the source code - and I could also parse source code to this structure, and play it in an interpreting environment with the above functions.

Answer: the lowest level, architecture dependent layer has to be programmed directly. I should identify the components of the execution layer (like memory, thread, "class loader" etc.) declare the same way as the lowest level data components (type, field, entity, aspect, channel, ...). The core functionality, process path has to be declared and played using the core items to find the terms and sentences of that layer. In this way I will have a configurable, segmented running environment instead of a monolithic "runtime". Then I will be ready to write that lowest level code for one architecture (be it Java, an OS or else).

When dealing with the abilities of the environment, I must forget actual programming languages and their features: I have to focus on what I want from the environment and why I want it. If I find the right approach, I can be sure that the actual language will help me in implementing it. So, what to support?

Exceptions

Components should not "peek into" the deployment path. When I require a service, I don't know how it will be implemented in the actual environment, like I want to access an entity, and I don't want to know that my environment resolves a storage address and requires the instance at this moment via HTTP requests from multiple servers that execute DB or LDAP queries for me. All points may fail due to different reasons; some should be handled and post processed before reaching me, some should never reach me, etc. 

Therefore I should support throwing exceptions in the way of ordinal messages. The environment should immediately stop execution and roll back the operations until a catch is registered. The catch declaration should not break unit boundaries: a unit can catch its own exceptions and react to them in the finest detail (actual exception objects). The exception can be classified to different categories, and catch declaration can refer to these categories. This is a hierarchical structure, and for data I use the entity -> aspect* concept for them. Should I use the same here?

Looks fine. The server encounters an LDAP error, creates and throws an LDAP exception with all the details. However, this exception is private (security reasons) and should never leave the server machine (it is removed from the entity when it gets into an outgoing channel). On the other hand, it can be caught, and another (public) aspect can be added: temporal server error exception, perhaps with a short explanation and the estimated availability information.

Entity meta-type

Okay, so the Entity has a kind of "meta-type aspect", which can be Exception, Persistent Data, Temporal Data, Worker Component?

What is a worker component? That is like a configured parser, a stream channel, a GUI panel. But is it different from any other data? Its life cycle is limited to a session: though some of them "live" as long as the actual running hardware is switched on (the environment components); some are limited to a specific user action (GUI elements); some are even more temporal (event process nodes in a parser). But Temporal Data, the shortest living items (messages) need a life cycle management: should not be disposed when used, should be disposed when all user components finished with them.

One entity may have different life cycle aspects. A GUI element has Persistent Data components: the GUI structure and message path declaration is persistent, it is part of the service declaration; in normal case it is final. However, multiple instances (entities) can refer to the same declaration: multiple windows of the same service type. They are different because the Temporal Data section of the entities are different: actual size, location, language, ... Some of the Temporal Data section MAY be used to update the Persistent section (default location, size, language) - or the declaration can contain Temporal content as well that act as default. The latter is the better: less administration.

Is worker component limited to its actual location (machine)? NO. The Worker Component data can travel through a channel and appear on another machine (for GUI, this is screen sharing on declaration level). That would be a useless limitation.

So, the meta-types are reduced to Persistent, Temporal and Exception meta-types. It seems that references can go in any direction, independent from the life cycle levels, but downward references act as "template" mechanisms. An Exception may refer to a Persistent or a Temporal entity, and that is a direct instance reference: the actual item can be accessed through the reference. On the other hand, a Persistent entity may contain an Exception, but that is only a template, and is "torn off" from the owner (replicated) when used (updated): the server may contain a "temporal server error" exception with "try again=5 min" attribute; this is mapped to any exception sent from the server. But in this case, the sent exception is not "the same instance", the server setting may be modified, it will not affect the existing instances. The same way: the (Temporal) Location aspect in the (Persistent) GUI declaration may be changed (the next windows will appear on a new location), but this does NOT affect the existing windows. On the contrary: a change in a Persistent Data displayed on multiple GUI panels MUST be reflected in all of them.