What is a Data-Driven System? To better illustrate the idea of a "data driven" system, think of an application for a credit card (or possibly a purchase of an online airplane ticket). You have to fill out your name, address and other info: this is the "form". If its on the web, then you click a button and the values of these fields are collected and whisked into a database on the web server. Then, based on your responses, and some calculations, and some other data in the database, a new web page is computed, and the resulting "report" is displayed to you (this report may in fact be a form asking for additional info). The bouncing between forms and reports can be quite complex: to purchase an airplane ticket, you start by filling out a form with the dates you want to travel, and the place you want to go. The first "report" is a listing of flight numbers, departure/arrival times, and ticket prices. This "report" is also a "form": you can click on a checkbox to select a flight, and then proceed onwards to make a purchase.
In fact, there's a huge class of applications that are "data driven" in this same way: these range from "bug trackers" such as Bugzilla, parts and inventory trackers, customer trackers, on-line shopping systems, systems for processing government paperwork (such as getting a new water meter installed, or getting the building permit), financial accounting, purchase requisitions, and the like. You fill out a form, there is little or no computation done, except to save th data, and the result of filling out the form is another form that appears on the bosses desk, or on the shipping clerks desk, or on the loan officers desk, etc. This large class of applications share one common trait: they can be described in a mostly declarative fashion. Pressing this button causes data to be copied from here to there, with little in the way of hard computation.
There are two ways that such applications are typically written: hard-coded and arbitrarily flexible. Bugzilla is a great example of the "hard-coded" style: you can customize the list of projects that it presents, and a (large) number of other aspects, but, in the end, Bugzilla can never be more than a bug tracker. Now, if you could get Bugzilla to store price information, then you could use Bugzilla as a purchase requisition system: instead of opening a bug, you ask for a new chair; instead of assigned the bug to a programmer, you assign the chair request to the boss, who will accept or decline, etc. MS Access represents the "arbitrarily flexible" approach: you can design any form and hook it to any database table; if you don't like your first shot at this, you can start all over again.
Why is it so hard to write applications? There have been many contributing technologies that have helped to make this kind of programming easier; but they all have their shortcomings. We'll try to review some of these here. MS Access made a lot possible, but it was fragile, wasn't really multi-user, was ugly, and didn't scale well. It couldn't talk to other systems. CRM/ERP systems, such as those from Siebel or PeopleSoft, get around these limitations, but are incredibly complicated and stunningly costly. The web (HTML and HTTP) revolutionized the creation of these kinds of applications, by allowing the programmer to run the database on the same physical box as the web server, and thus gave the web programmer multi-user capabilities "for free". However, the web did not provide any sort of high-level or declarative language tools: one coded in perl or PHP. Creating a web application is a lot like creating a desktop application; one still has a huge amount of mucking-around to do; its just that one gets "multi-user" almost for free on the web, whereas "multi-user" is still hard for a typical Gnome, KDE (or Microsoft) desktop app. For this reason, and for this reason alone (and really for no other reason), (and really that's the truth, we pay no lip service here to other bogus theories here), almost all large, complex applications are being developed for the web, and not for the desktop.
There are two other types of technologies that programmers look to when trying to code these types of applications. Desktop developers have traditionally cast their eyes upon CORBA, (Sun) RPC, and SOAP as a means for getting multi-user abilities. The idea is that a desktop application, when properly retro-fitted with CORBA, RPC or SOAP, could talk in some kind of language to some server, or even share data in some P2P type way. Indeed, SOAP is an important part of the Microsoft's .net framework; it is an important part precisely because Microsoft is a desktop application heavyweight.
The other technology that programmers look to is innovation in programming languages. When one programs large, complex object-oriented applications for a while, one eventually discovers the limitations of traditional languages like C and C++. What I refer to here, in particular, is the idea of "object introspection". With object introspection, you can ask an object to describe itself, thus making it much easier to ship that object over the net (marshal and unmarshal), store it in a file, and etc. It is no accident that both Java and C# both explicitly support introspection However, although these languages are quite innovative in this regard, they offer no panacea, none whatsoever, to the programmer of data-driven applications. This is because, at their root, they are still highly procedural general purpose programming languages, coming from the same stock as C, C++, perl or python. They don't really make it fundamentally easier to create data-driven applications.
A huge premise of DWI is the observation that data-driven applications can be developed using a declarative rather than a procedural language. Lets take a moment to explain what this means. For example, int i; is a declaration: it declares that i is an integer; its static; it doesn't cause any computation to be done in and of itself. for (i=0; i<100; i++) is a procedure: its a set of instructions to perform a certain activity. Declarations need not be boring like int i; C structs and C++ classes are declarations as well, as are function prototypes. Although in the above examples, declarations seem static, they need not be; one can use declarations to describe something dynamic. For example, a signal handler that has been set up is a kind of a declaration: until the signal arrives, nothing happens. It is merely a statement that if a signal were to arrive, then the signal handler should be invoked. DWI uses declarations in this kind of a way. The programmer makes a declaration that this-and-such widget is linked to that-and-such database table column. When a signal arrives, for example when the user presses a button, then, and only then, under the covers, the contents of the widget is copied to the database. In DWI, there are no variables, no assignments, no for-loops or iterations. There are, however, descriptions of linked chains of events, such that a simple signal can set off a rather complicated chain of events that eventually result in the GUI being updated. Although, in the end, declarative languages are not as powerful as procedural languages, they are still stunningly powerful and flexible and capable. For example, a purely declarative language can still cause a column of numbers to be added or graphed, and can be used to search for all employees whose first name is "Bob" and whose salary is more than $10,000. The premise of DWI is that a simpler, less complex declarative language is enough to code up most typical data-driven applications, and that a good interface to traditional procedural programming languages pretty much takes care of the rest.
Note that XML and HTML are essentially declarative languages, and so is "richtext" and the Abiword word-processor file format. Part of the power of a declarative language is that it is easy to build a WYSIWYG editor for a declarative language. By contrast, it is impossible to build a WYSIWYG interface to a procedural language. Take, for example, postscript, which is commonly used to do graphical layout for printing on paper. Postscript is a procedural language: it is not hard to write a postscript program that will compute the first million digits of Pi. One cannot create a WYSIWYG page layout program that uses postscript as its native language: if you feed it the postscript for computing Pi, it would have no idea what that means. You can't change the font, color or position of Pi; it is Lewis Carol nonsense to even think of it. The statement here is deep: if almost all data-driven applications can be described with a declarative language, then almost all such applications can be created with a WYSIWYG editor. You really don't need to be a programmer, or write a single line of C#, Java, Python or Scheme code create these apps. You just need that WYSIWYG editor. This may sound like magic, but its not: the MS Access program is a crude, primitive form of this kind of WYSIWYG editor. Glade is essentially in the same ballpark: Glade is WYSIWYG, and in the widget signals panel of glade, you can hook up signals to (a very limited) set of actions. In a certain crude sense, the goal of DWI is to vastly expand the set of pre-built signals that can be hooked up to a widget using Glade.
Note that Microsoft is proposing a declarative language for Longhorn called XAML, for some of the same reasons that DWI exists. If you look at the XAML intro, it becomes clear that the XML generated by Glade repesents significant prior art (one can only hope the patent examiners see this). Note also, work started on DWI before Microsoft came up with XAML. If you look at the XAML Event sample program at the bottom of that page, you will see right away that DWI provides a far more sophisticated, flexible and stronger framework. I beleive that DWI, combined with Glade, is superior to XAML.
Note that the DWI design is fundamentally "declarative" in that it essentially is like a declarative language. For example, an "SQL Trigger" is a declaration: by itself, it doesn't do anything. However, it does describe what should happen in a particular situation. DWI is similar, in that it describes what should happen when a user presses a button. DWI is a way of describing the chain of events, how things get hooked up to things: a button press can cause the value of a widget to be used in an SQL statement, the results of which are used to fill in another table widget. This description or "declaration" of the chain of events makes DWI very very different than procedural languages such as C, FORTRAN, Python or perl. The point behind this is that its a lot easier to describe the chain of events, than to write programs in C or perl (or scheme for that matter). The 'ease of use' of DWI depends on it being declarative in nature.
The declaration of the chain of events is done using XML. (It might be interesting to use a non-XML way of describing the chain of events, but XML is adequate, if a bit klunky.) The "run-dwi" evaluator reads in this static declaration file, and sets up engine structures that correspond to the XML declaration. Then the evaluator goes into the main loop, and signals and events make everything happens according to the pre-determined definitions in the XML file. There is no run-time interpretation. There is no 'just-in-time' compilation. It just runs along the fixed paths, where each path had been pre-defined in the XML file, and each run-through is initiated by the user clicking or typing in the GUI.
A DWI "application" consists of one or more <database> sections, each of which consists of one or more <window> sections. The application windows themselves interact with the user as a <report>, displaying data, and/or as a <form>, collecting user input. Here is an example file.
A brief but important note about the words "report" and "form". These two words provide the key conceptual underpinning for the DWI XML file format. DWI distinguishes between 'data sources' and 'data sinks'. When one writes a DWI file, one declares the 'pipes' that run between GUI data sources/sinks and database sources/sinks. When a 'pipe' connects GUI data sources to an SQL statement, this 'pipe' is given the arbitrary name "form". The 'pipe' that connects a database record-set (query result) to GUI elements (sinks), this is a "report". For programming architectural design reasons, these pipes are always one-way. But, of course, at the user interface level, you want to have any given widget to possibly be both a source and/or a sink. So, for instance, I can have a GtkEntry act as a source for one (or more) "forms". That just means that the value of the GtkEntry is read, and used to formulate an SQL query. And I can have that same GtkEntry also be a 'sink' for a report: for example, a report that puts today's date into the GtkEntry. So any given widget can be a be a source or sink (or both, several times over, in a one-to-many relationship). However a "form" always inputs, and a "report" always outputs.
A DWI application contains a rudimentary mechanism for handling application state: a global set of key-value pairs (implemented as a hash table). You can create a key-value pair at any time, and later use the values in database queries or in reports. Most form-entry/reporting applications will typically not need to maintain state; more complex applications will typically use global state to map tree-like/hierarchical structures into the database. In addition to the global key-value storage, values can be set and retrieved on individual widgets (using the widget data field).
Whenever a value is obtained from a widget, with the intent of putting it into the database, or from the database, with the intent of putting it in a widget, it can be passed through a "filter". The filter transforms the value in some way, such as stripping out leading whitespace, checking for null, or putting it through a lookup table. Currently, only a very minimal set of filters have been defined; although more are envisioned. (Basically, what's there seems nearly enough, although a good date-format filter would be nice.)
Many Gtk and Gnome widgets can be used for user input and display, such as GtkEntry for string input/output, GtkLabel for string output, and GtkCheckButton displaying/getting boolean values. Most Gtk and Gnome widgets for which it "makes sense" to set or read a value are supported. An effort was made to set values that 'make sense': thus, setting the value for a GtkCheckButton sets the state of the button as checked or not checked, instead of setting the button label. However, setting the value for a GtkButton sets the button label, since it doesn't 'make sense' to set anything else. Note that some 'unusual' I/O widgets are supported: for instance, the window title can be set on GtkWindow, as well as the label on a GtkFrame.
Sometimes it is useful to set other attributes for widgets, other than the 'obvious' one. To do this, the GtkArg mechanism is supported, thus allowing any and every attribute on any and every widget to be set. Thus, for example, the height of a widget can be set by specifying arg="GtkWidget::height", and using a value pulled from a database. More useful is perhaps making a widget conditionally sensitive to input (grayed out or not grayed out), with arg="GtkWidget:sensitive". The 'testwidgets.dui' example shows a large variety of widgets, as well as the usage of args.
Note also that the GtkCList and GtkCTree widgets are supported for multi-row input/output. The GtkCList ("columned list") widget is a basic, simple table, and is 'automatically' filled by DWI. The GtkCTree widget is a hierarchical columned tree; its a bit harder to fill out, as each sub-tree requires a separate database query.
Currently, DWI ignores data types, and treats everything as a string. This may change in the near future. Since SQL database table definitions are typed, there is a natural pressure to put this kind of typing into schema designers and the higher level app (such as MS Access). So far, however, there seems to be no particular need for typing within DWI. So we've resisted it.
"Meta" operations, such as table browsing, are supported, but barely. Other operations, such as dynamic form generation, are currently not supported. These "meta" operations are needed if one wants to build a database browser (or a graphical DWI designer!). Almost all "normal" data-driven applications do not need these features: tables and forms are predefined, and are tailored for the application. However, because they are needed for a DWI designer, they are mentioned here. The current plan of attack is to convert all DWI internal structures to GObjects, so that they can be treated as if they were widgets themselves. This adds a fair bit of mind-bending abstraction to the design and use of DWI, and so we are proceeding slowly to minimize mistakes.