Difference between revisions of "Roadmap"
(Add generic importer.) |
(→Eliminate deprecated widgets and libraries: Update to Gtk4) |
||
| (41 intermediate revisions by 12 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | = Overview = | |
| + | Below are the current major changes contemplated or in progress by the development team, roughly in priority order. Other proposed improvements of lower priority may be found on the [[WishList| Wish List]], but note that the development team believes that the work described here is a prerequisite to most everything found there. | ||
| − | = | + | = Replace GObject and GLib dependencies in the core = |
| − | + | Various parts of GnuCash use no objects, GObjects, and rather tortured emulation of objects in C. We have decided to reduce our dependence upon GLib, at least in the Engine and Backends, and to rewrite that part of the program in C++. In order to minimize external dependencies in that part of GnuCash, every effort will be made to depend only on the C++ Standard Library and [http://www.boost.org/ Boost]. We hope that this will make it easier to port GnuCash to other display platforms and GUI libraries. | |
| − | + | More information, including some suggested reading if you're not fluent in C++, may be found on the [[C++]] page. | |
| + | |||
| + | = Scheme minimization = | ||
| + | |||
| + | There are some parts of GnuCash that make a round-trip into scheme code for no really good reason. The developers would like to throw out those parts of Scheme. | ||
= Register rewrite = | = Register rewrite = | ||
| − | The current register is | + | The current register code started out as code taken from the [http://www.gnumeric.org/ Gnumeric] spreadsheet design, greatly extended and customized for GnuCash's needs. This is central to GnuCash's look and feel with the basic view resembling a checkbook register while expanded views provide more detail about transactions. |
| + | |||
| + | It has been partially rewritten to use GtkLayout with a GtkEntry as active element. However the controller code is still mostly as it was. This doesn't map very well with Gtk's View-Controller philosophy. In particular, the code jumps through several hoops to map text entry between the GtkEntry and the controller code. | ||
| + | |||
| + | As a first step it would likely simplify a lot if the GtkEntry would be the sole responsible of all text entry (where now this is handled in large parts by gnucash-sheet, extended with control code from the abstract register implementation). Only keystrokes that can't be handled by the GtkEntry (like those for navigating around in a register) should be delegated to the underlying gnc-sheet. | ||
| + | |||
| + | A rewrite using GtkTreeModel was begun in 2.5 but wasn't completed. It is still in the code base and can be enabled during build configuration. While it addressed the obsolete library problem and would ease conversion to Gtk3, it is still tightly coupled to Gtk+ and so precludes building UIs with other libraries. Moreover as it's no longer maintained it's possibly no longer working and missing newer features that have been implemented in the current register in the meantime. | ||
| − | + | In addition, Gtk4 is moving away from GtkTrees in favour of GtkGrids. If a major rewrite is to be considered beyond GtkLayout and GtkEntry, it would probably be wise to evaluate the newer options Gtk4 has to offer. | |
| − | + | = Eliminate deprecated widgets and libraries = | |
| − | = | + | GnuCash still uses several gtk widgets that have been marked as deprecated, many of which are not supported in Gtk4. Update all of these to API shared between Gtk3 and Gtk4. |
| + | |||
| + | = Remove module system = | ||
| + | |||
| + | The current module system confuses runtime-loaded modules with shared libraries. As well, it is (arguably) at the wrong level of granularity in places, leading to too many modules. The developers would like to replace most of the plugins by one common library or static application, leaving only 2-3 really optional parts as runtime-loaded plugins (e.g. aqbanking). | ||
| + | |||
| + | = Model-View-Controller Pattern = | ||
| + | |||
| + | While most of the "model" ''is'' properly encapsulated in src/engine, some has leaked into other parts of the program. Worse, there is little separation between view code and controller code. We need to separate the two to make it easier to implement GUIs with other GUI libraries. | ||
| + | |||
| + | = Testing = | ||
| + | |||
| + | Most subdirectories have a "test" directory containing tests of varying quality, but the coverage is poor and most of the tests are high-level tests that exercise the full module contained in the directory (for example, the dbi backend tests create a data structure in memory programmatically, then save it to a database and reload it and compare the reloaded data with the originals). These are good tests, but coverage is spotty. (Not all of the ways GnuCash stores data are covered in the dbi backend, for example.) | ||
| + | |||
| + | Modern practice calls for "unit tests" exercising every execution path in every exposed function in a tested module. Maintaining good unit tests during development has been extensively documented as a way to improve both productivity and quality. GnuCash doesn't have many unit tests, but it should. | ||
| + | |||
| + | Adding unit testing (with the [http://http://code.google.com/p/googletest/ Google Testing framework]) [[Testing |is in progress]]. Older tests written with GLib testing or the legacy test code in src/test-core will be converted to Google Test and extended further as part of the C++ rewrite. | ||
| + | |||
| + | = Database and QOF = | ||
| + | |||
| + | Now that we have the DBI backend supporting not just Sqlite3 but Postgresql and MySql as well, we want to move away from a "load everything into memory at the beginning" model to "load only what you need, and take out a backend write lock only on what you're editing" model to allow multiple simultaneous access, to speed up loading large datasets, better data integrity protection, and all of the other benefits provided by a good database. | ||
| − | + | The current "Query Object Framework" that GnuCash uses rather gets in the way, because while it handles queries acceptably well, it doesn't offer any of the other services that a proper database does. So the "QOF" parts should rather be thrown out. | |
| − | + | Some thoughts on what this requires: | |
| + | * Transactions and splits are by far the object types which cause the most problem. | ||
| + | * Some other objects contain a pointer to a transaction or split (e.g. an invoice contains the pointer to the posting transaction). Either the transaction needs to be loaded when the invoice is or else it needs to be loaded when it is first referenced. | ||
| + | * Some reports generate values by creating a query on splits and then summing them. Given a database engine, a report should be able to query the value of splits for an account or set of accounts on a specific date (with possibly other qualifiers as well e.g. reconcile status). A database engine can do this natively, C code could do it while we're converting, but the API should be defined. | ||
| + | * The set of splits for an account actually is modified with the running balance. xaccAccountGetBalanceAsOfDate() uses this fact by updating the running balances and finding the first balance on or after the date. Instead, the same db query used to get starting/ending balances should be used (sum all splits where account=desired account where date <= desired date). The running balance should be kept by the register because it should be recomputed if the register is resorted or filtered (would close some bugs). (bug #591098 causes the "present" column on the account page to be 0 until the account register is opened if not all transactions are loaded at startup). | ||
| − | + | ====Proposal==== | |
| + | * Replace QofQuery with SQL queries to the SQL backend. XML files will be loaded as before, but instead of populating Engine objects it will be loaded into an in-memory SQLite3 database to moderated by the backend. Autosaves and end-of-session saves will be exports from the SQLite3 in-memory database to the XML file. | ||
| + | * Non-editing GUI elements and reports will query the SQL backend instead of QofQuery. No Engine objects should be needed for queries. Non-editing GUI elements will take out read locks (so that multiple users can read the same database tuple). Only when a user edits should Engine objects be instantiated; when they are, the corresponding tuples in the database will be write (exclusive) locked. Note well that this will require careful deadlock management, because one cannot get a write lock on a read-locked tuple. Reports will run queries and will not use cursors or acquire any locks. | ||
| − | = | + | = Reports = |
| − | Right now reports are | + | Right now reports are Scheme scripts that programmatically generate HTML. |
| − | + | Scheme is impenetrable to most programmers. Expecting users to be able to write reports in Scheme is completely unreasonable. | |
| − | The | + | The ideal solution should have three modules: |
| − | + | * Record selection: Ideally graphical with an SQL-like "advanced" option | |
| + | * Layout: The original item here suggested an HTML template; that could be the "advanced" option, with a simple table/graph being the default | ||
| + | * Style: [[CSS]]. It's built into WebKit, we should use it. | ||
| + | * Javascript: WebKit supports Javascript so complicated interactivity can be added. Example: ability to expand/collapse levels of the account hierarchy in a report. This could be extended with Javascript interfaces to the API so that all of the report code is written in Javascript instead of Scheme. | ||
| − | + | Any report module will still need some sort of scripting language to "calculate the numbers". Currently we have Scheme for this, but the developers would like to get away from that. Python might be a better option. | |
Latest revision as of 17:40, 18 December 2022
Contents
Overview
Below are the current major changes contemplated or in progress by the development team, roughly in priority order. Other proposed improvements of lower priority may be found on the Wish List, but note that the development team believes that the work described here is a prerequisite to most everything found there.
Replace GObject and GLib dependencies in the core
Various parts of GnuCash use no objects, GObjects, and rather tortured emulation of objects in C. We have decided to reduce our dependence upon GLib, at least in the Engine and Backends, and to rewrite that part of the program in C++. In order to minimize external dependencies in that part of GnuCash, every effort will be made to depend only on the C++ Standard Library and Boost. We hope that this will make it easier to port GnuCash to other display platforms and GUI libraries.
More information, including some suggested reading if you're not fluent in C++, may be found on the C++ page.
Scheme minimization
There are some parts of GnuCash that make a round-trip into scheme code for no really good reason. The developers would like to throw out those parts of Scheme.
Register rewrite
The current register code started out as code taken from the Gnumeric spreadsheet design, greatly extended and customized for GnuCash's needs. This is central to GnuCash's look and feel with the basic view resembling a checkbook register while expanded views provide more detail about transactions.
It has been partially rewritten to use GtkLayout with a GtkEntry as active element. However the controller code is still mostly as it was. This doesn't map very well with Gtk's View-Controller philosophy. In particular, the code jumps through several hoops to map text entry between the GtkEntry and the controller code.
As a first step it would likely simplify a lot if the GtkEntry would be the sole responsible of all text entry (where now this is handled in large parts by gnucash-sheet, extended with control code from the abstract register implementation). Only keystrokes that can't be handled by the GtkEntry (like those for navigating around in a register) should be delegated to the underlying gnc-sheet.
A rewrite using GtkTreeModel was begun in 2.5 but wasn't completed. It is still in the code base and can be enabled during build configuration. While it addressed the obsolete library problem and would ease conversion to Gtk3, it is still tightly coupled to Gtk+ and so precludes building UIs with other libraries. Moreover as it's no longer maintained it's possibly no longer working and missing newer features that have been implemented in the current register in the meantime.
In addition, Gtk4 is moving away from GtkTrees in favour of GtkGrids. If a major rewrite is to be considered beyond GtkLayout and GtkEntry, it would probably be wise to evaluate the newer options Gtk4 has to offer.
Eliminate deprecated widgets and libraries
GnuCash still uses several gtk widgets that have been marked as deprecated, many of which are not supported in Gtk4. Update all of these to API shared between Gtk3 and Gtk4.
Remove module system
The current module system confuses runtime-loaded modules with shared libraries. As well, it is (arguably) at the wrong level of granularity in places, leading to too many modules. The developers would like to replace most of the plugins by one common library or static application, leaving only 2-3 really optional parts as runtime-loaded plugins (e.g. aqbanking).
Model-View-Controller Pattern
While most of the "model" is properly encapsulated in src/engine, some has leaked into other parts of the program. Worse, there is little separation between view code and controller code. We need to separate the two to make it easier to implement GUIs with other GUI libraries.
Testing
Most subdirectories have a "test" directory containing tests of varying quality, but the coverage is poor and most of the tests are high-level tests that exercise the full module contained in the directory (for example, the dbi backend tests create a data structure in memory programmatically, then save it to a database and reload it and compare the reloaded data with the originals). These are good tests, but coverage is spotty. (Not all of the ways GnuCash stores data are covered in the dbi backend, for example.)
Modern practice calls for "unit tests" exercising every execution path in every exposed function in a tested module. Maintaining good unit tests during development has been extensively documented as a way to improve both productivity and quality. GnuCash doesn't have many unit tests, but it should.
Adding unit testing (with the Google Testing framework) is in progress. Older tests written with GLib testing or the legacy test code in src/test-core will be converted to Google Test and extended further as part of the C++ rewrite.
Database and QOF
Now that we have the DBI backend supporting not just Sqlite3 but Postgresql and MySql as well, we want to move away from a "load everything into memory at the beginning" model to "load only what you need, and take out a backend write lock only on what you're editing" model to allow multiple simultaneous access, to speed up loading large datasets, better data integrity protection, and all of the other benefits provided by a good database.
The current "Query Object Framework" that GnuCash uses rather gets in the way, because while it handles queries acceptably well, it doesn't offer any of the other services that a proper database does. So the "QOF" parts should rather be thrown out.
Some thoughts on what this requires:
- Transactions and splits are by far the object types which cause the most problem.
- Some other objects contain a pointer to a transaction or split (e.g. an invoice contains the pointer to the posting transaction). Either the transaction needs to be loaded when the invoice is or else it needs to be loaded when it is first referenced.
- Some reports generate values by creating a query on splits and then summing them. Given a database engine, a report should be able to query the value of splits for an account or set of accounts on a specific date (with possibly other qualifiers as well e.g. reconcile status). A database engine can do this natively, C code could do it while we're converting, but the API should be defined.
- The set of splits for an account actually is modified with the running balance. xaccAccountGetBalanceAsOfDate() uses this fact by updating the running balances and finding the first balance on or after the date. Instead, the same db query used to get starting/ending balances should be used (sum all splits where account=desired account where date <= desired date). The running balance should be kept by the register because it should be recomputed if the register is resorted or filtered (would close some bugs). (bug #591098 causes the "present" column on the account page to be 0 until the account register is opened if not all transactions are loaded at startup).
Proposal
- Replace QofQuery with SQL queries to the SQL backend. XML files will be loaded as before, but instead of populating Engine objects it will be loaded into an in-memory SQLite3 database to moderated by the backend. Autosaves and end-of-session saves will be exports from the SQLite3 in-memory database to the XML file.
- Non-editing GUI elements and reports will query the SQL backend instead of QofQuery. No Engine objects should be needed for queries. Non-editing GUI elements will take out read locks (so that multiple users can read the same database tuple). Only when a user edits should Engine objects be instantiated; when they are, the corresponding tuples in the database will be write (exclusive) locked. Note well that this will require careful deadlock management, because one cannot get a write lock on a read-locked tuple. Reports will run queries and will not use cursors or acquire any locks.
Reports
Right now reports are Scheme scripts that programmatically generate HTML.
Scheme is impenetrable to most programmers. Expecting users to be able to write reports in Scheme is completely unreasonable.
The ideal solution should have three modules:
- Record selection: Ideally graphical with an SQL-like "advanced" option
- Layout: The original item here suggested an HTML template; that could be the "advanced" option, with a simple table/graph being the default
- Style: CSS. It's built into WebKit, we should use it.
- Javascript: WebKit supports Javascript so complicated interactivity can be added. Example: ability to expand/collapse levels of the account hierarchy in a report. This could be extended with Javascript interfaces to the API so that all of the report code is written in Javascript instead of Scheme.
Any report module will still need some sort of scripting language to "calculate the numbers". Currently we have Scheme for this, but the developers would like to get away from that. Python might be a better option.
