User:Fleminra
From GnuCash
Robert Fleming.
Not a GnuCash developer.
flеmіng⊚ҫѕ∙wаѕhіngtоn∙еdu (transcribe, don't copy+paste)
- Question: Are you really doing your research/development with the 1.8.x version?!? The source code references on GnuCash XML format sound like that. Please don't do that but instead develop against SVN/1.9.x/2.0.0. With 1.8.x, you will very likely encounter all sorts of stupid bugs and XML problems (the locale-depending encoding being the most obvious) which will cause all sorts of frustrating problems because the gnucash developers will not fix any of them in the 1.8.x code anymore. Please please please work with Subversion code so that any enhancement requests can actually go back into the gnucash code at all and also very quickly. And when collection public notes, any source code references should refer to SVN/1.9/2.0, not at all to 1.8.x, so that nobody should be confused to use 1.8.x code in the first place. --Cstim 04:52, 28 February 2006 (EST)
- At the moment I am indeed looking at 1.8 because so far it is the path of least resistance: it is the only version packaged for Debian (for use (i.e.
apt-get install gnucash) and for development (apt-get build-dep gnucash)). Even without the laziness factor, I'm reluctant to use 1.9 as long as there is any inordinate chance of data corruption (especially the not-easily-detectable kind). If I do get this working with 1.8, I'll certainly share it with anyone interested, but not submit it for inclusion. When 1.9/2.0 stabilizes (and/or becomes available for Debian), I'll probably want to port to that, and make some effort to get my work (or some kind of plugin manager UI) integrated into the GnuCash trunk. —Fleminra 16:28, 28 February 2006 (EST)- See https://lists.gnucash.org/pipermail/gnucash-devel/2006-February/016643.html --Cstim 04:46, 1 March 2006 (EST)
- OK, thanks. I had actually checked that [1] (not sure whether before or after it got uploaded). Anyway I guess it has to go through a process before appearing in the repository because it's still not showing up. I will definitely check it out when it reaches the mirrors. —Fleminra 05:28, 1 March 2006 (EST)
- See https://lists.gnucash.org/pipermail/gnucash-devel/2006-February/016643.html --Cstim 04:46, 1 March 2006 (EST)
- At the moment I am indeed looking at 1.8 because so far it is the path of least resistance: it is the only version packaged for Debian (for use (i.e.
Web site
Wiki vs. rest of web site
Has there been any discussion about moving static documentation like http://www.gnucash.org/en/architecture.phtml into the wiki? There is a typo/spelling error, and some of the other text could benefit from an update (e.g. might want to qualify the PostgreSQL backend support). —Fleminra 16:28, 28 February 2006 (EST)
- See https://lists.gnucash.org/pipermail/gnucash-devel/2006-February/016343.html ; making this the main gnucash site is certainly a TODO before 2.0.0 is released. Fixes the problem below as well. --Cstim 04:46, 1 March 2006 (EST)
Broken links
FYI, the http://download.sourceforge.net/gnucash links on the front page are broken (404). —Fleminra 16:28, 28 February 2006 (EST)
Personal wish list
A general note: Please please please make sure any enhancement requests are checked against the current Subversion/1.9.x/upcoming 2.0.0 version! The development of the 1.8.x series has stopped completely; nothing will be fixed or extended there. If you want to contribute to the development, currently there's no way around checking out the code from the Subversion server. --Cstim 04:51, 24 February 2006 (EST)
- Get GnuCash Wiki to Interwiki with Wikipedia to facilitate links like [[Wikipedia:remote procedure call]]
- Viable database back-end. XML's strength is data exchange among heterogenous platforms, which is not really what this part of GnuCash is doing.
- A way to suck in external, repetetive data with minimum of hassle. Requires extensibility and/or non-file-based external software interface. Would like to bypass the (practically) non-deterministic Bayesian import mechanism because in this case the data is consistent and well defined.
- What's wrong with deactivating the checkbox "use bayesian importer" in properties -> Online banking/Importing? Also, I've heard several requests about regexp/rule-based importers but so far nobody bothered to actually implement any of those, so that's why we still don't have anything else.
- I don't want to disable the Bayesian import altogether.. it's still quite useful for dynamic data like bank statements where transactions vary from month to month. I just want to be able to write plug-in (see below) that is smart enough to dump data into specific GnuCash accounts (actually not "smart" but "dumb" because this specific plug-in would be hardcoded and inflexible.) —Fleminra 13:56, 23 February 2006 (EST)
- Ok. But does this refer to existing importing features or to a new, yet-to-be-written importing feature? In the former case, it would make a valid enhancement request suitable for Bugzilla, effectively asking to turn this single preference setting into multiple settings on a per-importing-module basis. In the latter case, the programmer of such an yet-to-be-written feature already has the full flexibility to either import Transactions directly into the gnucash engine or passing them through the available matcher. --Cstim 04:51, 24 February 2006 (EST)
- Yet-to-be-written feature. This sounds like what I'm looking for. —Fleminra 06:02, 24 February 2006 (EST)
- Ok. But does this refer to existing importing features or to a new, yet-to-be-written importing feature? In the former case, it would make a valid enhancement request suitable for Bugzilla, effectively asking to turn this single preference setting into multiple settings on a per-importing-module basis. In the latter case, the programmer of such an yet-to-be-written feature already has the full flexibility to either import Transactions directly into the gnucash engine or passing them through the available matcher. --Cstim 04:51, 24 February 2006 (EST)
- I don't want to disable the Bayesian import altogether.. it's still quite useful for dynamic data like bank statements where transactions vary from month to month. I just want to be able to write plug-in (see below) that is smart enough to dump data into specific GnuCash accounts (actually not "smart" but "dumb" because this specific plug-in would be hardcoded and inflexible.) —Fleminra 13:56, 23 February 2006 (EST)
- What's wrong with deactivating the checkbox "use bayesian importer" in properties -> Online banking/Importing? Also, I've heard several requests about regexp/rule-based importers but so far nobody bothered to actually implement any of those, so that's why we still don't have anything else.
- Would like to do this without downloading/compiling GnuCash or dependent source code.
- Huh? I don't understand this part. What do you mean here? To use gnucash you have to download and compile it. If you don't want to do that, where's the connection to gnucash? Again, that qof project http://qof.sourceforge.net might already be what you are looking for (but its maintainer has just announced he'll be out of town for a few days/weeks)
- I would like to extend GnuCash (the binaries of which I get from Debian/Ubuntu) without needing to download the GnuCash source code. I should say "I would like to use GnuCash without having to compile it myself — I'll let my distributor do that." —Fleminra 13:56, 23 February 2006 (EST)
- As I said: The feature is already available in GnuCash. Modules like the OFX importer or the HBCI/Aqbanking importer are fully self-contained (technically, each one is a dynamic library loaded by gnucash at run-time); you or anyone else could very well develop yet another such module, which could be loaded and/or exchanged without recompiling anything else inside gnucash. (Except that currently the list of to-be-loaded modules is a hardcoded list in (SVN version) src/bin/gnucash-bin.c to avoid questions of module authorization, signatures, or integrity checks, but that could be changed or extended easily if needed.) --Cstim 04:51, 24 February 2006 (EST)
- Yep, that sounds like what I need. I have to admit I was judging GnuCash's extensibility based on what I see in the GUI or in the user documentation, and not from reviewing the source code. Indeed, it sounds like gnucash-bin.c would need to change for what I'm proposing (so I'll have to download/build GnuCash at least for now). I think I'll put together some proof-of-concept import plug-in(s) before asking for a plug-in manager in the GUI. Thanks for pointing me in the right direction.. I think think I know where to start now. —Fleminra 06:02, 24 February 2006 (EST)
- As I said: The feature is already available in GnuCash. Modules like the OFX importer or the HBCI/Aqbanking importer are fully self-contained (technically, each one is a dynamic library loaded by gnucash at run-time); you or anyone else could very well develop yet another such module, which could be loaded and/or exchanged without recompiling anything else inside gnucash. (Except that currently the list of to-be-loaded modules is a hardcoded list in (SVN version) src/bin/gnucash-bin.c to avoid questions of module authorization, signatures, or integrity checks, but that could be changed or extended easily if needed.) --Cstim 04:51, 24 February 2006 (EST)
- I would like to extend GnuCash (the binaries of which I get from Debian/Ubuntu) without needing to download the GnuCash source code. I should say "I would like to use GnuCash without having to compile it myself — I'll let my distributor do that." —Fleminra 13:56, 23 February 2006 (EST)
- Huh? I don't understand this part. What do you mean here? To use gnucash you have to download and compile it. If you don't want to do that, where's the connection to gnucash? Again, that qof project http://qof.sourceforge.net might already be what you are looking for (but its maintainer has just announced he'll be out of town for a few days/weeks)
- I'm not planning on waiting for my bank/employer's consent before making (my copy of) GnuCash suck in my bank/employer's data.
- (weekly!) paystub data from employer, including several tax and other deductions. This can be scraped from one of the company's intranet web sites.
- (weekly!) 401k contributions, spread across several funds. Can be scraped from netbenefits.fidelity.com.
- As for these requests: Do you mean your wish is that gnucash should "retrieve paystub data from employers (web) server" and "retrieve 401k contribution statements from some other (web) server"? In that case, a necessary condition for that wish is that this data is made available by that server through an openly documented protocol, like a http download file, or a ofx direct connect access, or a HBCI online banking server (the last one only exists in Germany). If that is the case, then it's a question of whether gnucash implements that particular online banking/statement retrieval protocol. --Cstim 08:01, 23 February 2006 (EST)
- I would like to do something like Finance::Quote: pull in useful data without the source's permission. After thinking about it, and re-reading my earlier comment, I guess what I want is for GnuCash to be "pluggable" in the Firefox/Thunderbird sense (where nearly all behavior can be modified/augmented) or in the XMMS sense (where plugins strictly fall into one of a few categories like input, processing, and output). For example, if this were in Python, GnuCash could do something like:
import users_custom_modulenew_transactions = users_custom_module.get_new_transactions()for transaction in transactions:# do something with 'transaction' to suck it into my GnuCash data# transaction would need to implement some specific API for GnuCash to get transaction data
- Because I think my employer is likely to never support any standard for getting paystub data.. For sure this is a hack.. in fact I would ultimately prefer that suppliers of finance data encode it in RDF and not even OFX/QIF, to make it most broadly accessible. And some financial instutions may disapprove of me doing this, so I would suggest these unauthorized plugins never be distributed with GnuCash (to redirect complaints from those institutions), but be shared by users by some other means (if at all... e.g. there may be no interest in my plug-in to get my employer's data). —Fleminra 13:56, 23 February 2006 (EST)
- Excuse me, but it seems to me that you seriously underestimate the complexity of the involved requirements for such a feature. It is a fully necessary condition that the "original data source" will actually offer that data in a known format. In almost all cases this implicitly also means that the data source has given permission to pull down that data, as is the case for Finance::Quote or ofx directconnect. I have no idea what you have in mind here -- do you think that anyone who has data stored somewhere will also have it accessible via some web server? Forget this, please. Either someone explicitly decided to offer the data for download, in which case the next question would be of download protocols (http, http+ssl, ftp, hbci, whatever?) and file formats. Or someone does not offer the data for download, in which case you can forget any importing feature into any of your programs, including gnucash. Getting a data source to offer its data for download is completely outside of the scope of gnucash. If you can name a particular data source that does offer its data for download, then we will be able to continue to talk about this in the scope of the gnucash project, and the next questions will be about the protocol and file formats this data is in. But a known download availability is a necessary condition that cannot by any means be skipped. --Cstim 04:51, 24 February 2006 (EST)
- In the case of my pay stubs, I'm already done.. the only missing part getting GnuCash to invoke it. The paystub data is available on an intranet web site, in broken, sloppy, undocumented, and proprietary — yet known-to-me (reverse engineered) and consistent — HTML. I've already written a script that goes through the contortions of logging in, dealing with my employer's home-brewed server-side session scheme (i.e. not cookies), downloading/tidying the broken HTML, and extracting useful data (using XPath). For now, it just dumps it into a concise XML file. It's on the order of 100 lines of Python. I will write one for PG&E to share here (maybe tomorrow). All the data I'm talking about is already (securely) accessible to me in some electronic way. From what you've said, I'm now optimistic that this should be straightforward. All I should need to do is hard-code my new plugins into gnucash-bin.c (for proof of concept at least), and clone the interface implemented by one of the existing import modules. It seems to me the main obstacle may be collecting the dependencies for the build (I haven't tried to build Gnucash for a few years). —Fleminra 06:02, 24 February 2006 (EST)
- Ok, so you are talking about a non-public data source which happens to be available to you. Fine. I guess it shouldn't be too difficult for you to write yet another import module; the import-export/ofx and import-export/hbci parts are obviously working examples, although both a bit more specialized than what you need. Just keep in mind that any development should only happen with the Subversion version, not at all with the 1.8 version. Also, the dependencies for the svn version should be fulfilled on any recent Gnome-development preselection. --Cstim 08:28, 24 February 2006 (EST)
- In the case of my pay stubs, I'm already done.. the only missing part getting GnuCash to invoke it. The paystub data is available on an intranet web site, in broken, sloppy, undocumented, and proprietary — yet known-to-me (reverse engineered) and consistent — HTML. I've already written a script that goes through the contortions of logging in, dealing with my employer's home-brewed server-side session scheme (i.e. not cookies), downloading/tidying the broken HTML, and extracting useful data (using XPath). For now, it just dumps it into a concise XML file. It's on the order of 100 lines of Python. I will write one for PG&E to share here (maybe tomorrow). All the data I'm talking about is already (securely) accessible to me in some electronic way. From what you've said, I'm now optimistic that this should be straightforward. All I should need to do is hard-code my new plugins into gnucash-bin.c (for proof of concept at least), and clone the interface implemented by one of the existing import modules. It seems to me the main obstacle may be collecting the dependencies for the build (I haven't tried to build Gnucash for a few years). —Fleminra 06:02, 24 February 2006 (EST)
- Excuse me, but it seems to me that you seriously underestimate the complexity of the involved requirements for such a feature. It is a fully necessary condition that the "original data source" will actually offer that data in a known format. In almost all cases this implicitly also means that the data source has given permission to pull down that data, as is the case for Finance::Quote or ofx directconnect. I have no idea what you have in mind here -- do you think that anyone who has data stored somewhere will also have it accessible via some web server? Forget this, please. Either someone explicitly decided to offer the data for download, in which case the next question would be of download protocols (http, http+ssl, ftp, hbci, whatever?) and file formats. Or someone does not offer the data for download, in which case you can forget any importing feature into any of your programs, including gnucash. Getting a data source to offer its data for download is completely outside of the scope of gnucash. If you can name a particular data source that does offer its data for download, then we will be able to continue to talk about this in the scope of the gnucash project, and the next questions will be about the protocol and file formats this data is in. But a known download availability is a necessary condition that cannot by any means be skipped. --Cstim 04:51, 24 February 2006 (EST)
- I would like to do something like Finance::Quote: pull in useful data without the source's permission. After thinking about it, and re-reading my earlier comment, I guess what I want is for GnuCash to be "pluggable" in the Firefox/Thunderbird sense (where nearly all behavior can be modified/augmented) or in the XMMS sense (where plugins strictly fall into one of a few categories like input, processing, and output). For example, if this were in Python, GnuCash could do something like:
- As for these requests: Do you mean your wish is that gnucash should "retrieve paystub data from employers (web) server" and "retrieve 401k contribution statements from some other (web) server"? In that case, a necessary condition for that wish is that this data is made available by that server through an openly documented protocol, like a http download file, or a ofx direct connect access, or a HBCI online banking server (the last one only exists in Germany). If that is the case, then it's a question of whether gnucash implements that particular online banking/statement retrieval protocol. --Cstim 08:01, 23 February 2006 (EST)
- utility bills: how much goes to electricy, gas, tax?
- A way to link GnuCash data to/from arbitrary other data, like my relational database that tracks all those electronics rebates, link to e-mails pertinent to a particular account/transaction/split, link to locally archived PDF receipts, link to related photographs, etc. I already link into the GnuCash file using GUIDs and XPath (for the rebate-tracking application), but parsing XML documents on the order of 10 MB is unhealthy for application responsiveness. (see Semantic Web, GNOME Storage.)
