2023-10-08 GnuCash IRC logs

00:11:36 *** chris has quit IRC
00:48:23 *** giuseppef has quit IRC
00:49:11 *** jwhite has quit IRC
01:10:17 *** chris has joined #gnucash
01:10:17 *** ChanServ sets mode: +v chris
01:10:17 *** gncbot sets mode: +o chris
01:43:43 *** fell has quit IRC
01:45:03 *** fell has joined #gnucash
01:45:03 *** ChanServ sets mode: +o fell
02:05:06 *** chris has quit IRC
02:25:01 *** chris has joined #gnucash
02:25:01 *** ChanServ sets mode: +v chris
02:25:01 *** gncbot sets mode: +o chris
02:51:16 <fell> jralls, I got an error on MacOs CI Test:
02:51:18 <fell> tar: Error opening archive: Failed to open '/Users/runner/work/gnucash/gnucash/gnucash-dependencies.tar.xz'
02:51:19 <fell> Error: Process completed with exit code 1.
03:02:18 *** gandalf has joined #gnucash
03:03:36 *** chris has quit IRC
03:15:56 *** giuseppef has joined #gnucash
03:15:56 *** ChanServ sets mode: +v giuseppef
03:53:04 *** Aussie_matt has joined #gnucash
04:08:42 *** chf has quit IRC
04:09:26 *** chf has joined #gnucash
04:18:04 *** chris has joined #gnucash
04:18:04 *** ChanServ sets mode: +v chris
04:18:05 *** gncbot sets mode: +o chris
04:22:29 *** sunyibo has joined #gnucash
04:22:30 *** ChanServ sets mode: +v sunyibo
04:49:09 *** gjanssens has joined #gnucash
04:49:09 *** ChanServ sets mode: +o gjanssens
05:32:09 *** gandalf1 has joined #gnucash
05:32:36 *** gandalf has quit IRC
05:32:36 *** gandalf1 is now known as gandalf
06:41:21 *** chris has quit IRC
07:36:52 *** chris has joined #gnucash
07:36:53 *** ChanServ sets mode: +v chris
07:36:53 *** gncbot sets mode: +o chris
08:14:26 *** chris has quit IRC
08:23:37 *** gandalf1 has joined #gnucash
08:24:05 *** gandalf has quit IRC
08:24:05 *** gandalf1 is now known as gandalf
08:45:32 *** jwhite has joined #gnucash
08:45:32 *** ChanServ sets mode: +v jwhite
09:11:38 *** gandalf1 has joined #gnucash
09:12:05 *** gandalf has quit IRC
09:12:05 *** gandalf1 is now known as gandalf
09:41:46 * Simon is disappointed to discover that the database options aren't any faster at loading than using XML
09:42:27 *** sunyibo has quit IRC
09:42:44 <Simon> either they all (XML, sqlite3, postgresql) all somehow have very similar loading overhead or the bulk of the time is in the generic part
09:48:42 *** Aussie_matt has quit IRC
10:37:51 <warlord> Simon, the reason is that all systems need to load the whole data into RAM.
10:38:49 <warlord> The only way to "speed up" the SQL backends would be to convert all of gnucash into an actual SQL-responsive app, and then you could just run SQL Queries to get the data you needed in real-time.
10:38:52 <Simon> yes I know that but how are they all equally similar at parsing time?
10:40:45 <warlord> Because the "load object into RAM" takes the same about of time, regardless of how the raw-data is read or where it is read from. I.e., GnuCash does the same amount of work to load, regardless of the backend.
10:42:38 <warlord> @tell jralls what about the XML backend?
10:42:38 <gncbot> warlord: The operation succeeded.
10:43:27 <Simon> I'd expect the parsing time of XML to be significantly higher than the other two
10:46:39 <warlord> Nope. Actually, in some cases, the loading from SQL is slower. (This is because the spinning-rust holds compressed data so there is less data to read off the spinning rust vs SQL)
11:14:59 *** gandalf1 has joined #gnucash
11:15:26 *** gandalf has quit IRC
11:15:26 *** gandalf1 is now known as gandalf
11:19:29 *** gandalf1 has joined #gnucash
11:19:58 *** gandalf has quit IRC
11:19:58 *** gandalf1 is now known as gandalf
11:47:52 *** FeuerwehrMichael has joined #gnucash
11:47:52 *** ChanServ sets mode: +v FeuerwehrMichael
11:52:48 *** FeuerwehrMichael has quit IRC
11:59:23 *** jervin has joined #gnucash
12:05:28 *** chris has joined #gnucash
12:05:28 *** ChanServ sets mode: +v chris
12:05:28 *** gncbot sets mode: +o chris
12:58:17 *** jervin has quit IRC
12:58:28 *** jervin has joined #gnucash
12:59:16 *** chris has quit IRC
13:01:49 *** gandalf1 has joined #gnucash
13:02:00 *** gandalf has quit IRC
13:02:00 *** gandalf1 is now known as gandalf
13:08:34 *** mydogsnameisrudy has joined #gnucash
13:16:34 *** gandalf1 has joined #gnucash
13:17:01 *** gandalf has quit IRC
13:17:01 *** gandalf1 is now known as gandalf
13:48:14 *** jralls has joined #gnucash
13:48:14 *** ChanServ sets mode: +o jralls
13:57:49 *** palerider has joined #gnucash
13:58:28 *** palerider has left #gnucash
14:06:01 <jralls> Simon, as it happens I just profiled loading the other day. The main time-consumer is xaccTransCommitEdit and calling that on every transaction is of course common to both backends.
14:06:01 <gncbot> jralls: Sent 3 hours and 23 minutes ago: <warlord> what about the XML backend?
14:06:56 *** teirdes has joined #gnucash
14:07:18 <jralls> warlord, A history lesson, perhaps? But after asking that I looked at the origins and it looks like most of it is before your time.
14:09:16 *** ChanServ sets mode: +v teirdes
14:10:24 <teirdes> hi. i've implemented the belgian account structure in dutch in a .gnucash/XML format but i can't get gnucash to recognise it even after placing it in /usr/share/gnucash/accounts
14:10:27 <jralls> Though I wonder if it makes sense to have kept the v1 stuff around so long, and ISTR mikee putting effort into keeping it working. It seems to have been used only for GC-1.6.
14:10:51 <teirdes> it was my longer term plan to also prepare the structure in french, perhaps with the simplified structures since 2018 for small or micro enterprises also.
14:11:05 <teirdes> kapitaalloze vennootschapen
14:15:19 <jralls> teirdes, did you create a new directory be_NL for it? How did you try to access it?
14:22:16 <Simon> for XML it would be nice to use a compile time trie for string matching but I'm not sure it'd save any time
14:22:29 <jralls> tierdes, sorry, the directory would be nl_BE and you'd need to rename nl to nl_NL.
14:23:05 <Simon> I wonder if the CPU is doing 64-bit string memory comparisons for strcmp...
14:24:03 <Simon> assuming alignment of the string data that would probably be faster than individual character comparsions
14:25:20 <jralls> Simon, trie? Do you mean use Hanna's compile-time RE library? As for string comparisons, depends on which ones. Some of them use the string-cache's pointer comarisons, which are very fast. But strcmp(), used by g_strcmp0 and std::string::operator==(), is indeed char wise.
14:26:57 <jralls> It can't do uint64_t memcmp because it has to look for the terminating '\0'.
14:28:38 *** gandalf1 has joined #gnucash
14:29:05 *** gandalf has quit IRC
14:29:06 *** gandalf1 is now known as gandalf
14:29:42 <Simon> https://github.com/smilingthax/cttrie (there are no good ways to express this in C++ afaik)
14:30:56 <Simon> it could do uint64_t comparisons as long as the strings are aligned and terminated with enough \0 characters
14:31:20 <Simon> that would allow it to get things like "price:.." in one integer comparison
14:31:43 <Simon> the key being compared can be prepared for that, and the key being loaded can be aligned
14:32:00 <Simon> terminating with enough \0 is just a memset at the start
14:32:15 <warlord> I take the blame for migrating from Enum to String
14:32:25 <warlord> jralls, yes, most of the xml backend predates me.
14:32:37 <warlord> No clue why we still have the v1 code in there.
14:32:44 <Simon> the main problem with string cache on load is that it has to do a hash map lookup on the entire string cache, not just on the potential keys
14:33:32 <Simon> whereas if I was optimising the string key comparison used in the XML parser I think I'd go with the integer comparison
14:33:35 *** gandalf has quit IRC
14:34:28 *** gandalf has joined #gnucash
14:34:46 <jralls> > "terminated with enough \0 " An impossible precondition.
14:35:12 <Simon> no, it's not
14:35:22 <Simon> the strings are loaded into memory from the XML file
14:35:34 <jralls> By a third-party library, libxml2.
14:36:18 <jralls> In any case, the xml parsing isn't affecting the load times.
14:38:58 <jralls> That's down to the crappy way GObject constructors work where every single parameter--and some of GnuCash's objects have a lot of then--must be pushed into a GValue, passed through g_object_set to a decoder function, extracted from the GValue, and then passed to a setter function.
14:41:44 <jralls> The fix for that is getting rid of GObject, but there's other work that needs to happen first. High on that list is getting rid of QofQuery. My plan for that is to parse the XML directly into an in-memory SQLite DB so that QofQuery can be replaced with SQL queries. That will also allow creating only the transactions needed for the open register tab, which should get us under a second regardless of file size.
14:42:04 <jralls> Err, *to under a second*.
14:44:01 <Simon> xaccTransCommitEdit() hardly seems to *do* anything with the XML backend... oh no... xaccTransSortSplits()
14:44:45 <Simon> under a second despite having to build SQLite indexes?
14:45:07 <Simon> although I suppose at that point I would just migrate to postgres
14:49:07 <jralls> Postgres currently has the same bottleneck, creating all of the transactions. Obviously once we don't need to do that the SQL backends will benefit even more than the XML one.
14:50:09 <jralls> Another benefit for them will be that that same XML<->SQL code will allow creating an XML backup without having to do the file>save-as drill.
14:52:54 <jralls> I don't see a way to make that much faster than the current XML save though, because it still requires building a DOM for the whole document. I don't expect much speed difference traversing the QofCollections or a query result.
14:53:25 <Simon> xaccTransCommitEdit() is only about 10% of the time for me
14:55:03 <jralls> How are you doing the profile? Call-counters like gcc -p give the wrong answer.
14:55:12 <Simon> I made the function do nothing during loading
14:56:55 <jralls> Interesting, but it doesn't tell you anything about what takes over as the hot stack.
14:58:13 <Simon> I think I timed the pricedb part of my file at 40%, so I probably need to take separate timings of those two main parts of the file
14:59:57 <jralls> Have a look at perf and flamegraph. Both are in most linux distros.
15:01:05 <jralls> A far more productive way to figure out performance bottlenecks than hacking out chunks of code.
15:46:55 *** jervin has quit IRC
15:52:08 *** gjanssens has quit IRC
16:00:18 *** gandalf has quit IRC
16:15:31 *** NoobAlice has joined #gnucash
16:15:31 *** ChanServ sets mode: +v NoobAlice
16:25:06 *** chf has quit IRC
16:36:00 *** chf has joined #gnucash
16:51:47 *** jervin has joined #gnucash
16:54:11 *** jralls has quit IRC
16:54:38 *** jralls has joined #gnucash
16:54:38 *** ChanServ sets mode: +o jralls
16:57:40 *** jralls has quit IRC
17:11:54 *** jralls has joined #gnucash
17:11:54 *** ChanServ sets mode: +o jralls
17:29:50 *** sunyibo has joined #gnucash
17:29:50 *** ChanServ sets mode: +v sunyibo
17:42:27 <teirdes> jralls: yes and i just tried to restart
17:42:42 <teirdes> it recognises the new directory but it doesn't recognise the template within
17:49:52 <jralls> For categories does it now show BE as an option when nl is selected for the language?
18:08:30 <jralls> teirdes ^
19:36:30 *** teirdes has quit IRC
19:51:10 *** jralls has quit IRC
19:51:41 *** jralls has joined #gnucash
19:51:41 *** ChanServ sets mode: +o jralls
19:54:43 *** jralls has quit IRC
20:00:03 *** raghukamath has quit IRC
20:12:07 *** raghukamath has joined #gnucash
20:12:07 *** ChanServ sets mode: +v raghukamath
20:14:51 *** jralls has joined #gnucash
20:14:51 *** ChanServ sets mode: +o jralls
21:24:46 *** jralls has quit IRC
21:32:49 *** jralls has joined #gnucash
21:32:49 *** ChanServ sets mode: +o jralls
21:35:50 *** jralls has quit IRC
21:40:19 *** jralls has joined #gnucash
21:40:19 *** ChanServ sets mode: +o jralls
21:43:20 *** jralls has quit IRC
22:00:09 *** vasukan has quit IRC
22:03:38 *** jralls has joined #gnucash
22:03:39 *** ChanServ sets mode: +o jralls
22:07:33 *** jralls has quit IRC
22:24:29 *** jralls has joined #gnucash
22:24:29 *** ChanServ sets mode: +o jralls
22:25:21 *** jralls has quit IRC
22:25:30 *** jralls has joined #gnucash
22:25:30 *** ChanServ sets mode: +o jralls
22:26:50 *** jralls has quit IRC
22:27:17 *** jralls has joined #gnucash
22:27:17 *** ChanServ sets mode: +o jralls
22:30:18 *** jralls has quit IRC
22:38:33 *** chris has joined #gnucash
22:38:33 *** ChanServ sets mode: +v chris
22:38:33 *** gncbot sets mode: +o chris
22:41:07 *** jralls has joined #gnucash
22:41:07 *** ChanServ sets mode: +o jralls
22:42:29 *** jralls has quit IRC
22:43:16 *** jralls has joined #gnucash
22:43:16 *** ChanServ sets mode: +o jralls
22:43:40 *** jralls has quit IRC
22:43:45 *** jralls has joined #gnucash
22:43:45 *** ChanServ sets mode: +o jralls
22:45:06 *** jralls has quit IRC
22:45:34 *** jralls has joined #gnucash
22:45:34 *** ChanServ sets mode: +o jralls
22:46:16 *** jralls has quit IRC
22:46:23 *** jralls has joined #gnucash
22:46:23 *** ChanServ sets mode: +o jralls
22:47:44 *** jralls has quit IRC
22:48:17 *** jralls has joined #gnucash
22:48:17 *** ChanServ sets mode: +o jralls
22:49:00 *** jralls has quit IRC
22:49:06 *** jralls has joined #gnucash
22:49:06 *** ChanServ sets mode: +o jralls
22:50:28 *** jralls has quit IRC
22:51:02 *** jralls has joined #gnucash
22:51:02 *** ChanServ sets mode: +o jralls
22:51:44 *** jralls has quit IRC
22:51:51 *** jralls has joined #gnucash
22:51:51 *** ChanServ sets mode: +o jralls
22:53:13 *** jralls has quit IRC
22:53:40 *** jralls has joined #gnucash
22:53:40 *** ChanServ sets mode: +o jralls
22:54:23 *** jralls has quit IRC
22:54:29 *** jralls has joined #gnucash
22:54:29 *** ChanServ sets mode: +o jralls
22:55:58 *** jralls has quit IRC
22:56:32 *** jralls has joined #gnucash
22:56:32 *** ChanServ sets mode: +o jralls
22:58:24 *** Aussie_matt has joined #gnucash
22:59:33 *** jralls has quit IRC
23:01:30 *** NoobAlice has quit IRC
23:03:44 *** jralls has joined #gnucash
23:03:44 *** ChanServ sets mode: +o jralls
23:05:06 *** jralls has quit IRC
23:05:33 *** jralls has joined #gnucash
23:05:33 *** ChanServ sets mode: +o jralls
23:06:17 *** jralls has quit IRC
23:06:23 *** jralls-m1 has joined #gnucash
23:07:44 *** jralls-m1 has quit IRC
23:08:16 *** jralls has joined #gnucash
23:08:16 *** ChanServ sets mode: +o jralls
23:08:57 *** jralls has quit IRC
23:09:04 *** jralls has joined #gnucash
23:09:04 *** ChanServ sets mode: +o jralls
23:09:58 *** jralls has quit IRC
23:10:53 *** jralls has joined #gnucash
23:10:53 *** ChanServ sets mode: +o jralls
23:11:37 *** jralls has quit IRC
23:11:42 *** jralls-m1 has joined #gnucash
23:13:04 *** jralls-m1 has quit IRC
23:13:32 *** jralls has joined #gnucash
23:13:32 *** ChanServ sets mode: +o jralls
23:16:32 *** jralls has quit IRC
23:28:38 *** jralls has joined #gnucash
23:28:38 *** ChanServ sets mode: +o jralls
23:30:00 *** jralls has quit IRC
23:30:55 *** jralls has joined #gnucash
23:30:55 *** ChanServ sets mode: +o jralls
23:31:37 *** jralls has quit IRC
23:31:44 *** jralls has joined #gnucash
23:31:44 *** ChanServ sets mode: +o jralls
23:33:07 *** jralls has quit IRC
23:33:43 *** jralls has joined #gnucash
23:33:43 *** ChanServ sets mode: +o jralls
23:34:23 *** jralls has quit IRC
23:34:30 *** jralls has joined #gnucash
23:34:31 *** ChanServ sets mode: +o jralls
23:35:24 *** chris has quit IRC
23:35:52 *** jralls has quit IRC
23:36:47 *** jralls has joined #gnucash
23:36:47 *** ChanServ sets mode: +o jralls
23:39:48 *** jralls has quit IRC
23:54:53 *** jralls has joined #gnucash
23:54:53 *** ChanServ sets mode: +o jralls
23:57:53 *** jralls has quit IRC