2020-03-05 GnuCash IRC logs
00:05:58 *** jervin has quit IRC
00:22:10 *** lcanaska has quit IRC
00:24:36 *** lcanaska has joined #gnucash
00:24:48 *** fell_laptop is now known as fell
00:33:48 *** pulsar has joined #gnucash
00:34:30 <fell> jralls: After gjanssens intervention the master build 2020-03-04 succeeded. So we have to wait for maint 2020-03-05.
00:35:14 <jralls> fell, OK.
00:41:22 *** lcanaska1 has joined #gnucash
00:42:36 *** lcanaska has quit IRC
00:59:00 *** omnireq_ has quit IRC
00:59:11 *** omnireq_ has joined #gnucash
01:02:44 *** jralls has quit IRC
01:02:49 *** jralls has joined #gnucash
01:03:32 *** jralls has quit IRC
01:03:38 *** jralls has joined #gnucash
01:04:20 *** jralls has quit IRC
01:04:25 *** jralls has joined #gnucash
01:05:12 *** jralls has joined #gnucash
01:05:55 *** jralls has quit IRC
01:06:00 *** jralls has joined #gnucash
01:06:37 *** Mechtilde has joined #gnucash
01:06:42 *** jralls has quit IRC
01:06:47 *** jralls has joined #gnucash
01:07:30 *** jralls has quit IRC
01:27:53 *** fell_laptop has joined #gnucash
01:27:53 *** ChanServ sets mode: +o fell_laptop
01:29:31 *** fell has quit IRC
01:45:38 *** sbluhm has joined #gnucash
01:45:38 *** ChanServ sets mode: +v sbluhm
01:47:11 *** fell_laptop is now known as fell
02:02:07 *** sbluhm has quit IRC
02:11:44 *** Mechtilde has quit IRC
02:21:30 *** omnireq_ has quit IRC
02:21:41 *** omnireq_ has joined #gnucash
02:41:54 *** sbluhm has joined #gnucash
02:49:39 *** Aussie_matt has quit IRC
02:53:01 *** FH_thecat has quit IRC
03:07:04 *** Mechtilde has joined #gnucash
03:27:08 *** gjanssens has joined #gnucash
03:27:08 *** gncbot sets mode: +o gjanssens
03:27:08 *** ChanServ sets mode: +o gjanssens
03:47:26 *** jralls has joined #gnucash
03:50:22 *** bertbob has quit IRC
03:50:26 *** jralls has quit IRC
03:50:39 *** bertbob has joined #gnucash
03:50:40 *** ChanServ sets mode: +v bertbob
03:54:36 *** bertbob has quit IRC
03:55:42 *** bertbob has joined #gnucash
03:55:43 *** ChanServ sets mode: +v bertbob
03:59:37 *** bertbob has quit IRC
04:00:56 *** bertbob has joined #gnucash
04:00:56 *** ChanServ sets mode: +v bertbob
04:10:06 *** bertbob has quit IRC
04:11:37 *** bertbob has joined #gnucash
04:11:38 *** ChanServ sets mode: +v bertbob
04:24:10 *** FH_thecat has joined #gnucash
04:29:06 <gjanssens> .
04:31:31 *** ArtGravity has quit IRC
04:46:47 *** pulsar has joined #gnucash
05:14:38 *** ramontjunior has joined #gnucash
05:16:03 *** User__ has joined #gnucash
05:23:20 *** pulsar has joined #gnucash
05:24:48 *** pulsar has quit IRC
05:48:00 <chris> gjanssens: I was looking for an easy C profiling statement. thx for sysprof, https://imgur.com/a/oMGm2Gz is result for importing large qif file. lots of traversals in xaccTransCommitEdit.
05:51:41 *** User__ has quit IRC
05:56:13 *** JulianHofer[m] has quit IRC
05:56:23 *** Couto[m] has quit IRC
05:56:26 *** ElonSatoshi[m] has quit IRC
05:56:27 *** peter-butler[m] has quit IRC
05:56:28 *** mmkodali[m] has quit IRC
05:56:31 *** sunyibo[m] has quit IRC
05:56:33 *** luwum[m] has quit IRC
05:58:08 *** ElonSatoshi[m] has joined #gnucash
06:20:04 *** JulianHofer[m] has joined #gnucash
06:20:05 *** peter-butler[m] has joined #gnucash
06:20:06 *** mmkodali[m] has joined #gnucash
06:20:07 *** sunyibo[m] has joined #gnucash
06:20:07 *** Couto[m] has joined #gnucash
06:20:08 *** luwum[m] has joined #gnucash
06:20:40 *** Jimraehl1 has joined #gnucash
06:22:01 *** Jimraehl1 has quit IRC
06:42:00 *** omnireq_ has quit IRC
06:42:11 *** omnireq_ has joined #gnucash
06:59:06 <chris> I think I've fixed 797640
07:04:41 <chris> and it seems to match my expectations
07:22:07 *** pulsar has joined #gnucash
07:23:30 *** omnireq_ has quit IRC
07:23:41 *** omnireq_ has joined #gnucash
07:25:53 *** pulsar has quit IRC
07:38:50 *** chris has quit IRC
07:47:30 *** chris has joined #gnucash
07:47:30 *** ChanServ sets mode: +v chris
08:25:30 *** omnireq_ has quit IRC
08:25:45 *** omnireq_ has joined #gnucash
08:30:27 *** jralls has joined #gnucash
08:42:49 *** gour has joined #gnucash
08:42:49 *** ChanServ sets mode: +v gour
08:44:35 <chris> jralls: I don't suppose hash-tables searching can be sped up using https://developer.gnome.org/glib/stable/glib-Hash-Tables.html ?
08:44:54 * chris displaying ignorance on topic
08:59:15 <warlord> chris, we use that object -- unless there is a specific API you're talking about?
09:00:43 *** waeking has quit IRC
09:01:44 *** waeking has joined #gnucash
09:01:44 *** ChanServ sets mode: +v waeking
09:02:09 <chris> warlord: I'm referring to the profiling 21000line QIF and 85% of time spent in xaccTransCommitEdit: see https://imgur.com/a/oMGm2Gz
09:02:42 <chris> g_list_foreach < g_hash_table_foreach
09:06:01 <chris> maybe I have no idea how to read profiling data
09:32:12 <warlord> It's just telling you where it is spending its time, not necessarily which is "faster". Just because it's spending less time in g_list_foreach() does not mean that it would be faster, overall, to use a glist vs a ghash.
09:32:27 *** jralls has quit IRC
09:33:32 <chris> ok. clutching at straws here. it's quite disconcerting that importing ofx/qif/csv takes so much time.
09:34:51 <warlord> I would expect that it depends on how many transactions are being imported.. And this is why we (should) have a progress bar.
09:36:23 <warlord> Is it really that slow if you're only importing 1 transaction? Or is it only slow if you're importing 1000?
09:36:36 <warlord> and define "slow"
09:40:29 <chris> slow - importing OFX produces slowdown... e.g. OFX with ~1000 txns takes 50 seconds.
09:43:42 <warlord> So ~20/s..
09:44:04 <warlord> That does seem a LITTLE slow...
09:44:11 <chris> (Finally!)
09:45:32 <chris> C doesn't have guile's easy (statprof (lambda () (slow-fn)))
09:46:15 <warlord> okay, looking at your image, what is says is that it's spending 28% of the time (cumulatively) within the iteration function. Looks like 16% of that 28% is in the GUID Match Predicate.
09:46:36 <chris> exactly. string-comparison isn't rocket science. why so slow.
09:46:37 <warlord> (and its children)
09:46:45 <warlord> It's not a string-compare
09:47:17 <warlord> Where is it comparing strings? It's doing a GUID match.
09:47:19 <warlord> Expand that out.
09:48:03 <chris> need to redo.
09:48:11 <warlord> Also, the xaccTransCommitEdit is also a lot of the time..
09:48:24 * warlord has a work call in 10 minutes so wont be responding.
09:49:07 <chris> ^exactly.
09:53:09 *** KevinDB has quit IRC
09:54:04 *** jralls has joined #gnucash
09:54:28 *** KevinDB has joined #gnucash
09:54:28 *** ChanServ sets mode: +v KevinDB
09:54:54 <chris> the sysprof capture .zip is only 1Mb maybe I'll share that instead
09:56:49 <chris> https://drive.google.com/open?id=1noGarzjHcB-toObKH5q3mBmD7gSgghNR
09:57:03 <warlord> I wont be able to look at it for a while.
09:57:15 <warlord> (and I dont currently have sysprof installed ;)
09:57:47 <jralls> chris, Do you mean using a GHashTable instead of a Guile one? I'd be surprised if there was a significant performance difference.
09:58:18 <chris> jralls probably right. nevertheless conclusion is still xaccTransCommitEdit is too slow.
10:00:01 <chris> guid_match_predicate from libgnc_mod_engine
10:00:42 <jralls> chris, xaccTransCommitEdit is C so if it's using a hash table then it would be a GHashTable.
10:01:11 <chris> that's what i meant
10:01:41 <chris> anyway feel free to datamine the sysprof... done while importing a large qif
10:01:51 *** bertbob has quit IRC
10:02:31 <jralls> The main problem with hash tables is that they're implemented on top of linked lists so the actual data are splattered all over memory so you get lots of cache misses.
10:07:22 <jralls> I pulled the file, I'll need to move it to a Linux VM where I can get sysprof to read it.
10:07:29 <chris> cool :)
10:09:42 *** bertbob has joined #gnucash
10:09:42 *** ChanServ sets mode: +v bertbob
10:09:46 <jralls> But I doubt that xaccTransCommitEdit is a hot function, it doesn't really do anything itself.
10:10:21 <chris> but it does. it's called by qif-to-gnc.scm and is responsible for 85% of time during qif import.
10:11:28 <chris> see https://pastebin.com/raw/63ShWL3H statprof
10:11:36 <jralls> Go look at the code: https://github.com/Gnucash/gnucash/blob/maint/libgnucash/engine/Transaction.c#L1643. Any work is in functions it calls.
10:12:24 <chris> statprof 1st one is creating many txns, 2nd is destroying them. xaccTransDestroy also calls xacctranscommitedit.
10:17:39 <jralls> Roger, though those profile reports aren't very good because they lose the who-called-what relationship between the functions. Is there a stack-depth limitation so that it's ignoring the functions that xaccTransCommitEdit calls?
10:18:08 <chris> i have no idea. i'm new to this.
10:20:54 <jralls> Or maybe it's only looking at what the Guile interpreter sees.
10:25:33 <chris> ^ correct
10:26:00 <chris> I ranted that C has no profiler, meant it has no builtin profiler that I could modify C statements in isolation to find the slowdown.
10:33:39 *** sbluhm has quit IRC
10:33:40 <jralls> Profiling is expensive so building with it turned on would cause a major performance hit. GCC has a flag to turn it on and a companion program, GPROF, to analyze the results.
10:33:42 <jralls> https://www.thegeekstuff.com/2012/08/gprof-tutorial/
10:35:13 *** ArtGravity has joined #gnucash
10:35:13 *** ChanServ sets mode: +v ArtGravity
10:35:43 *** omnireq_ has quit IRC
10:36:07 *** kael has joined #gnucash
10:36:07 *** ChanServ sets mode: +v kael
10:37:07 <jralls> Another approach is programs that get a stack trace of the running program at intervals and report on how many times each function occurs in the samples.
10:38:32 <jralls> MacOS's Xcode has a very nice one that I use alot. gjanssens found a similar tool for Linux and mentioned it here last week.
10:41:22 <chris> the stack-trace approach is the guile one I think
10:41:50 <warlord> There is also Valgrind and Cachegrind
10:43:17 <jralls> Valgrind and Cachegrind are helpful for memory-allocation related performance issues but won't find a poorly designed hot loop.
10:43:41 <chris> some CMake magic to enable gprof would be nice please :)
10:44:49 <jralls> I imagine tha the Guile statprof is similar to the Python one since Andy Wingo wrote the original python one and the python one is a stack sampler.
10:45:21 * chris willing to bet that xaccTransCommitEdit speedup will also significantly improve file load performance
10:45:40 <chris> interesting that wingo is such a polyglot
10:46:38 <jralls> Just add -DCMAKE_F_FLAGS='-pg' to your cmake command line. Be sure to also set -DCMAKE_BUILD_TYPE=Debug so that the function names are in the binary.
10:47:30 <chris> ok thx. gtg now late
10:48:30 <jralls> But like I said turning that on slows the program substantially. Better to find gjanssens post last week about the Linux sampler.
10:48:53 <jralls> Samplers have another advantage, they're a lot easier to use than gprof.
10:51:42 *** sbluhm has joined #gnucash
10:57:03 *** Mechtilde has quit IRC
11:02:17 *** KaiForce_ has joined #gnucash
11:02:39 <gjanssens> jralls: I mentioned it to chris yesterday as well: heaptrack
11:02:48 *** KaiForce has quit IRC
11:02:58 *** KaiForce_ is now known as KaiForce
11:04:40 <jralls> gjanssens, is that https://github.com/KDE/heaptrack?
11:05:37 *** gour has quit IRC
11:06:21 <gjanssens> yes
11:06:55 *** ArtGravity has quit IRC
11:07:08 <gjanssens> It works by sampling like sysprof does and probably also like the xcode one does
11:07:43 <gjanssens> I found it fairly easy to get started with.
11:08:39 <gjanssens> It has a nice interactive gui tool which presents the profile data in various ways and that allows you to drill down
11:08:47 <gjanssens> (If you like guis of course)
11:10:56 *** jralls_ has joined #gnucash
11:11:12 <jralls_> So it looks at more than just the heap? The readme on that page suggests that it's a lighter-weight approach to Valgrind.
11:11:55 *** omnireq has joined #gnucash
11:11:55 *** ChanServ sets mode: +v omnireq
11:12:13 *** jralls has quit IRC
11:13:37 *** Antic has joined #gnucash
11:13:49 *** ChanServ sets mode: +v Antic
11:14:49 <Antic> Is there a way to preview what changes have been made before I save a GnuCash file? (I prefer not to auto-save)
11:15:08 <gjanssens> Antic: not that I'm aware of.
11:16:29 *** jralls_ is now known as jralls
11:20:49 <gjanssens> jralls: perhaps my knowledge of profiling tools is too limited to judge here
11:22:00 <gjanssens> But I think you're right - it analyzes heap allocations
11:22:19 <gjanssens> It has features to find temporary allocations and memory leaks
11:22:40 <gjanssens> There's not really a cpu timing tab as far as I can see
11:23:08 <gjanssens> So perhaps sysprof may present a more complete picture
11:23:29 <jralls> Certainly useful but probably not what chris is looking for, particularly since the gslice mechanism makes a lot of allocations invisible to tools like that.
11:24:22 <gjanssens> Indeed
11:24:44 <gjanssens> I should revisit sysprof at some point...
11:26:12 *** waeking has quit IRC
11:27:08 *** guak has joined #gnucash
11:28:44 <jralls> There's also https://oprofile.sourceforge.io/news/ and maybe https://perf.wiki.kernel.org/index.php/Tutorial.
11:32:33 *** Gerd has joined #gnucash
11:35:43 *** Yotson has quit IRC
11:35:49 *** Yotson has joined #gnucash
11:52:54 *** Mechtilde has joined #gnucash
12:03:26 *** waeking has joined #gnucash
12:03:26 *** ChanServ sets mode: +v waeking
12:03:26 *** waeking has quit IRC
12:04:38 *** guak has quit IRC
12:12:41 *** guak has joined #gnucash
12:24:30 *** Gerd has quit IRC
12:28:30 *** calvinct has joined #gnucash
12:58:19 *** Gerd has joined #gnucash
13:01:37 *** kael has quit IRC
13:48:36 *** Gerd has quit IRC
14:05:49 *** jervin has joined #gnucash
14:10:17 *** jervin has quit IRC
14:21:15 *** frakturfreak has joined #gnucash
14:21:15 *** ChanServ sets mode: +v frakturfreak
14:32:18 *** Han has joined #gnucash
14:49:55 <gjanssens> jralls, warlord, chris, fell: Just a heads up I'll be mostly afk for two weeks starting coming Saturday.
14:50:31 <gjanssens> I'll be on holidays (to Indonesia)
14:50:54 <fell> Visiting active volcanos?
14:50:57 *** jralls_ has joined #gnucash
14:51:02 *** jralls has quit IRC
14:53:42 <gjanssens> fell: no :) We actually will hardly see anything of Indonesia itself. It's more of a retreat kind of trip
15:04:14 <warlord> gjanssens, have a great trip!
15:04:23 <gjanssens> Tx warlord
15:06:13 *** calvinct has quit IRC
15:06:18 *** ramontjunior has quit IRC
15:07:24 <fell> and relax!
15:07:30 <fell> gjanssens, as you have been working on 'process payment', do we have somewhere documentation about pre-/overpayment?
15:08:02 <fell> The docs do not mention it.
15:08:39 <gjanssens> No I don't think we have.
15:19:03 *** blathijs has quit IRC
15:21:13 *** blathijs has joined #gnucash
15:33:09 *** jralls has joined #gnucash
15:33:59 *** jralls_ has quit IRC
15:40:09 *** jralls has quit IRC
15:42:47 *** jralls has joined #gnucash
16:28:17 *** bertbob has quit IRC
16:30:17 *** bertbob has joined #gnucash
16:30:18 *** ChanServ sets mode: +v bertbob
16:34:26 *** bertbob has quit IRC
16:34:51 *** Mechtilde has quit IRC
16:37:13 *** Gerd has joined #gnucash
16:45:53 *** gjanssens has quit IRC
16:49:31 *** Unhammer has quit IRC
16:52:44 *** waeking has joined #gnucash
16:52:44 *** ChanServ sets mode: +v waeking
16:53:31 *** kael has joined #gnucash
16:53:31 *** ChanServ sets mode: +v kael
17:00:40 *** Unhammer has joined #gnucash
17:00:40 *** ChanServ sets mode: +v Unhammer
17:04:02 *** frakturfreak has quit IRC
17:18:46 *** sbluhm has quit IRC
17:22:03 *** kael has quit IRC
17:27:19 *** bertbob has joined #gnucash
17:27:19 *** ChanServ sets mode: +v bertbob
17:31:05 *** oozer has joined #gnucash
17:32:15 *** Han has quit IRC
17:34:35 *** jervin has joined #gnucash
17:46:18 *** Gerd has quit IRC
17:48:13 <chris> @tell gjanssens have a well deserved holiday!
17:48:13 <gncbot> chris: The operation succeeded.
17:48:25 *** calvinct has joined #gnucash
17:50:05 *** Gerd has joined #gnucash
18:14:41 *** kael has joined #gnucash
18:14:41 *** ChanServ sets mode: +v kael
18:15:06 <chris> my sysprof shows xaccSplitGetBook is called many times by xaccSplitOrder. low-hanging fruit to cache book.
18:19:22 *** kael has quit IRC
18:27:36 *** Gerd has quit IRC
18:35:48 <chris> guile's statprof written by Rob Browning, updated by wingo, ported by wingo to python.
19:01:01 <chris> Split.c#L1615 may be the hotspot - qof_event_gen after adding split
19:06:36 *** omnireq has quit IRC
19:19:30 *** Aussie_matt has joined #gnucash
19:33:47 *** Antic has quit IRC
20:15:20 *** omnireq has joined #gnucash
20:23:58 *** Agfarmer18 has joined #gnucash
20:27:20 *** Agfarmer18 has quit IRC
20:28:42 *** oozer has quit IRC
20:36:25 *** omnireq has quit IRC
20:36:30 *** omnireq has joined #gnucash
21:07:44 *** jralls has quit IRC
21:18:02 *** omnireq_ has joined #gnucash
21:18:06 *** omnireq has quit IRC
21:46:42 *** jralls has joined #gnucash
21:48:04 *** guak has quit IRC
22:02:50 *** jralls has quit IRC
22:24:37 *** jralls has joined #gnucash
22:58:20 *** omnireq_ has quit IRC
22:58:47 *** omnireq_ has joined #gnucash
23:19:50 *** omnireq_ has quit IRC
23:21:09 *** omnireq_ has joined #gnucash
23:47:41 *** jervin has quit IRC