2017-11-16 GnuCash IRC logs

01:02:37 *** Mechtilde has joined #gnucash
01:24:29 *** Mechtilde has quit IRC
01:45:21 *** fell has quit IRC
01:46:22 *** fell has joined #gnucash
02:22:08 *** fabior has joined #gnucash
02:42:08 *** fabior has quit IRC
02:57:56 *** fekepp has quit IRC
03:21:52 *** gjanssens has joined #gnucash
03:21:52 *** ChanServ sets mode: +o gjanssens
03:21:55 <gjanssens> .
04:14:35 *** pilotauto has quit IRC
05:41:08 *** fabior has joined #gnucash
05:46:18 *** fabior has quit IRC
06:10:51 *** storyjesse has joined #gnucash
06:38:13 <chris__> lmat: I will require a time64 conversion for gnc_date_option_value_get_absolute in libgnucash/app-utils/option-util.c :)
06:43:27 <chris__> lmat: xaccQueryAddDateMatchTT will eventually benefit from a rewrite to avoid calling xaccQueryAddDateMatchTS
07:01:49 *** Jimraehl1 has joined #gnucash
07:03:00 *** Jimraehl1 has left #gnucash
07:36:48 *** chris has joined #gnucash
08:26:55 *** Mechtilde has joined #gnucash
09:15:59 *** chris__ has quit IRC
09:35:46 *** fabior has joined #gnucash
09:37:49 *** noah has joined #gnucash
09:49:43 *** FroL_Onn has joined #gnucash
09:51:16 <FroL_Onn> Dear all, several days ago I've visited this channel to ask about currencies rates downloading via GnuCash. I came back just to tell that updating Finance::Quote via cpan up to 1.47 solved the problem!
09:51:42 <warlord> FroL_Onn: glad to hear.
10:15:52 *** kael has joined #gnucash
10:17:34 *** storyjesse has quit IRC
10:21:43 *** ArtGravity has joined #gnucash
10:22:04 *** fabior has quit IRC
10:28:20 <lmat> chris: aye.
10:31:07 *** Mechtilde has quit IRC
10:38:52 <lmat> chris: This is a real treat. gnc_date_option_value_get_absolute takes a time_pair (I guess), it calls scheme gnc:date-option-absolute-time passing the given argument. The latter (gnc:date-option-absolute-time) should return a timepair wherefrom the 'car' is returned. If you would update or provide an alternative to gnc:date-option-absolute-time to take a time64 rather than a timepair, that will fix your
10:38:58 <lmat> problem.
10:39:19 <lmat> chris: Also, may as well do the thing right: perhaps you could change gnc:date-option-absolute-time to return a time64 rather than a timepair? Then I can change the C code to treat it accordingly.
10:44:52 *** fabior has joined #gnucash
10:52:49 *** gncbot sets mode: +o fell
10:53:53 *** FroL_Onn has quit IRC
11:09:50 *** fabior has quit IRC
11:10:49 *** ArtGravity has quit IRC
11:21:01 *** User has joined #gnucash
11:26:28 *** fabior has joined #gnucash
11:37:40 *** fabior has quit IRC
11:46:26 *** GabrieleV has quit IRC
11:50:35 *** GabrieleV has joined #gnucash
11:53:43 *** GabrieleV has quit IRC
12:00:35 *** GabrieleV has joined #gnucash
12:05:07 *** GabrieleV has quit IRC
12:05:10 *** fabior has joined #gnucash
12:08:58 *** Mechtilde has joined #gnucash
12:11:29 <jralls> gjanssens: Too invasive because the proposed change is to the behavior of Timespec: Early authors treated [{0, 0} as an invalid Timespec. Making that a valid Timespec and using INT64_MAX instead would touch at least Transaction.c, Split.c, and both backends.
12:14:48 <jralls> gjanssens: "At least" because those are the files needed to fix https://bugzilla.gnome.org/show_bug.cgi?id=784623. There are probably other instances that should be tracked down and changed as well.
12:15:35 *** GabrieleV has joined #gnucash
12:16:27 <jralls> CDB-Man_: Might https://bugzilla.gnome.org/show_bug.cgi?id=775368 have something to do with your imbalance problem?
12:23:40 * gjanssens is reading that bug to get a better understanding of the bug
12:25:15 <gjanssens> jralls: comment 13 mentions removing STRICT_TRANS_TABLES from the sql mode seems to fix it as well
12:25:35 <gjanssens> The mysql documentation suggests this can also be done by the client
12:25:59 <jralls> gjanssens: More of a work-around than a fix, and apparently works only for MySQL 5.5 and earlier.
12:26:08 <gjanssens> Perhaps that could be a workaround on 2.6
12:26:25 <gjanssens> Heh, you were faster
12:26:50 <gjanssens> The comment seems to suggest differently
12:27:10 <gjanssens> The auther did this on MySQL 5..7
12:27:43 <gjanssens> But granted it won't work on mysql 5.6 as that doesn't have that mode yet
12:30:23 <gjanssens> If that's annoying enough to want to pursue a better temporary fix for 2.6 would you think this would require a schema change ?
12:30:23 <jralls> Ah, perhaps I misunderstood the followup. There's also the issue that David C brings up in comment 14, that timezone handling can cause GnuCash to try to store times earlier that 1970-01-01 00:00:00. That won't work at all.
12:31:10 <gjanssens> Or only changing the internal interpretation of the stored data
12:31:31 <gjanssens> Combined with mysql safe writing ?
12:32:52 <gjanssens> And when do we hit this issue ? When reading 0 values from a Timespec field or when writing it ?
12:34:36 <jralls> I'm not sure I quite understand. The schema fix, using DATETIME instead of TIMESTAMP for MySQL, is the true fix, and is needed for MySQL to not have the 2038 bug. It's actually a MySQL bug in that their implementation of TIMESTAMP violates the SQL standard.
12:35:16 <jralls> But they
12:35:48 <gjanssens> That proper fix is for 2.8 as I understand it
12:35:48 <jralls> But they've had their heads up you-know-what since at least 2008 about that so they're not likely to fix it.
12:35:55 <jralls> Right.
12:36:39 <gjanssens> And you asked my opinion on what to do on 2.6, so I'm now trying to estimate the impact
12:37:25 <jralls> Right. I'm not questioning the motivation, I don't understand the "if it's annoying enough" questions.
12:39:27 <gjanssens> With "annoying enough" I meant "would it be too annoying for the users to have to skip mysql 5.6) for the time being
12:40:27 <gjanssens> Well, anything up until 5.7.5 actually, as that is the version that first has the STRICT_TRANS_TABLES mode
12:41:10 <jralls> The original problem was that we weren't storing a timestamp if the Timespec was {0, 0} and MySQL was returning 0000-00-00 00:00:00 and that crashed the date parser. So I changed it to store 1970-01-01 00:00:00 instead. I think that was the wrong thing to do.
12:42:27 <gjanssens> Ok, is that old or new MySQL returning 0000-00-00 00:00:00 ?
12:43:40 <gjanssens> But at least I get from this that disabling STRICT_TRANS_TABLES won't be sufficient because of the timezone hazard, right ?
12:44:41 <jralls> I think it's always done that, but I'm not absolutely sure. Yes, the timezone hazard would also need to be special-cased.
12:46:38 <jralls> Hmm, maybe I should just change the schema in 2.6. Just because nobody's reported being tripped up by having an SX extend past 2037 doesn't mean it won't happen.
12:47:47 <gjanssens> The drawback of doing that is users can't switch between 2.6.19 and 2.6.18
12:48:03 <gjanssens> That's why schema changes are generally done over major release cycles
12:48:49 <gjanssens> But what I don't get is why the 0000-00-00 00:00:00 return value started crashing the date parser ?
12:48:53 <jralls> gjanssens: Well, at this point they can't use 2.6.18 at all, so that's not necessarily a problem. :-(
12:49:20 <jralls> Because it's outside the valid date range for GDateTime.
12:49:55 <jralls> Never mind boost::date_time, which starts 1399 years later.
12:50:12 <gjanssens> That I can follow. Is there a bug report for this crash that you used to fix this ?
12:50:36 <jralls> No, it was a report on the user list. Give me a minute...
12:50:52 <gjanssens> In the bug David T. suggested your "fix" was still in there
12:51:14 <jralls> It is.
12:51:25 <gjanssens> Of course :)
12:52:11 <gjanssens> I'm trying to estimate whether this has always been a problem and nobody noticed or whether the crash is due to a change in MySQL/libdbi behaviour
12:55:14 <jralls> This is the thread that got me to make the change: https://lists.gnucash.org/pipermail/gnucash-user/2017-June/071030.html
13:00:25 <jralls> So there's two aspects to this: One is about the handling of KVP in SQL: Because SQL can't do multiple types, Phil Longstaff implemented a table with a column for each possible type. It should store NULL in all fields except the one that matches the type.
13:01:53 <jralls> The other is that the user who reported the problem on the list had a bad date_posted in a transaction and "that" got stored as a NULL, which returned a date that the parser couldn't handle. I mistakenly "fixed" the wrong end of the problem.
13:04:33 <gjanssens> Ok, it seems to me for 2.6 you could revert your initial change to write 1970-01-01... and make the parser properly accept the 0000-00-00.... returned by mysql
13:05:25 <gjanssens> Internally in gnucash we should treat it as whetever we consider invalid, like {0,0}
13:07:01 <gjanssens> What problem do you see in the KVP handling ?
13:08:04 <jralls> Right. The thing is that since the GncDateTime change (introduced in 2.5.something) {0, 0} isn't invalid. Hence my question about changing to INT64-MAX like we decided to do in csv-imp.
13:09:10 <gjanssens> It evaluates to the 1970-01-01... date, right ?
13:09:27 <jralls> OTOH with the current schema {0, 0} and earlier *is* invalid for MySQL.
13:09:58 <jralls> Yes, 0 is midnight, 1 January 1970.
13:10:37 <gjanssens> Ok, how big are the odds we currently have users with sensible data that is using exactly the 1970 timestamp ?
13:10:46 <gjanssens> As in deliberately entered.
13:11:14 <gjanssens> I'm tempted to treat it as invalid in 2.6 still just to minimize the amount of work to do
13:11:25 <gjanssens> Invalid in the mysql context
13:12:16 <gjanssens> If the sql code gets a request to store a timestamp equal to or less than 1970-01-01, treat it as invalid and store NULL
13:12:17 <jralls> I think it's pretty small. Someone would have to be pretty compulsive to enter 50 year old transactions.
13:13:14 <gjanssens> When reading from MySQL, if you get a 0000-00-00 timestamp back, use {0,0} for now. I think most of gnucash should handle that correctly.
13:14:23 <gjanssens> The cleanup can be done properly for 2.8
13:14:53 *** fabior has quit IRC
13:15:10 <jralls> OK. What about dates past 2037 that will also fail on MySQL? Those get more likely every year.
13:15:47 <jralls> A special error code in QofBackend?
13:15:56 <gjanssens> Better get 2.8 out as soon as possible!
13:16:30 <gjanssens> I like a special error code. At least we can then inform the user there is an issue with its data in 2.6
13:16:54 <jralls> OK.
13:41:41 <lmat> gjanssens: Regarding "2.6 only needs to understand the format" earlier, do I need to do anything to make sure 2.6 doesn't write the file?
13:42:24 <lmat> gjanssens: I'm ready to write the code to make 2.6 be able to read the flat bayes kvp. (It looks like this was moved from import-match-map.c to Account.c on master.)
13:42:47 *** fabior has joined #gnucash
13:49:59 *** fabior has quit IRC
13:55:39 <jralls> lmat: I moved it into Account.c as part of KVP privatization. If you want 2.6 to be able to read but not write you'll need to set a feature and force opening as read-only.
13:56:21 <gjanssens> lmat: it can write as I explained yesterday
13:56:46 <gjanssens> But it should write new keys in the non-flattened format
13:57:37 <gjanssens> I was just thinking about a feature flag though as well
13:57:59 <lmat> gjanssens: Oh, I see! That's a bit of a snag. The code I have currently assumes that the data are converted or not converted rather than some mix.
13:58:49 *** carwynnelson has joined #gnucash
13:58:50 <gjanssens> The point is whatever 2.6 writes should be readable by older gnucash 2.6 versions
13:58:53 <lmat> gjanssens: (October 13, I said, " The tests for the conversion from account names to guids are interestingly robust. Is it likely that we'll see books with imap information that has some accounts in GUIDs already, and some with account names?" I guess I have my answer!)
13:59:26 <gjanssens> Yes
14:00:01 <gjanssens> However if this is easier to code, I can imagine another valid way of handling things
14:00:11 <lmat> I'm all ears.
14:00:28 <gjanssens> 2.8 always writes flattened/guid converted
14:00:45 <jralls> lmat: It's important to distinguish between imap and imap-bayes in this case: imap has used GUIDs for a long time, imap-bayes only for 2.8. I thin you discovered that yesterday.
14:00:52 <lmat> certainly.
14:01:33 <gjanssens> 2.6.(19) will choose based on whether there are already flattened/guid converted imap-bayes entries
14:01:53 <gjanssens> If so, it will also resort to writing flattened/guid converted only
14:01:58 <gjanssens> If not, it will write the old style
14:02:33 <gjanssens> In both cases at least one flattened/guid converted entry is found, raise a feature flag
14:02:45 <lmat> I understand.
14:03:33 <gjanssens> That's probably easier to implement as you can keep old and new writing behaviour side by side in 2.6
14:04:19 <gjanssens> And the net effect is the same: if one flattened entry is found, older 2.6 will stop reading the file because of the feature flag
14:04:47 <gjanssens> And only a first use of 2.8 can change the storage format
14:06:38 <lmat> gjanssens: May as well mention that 2.6 bayes import maps are busted if a user has "/" in his tokens. They don't work at all.
14:07:09 <gjanssens> lmat: you mean as they are currently implemented ?
14:07:14 <lmat> gjanssens: yes.
14:07:24 <lmat> gjanssens: This removes the necessity of having to reconvert the file in 2.8.
14:08:29 *** Mechtilde has quit IRC
14:09:16 <gjanssens> Hang on, so if I have a file I have never used with 2.8 and there are "/" in my bayes import maps, that won't work on 2.6.18 and earlier either
14:09:28 <lmat> yes.
14:09:56 <gjanssens> Ok so what has reconverting in 2.8 have to do with this ?
14:11:05 <lmat> Oh, I'm incorrect.
14:11:29 <lmat> "busted if a user has "/" in his tokens." is wrong. I forgot, 2.6 doesn't parse "/" within kvp keys!
14:11:48 <gjanssens> Ok
14:12:13 <lmat> But, to answer your question: following these guidance, the file should always be consistent: either fully not converted or fully converted.
14:12:14 <gjanssens> But if implemented in the alternative way, you are correct 2.8 needs to do the conversion only once
14:12:24 <gjanssens> Indeed
14:12:35 <lmat> Right. Once 2.8 does the conversion, later 2.6s will perpetuate the conversion.
14:12:43 <gjanssens> Exactly
14:12:53 <lmat> Now let's see if I can rewrite that code in C :-/
14:12:57 <gjanssens> That's quite elegant actually
14:13:15 <lmat> And early 2.6s will refuse to read converted files.
14:13:29 <gjanssens> Yep
14:13:43 <gjanssens> Bummer about the C++ -> C job
14:13:56 <jralls> I don't think that's correct. Isn't that the nature of https://bugzilla.gnome.org/show_bug.cgi?id=761682?
14:14:46 <lmat> yeah...it's nice and short in c++ usind std::aggregate, std::string::operator +, std::vector, etc. ::sigh::
14:14:58 <lmat> jralls: Say more :-) you don't think *what* is correct?
14:16:30 <jralls> Sorry, the conversation moved on while I was finding the bug and I think you realized it: 2.6 and previous *do* accept '/' in key names.
14:16:44 <lmat> I say that 2.6 doesn't parse "/" (and doesn't exhibit 761682) based on my testing lately.
14:16:49 <lmat> Okay, yup.
14:18:09 <lmat> Tell me about the release schedule. 2.7 is available for unstable testing currently? And 2.8 is thought to be released early 2018? Is there a plan for a freezing of the code and user testing? If so, how long?
14:19:56 <jralls> I said on https://wiki.gnucash.org/wiki/Release_Schedule#Schedule "Tentative code and string freeze in January 2018".
14:20:38 <lmat> jralls: Okay, thanks for pointing that page out to me.
14:20:43 <jralls> But all of that depends on whether we're ready for it. There are some critical issues to fix and get user tested.
14:21:11 <jralls> Flat KVP is certainly among them.
14:22:27 <lmat> yeah. I'll keep working then!
14:22:44 <jralls> I also said that 2.6.19 would be concurrent with the 2.8.0 release, but I now plan to do that as a regular quarterly release because of the MySQL date mess.
14:23:38 <jralls> It would be great if you can get the 2.6 flat-KVP changes ready by then.
14:24:43 <lmat> jralls: Why would we care for 2.6 to be able to read flat KVP bayes map? 2.8 is the first version to be able to write such a map.
14:25:36 <jralls> lmat: Isn't that the whole discussion you've been having with gjanssens for the last hour?
14:26:01 <jralls> lmat: 2.6.last needs to be able to read files written by 2.8.0.
14:26:08 <lmat> jralls: When 2.8 is released, it is important that latest 2.6 is able to read 2.8-generated files. Oh...
14:26:33 <lmat> jralls: You're talking about a 2.6.19 release, then a 2.8 release. (no 2.6.20 before or with 2.8)
14:27:50 *** frakturfreak has joined #gnucash
14:27:56 <jralls> I'm now talking about a 2.6.19 on 17 December and a 2.6.20 concurrent with 2.8.0. That last assumes 2.8.0 can be ready by 31 March, when we'd normally do the next quarterly release of 2.6.x.
14:28:05 <gjanssens> lmat: the need for a 2.6.20 release depends on whether we still have backwards compatibility issues by then
14:28:55 <lmat> Okay, I see.
14:29:03 <lmat> Have there been any movements in the direction of automated functional testing?
14:30:15 <jralls> gjanssens: If there are any changes pushed to maint after 2.6.19 I'll do a 2.6.20 to pick them up, just as I did 2.4.15 concurrent with 2.6.0.
14:30:42 <jralls> lmat: How does automated functional testing differ from unit tests?
14:31:41 <gjanssens> jralls: ah, right
14:32:27 <lmat> jralls: It would be using the application with a mouse and keyboard in an automated fashion. Whereas unit tests test "units" (usually small portions of codes such as single functions or classes), functional tests will perform whole functions like integration testing or end-to-end testing.
14:33:33 <lmat> It may not be worth the trouble, but I wanted to know if anyone has brought it up before.
14:34:22 <jralls> Ah, GUI testing. That's really hard to implement in a non-fragile way on Gtk. WxWidgets has a more test-friendly implmentation, see https://sourceforge.net/projects/wxguitest/.
14:34:26 <lmat> https://ldtp.freedesktop.org/wiki/ is an example I think.
14:34:53 <lmat> jralls: I associate automated functional testing as brittle, but I'm interested to see this wxwidgets thing! reading.
14:36:11 <lmat> Well that's great.
14:37:09 <lmat> How does converting the GUI go? It has to be done "whole-hog", right?
14:37:14 <lmat> (rather than piecemeal)
14:37:20 <jralls> Interesting. I hadn't thought of using the a11y API for discovering GUI elements.
14:38:10 <jralls> lmat: You mean converting from Gtk to wx or QT? Yes, it's all-or-nothing.
14:39:04 <jralls> Though one could do the work a bit at a time on Linux, where wx uses gtk as the backend.
14:39:32 <lmat> hmm
14:39:48 <jralls> But one would have to have everything working before one could start testing and tweaking on the other platforms.
14:41:14 <gjanssens> Just for completeness, I found this for Qt based gui testing: http://doc.qt.io/qt-5/qtest-tutorial.html
14:41:19 <jralls> That said, it's possible to write functional tests on non-GUI modules. Much of the oldest test code is functional.
14:41:59 *** gjanssens is now known as gjanssens_afk
15:17:48 *** fabior has joined #gnucash
15:29:55 *** fabior has quit IRC
15:38:13 <lmat> I'm trying to build gnucash/maint, but it's not working with the following error:
15:38:16 <lmat> /usr/lib/gcc/x86_64-pc-linux-gnu/7.2.0/../../../../lib/libwebkitgtk-1.0.so: undefined reference to `cairo_gl_surface_get_width'
15:39:47 <lmat> nm -D on the library shows U cairo_gl_surface_get_width
15:42:51 <lmat> Oh, from the archlinux gnucash package page: " the only cure for it is to rebuild webkitgtk2 and goffice0.8. "
15:43:20 <lmat> Now I'm trying to decide which is more trouble: a new docker image for this or rebuilding webkitgtk :-/
15:46:12 <lmat> I'll try an ubnutu 14.04 docker image.
15:47:36 *** fabior has joined #gnucash
15:50:49 *** fabior has quit IRC
15:51:45 *** fabior has joined #gnucash
15:56:51 *** carwynnelson has quit IRC
15:58:00 <jralls> lmat: Might be easier to rebuild cairo with GL support.
16:04:43 *** carwynnelson has joined #gnucash
16:05:40 *** carwynne_ has joined #gnucash
16:05:42 *** carwynnelson has quit IRC
16:15:01 *** kael has quit IRC
16:34:49 *** pilotauto has joined #gnucash
16:34:57 *** fabior has quit IRC
16:52:30 *** bunk3m has joined #gnucash
17:01:04 *** carwynne_ has quit IRC
17:13:41 *** chris__ has joined #gnucash
17:13:44 *** chris__ is now known as chris_
17:15:50 *** chris is now known as chris-laptop
17:16:12 *** chris_ is now known as chris
17:19:33 *** carwynnelson has joined #gnucash
17:26:31 <chris> lmat: I figure gnc_date_option_value_get_absolute calls scheme (gnc:date-option-absolute-time)... which calls (gnc:get-absolute-from-relative-date) ... which calls the (gnc:reldate-get-fn) = (vector x 3)... which means numerous functions such as (gnc:get-start-cal-year) -- in my branch I've already converted all the return values to time64.
17:28:28 <lmat> chris: Okay, so gnc:date-option-absolute-time takes and returns time64?
17:29:41 <lmat> chris: And what time is it where you are?
17:30:35 *** gjanssens_afk has quit IRC
17:31:07 <chris> well (gnc:date-option-absolute-time) takes a 'option-value' whose car is either 'absolute or 'relative, and its cdr is 'start-cal-year / 'end-cal-year / 'start-prev-year etc
17:31:30 <chris> example of option-value: (cons 'absolute 'start-this-month)
17:31:54 <chris> oh no mistake
17:32:20 <chris> its car is 'absolute, its cdr seems to be storing the time
17:32:27 <chris> i'ts 6:30am here
17:35:24 <lmat> chris: That's Beijing time?
17:35:33 <chris> the code for (gnc:date-option-absolute-time) - take option-value - if its car is 'absolute then return its cdr. else call (gnc:get-absolute-from-relative-date) on its cdr and return it.
17:35:45 <lmat> Hmm, so what is being passed to gnc_date_option_value_get_absolute?
17:38:05 <chris> hm give 1min i'll test
17:38:09 <chris> Perth,WA :)
17:43:34 <chris> ok the argument to gnc_date_option_value_get_absolute - its car is always 'absolute or 'relative, its cdr is either 'start-accounting-period 'start-cal-year etc.... or a timepair
18:20:25 <chris> I'll need to dig into (gnc:make-date-option)
18:25:34 <chris> Well, (gnc:make-date-option) will just generate a few data structures which gets passed on to the GUI... somewhere else actually creates the timepair. I could either fix this with a shim now and move on to the other scm files, or, dig further and possibly get lost
18:28:56 *** User has quit IRC
18:30:17 *** frakturfreak has quit IRC
18:40:24 *** noah has quit IRC
18:47:13 *** carwynnelson has quit IRC
19:31:26 *** kael has joined #gnucash
19:32:15 *** kael has quit IRC
19:38:37 *** codesmythe has joined #gnucash
20:44:18 *** dougransom has joined #gnucash
21:18:41 *** bunk3m has quit IRC
21:40:46 *** User has joined #gnucash
21:52:10 *** raeburn has joined #gnucash
21:55:45 *** raeburn has quit IRC
22:00:11 *** User has quit IRC
22:38:33 *** kael has joined #gnucash
23:21:18 *** marusich has joined #gnucash
23:41:59 *** kael1 has joined #gnucash
23:43:28 *** kael has quit IRC
23:43:29 *** kael1 is now known as kael