12:01:21 #startmeeting Mer QA meeting 12/4/2012 12:01:21 Meeting started Thu Apr 12 12:01:21 2012 UTC. The chair is E-P. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings. 12:01:21 Useful Commands: #action #agreed #help #info #idea #link #topic. 12:01:55 o/ 12:01:59 Welcome all to Mer's second QA meeting 12:03:04 Meeting agenda and last meeting's actions can be found from here http://www.mail-archive.com/mer-general@lists.merproject.org/msg00414.html 12:03:24 thanks :) 12:03:37 any additions/changes to agenda? 12:03:42 looks fine to me 12:03:52 +1 12:04:26 o/ 12:04:59 +1 12:05:12 ok, let's start with the process 12:05:19 http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.log.txt to catch up, for those late joining 12:05:38 #topic QA Process 12:05:52 we had good discussion last time with lbt 12:06:54 and one action was filed about the how BOSS makes the test plan 12:07:13 :nod: there hasn't moved much on that side sadly to my knowledge 12:07:35 the current process is fairly simplified and done on a per-change basis 12:08:05 and a lot of manual work done to make a release 12:09:12 which on the other side means we can do things properly with QA :) 12:09:16 * Sage starts following 12:09:30 when a change is made (via gerrit), that now triggers automatically package rebuild in OBS? 12:09:50 or does that also require some manual work? 12:10:55 when a change is made in gerrit, it (automatically) will first check if the changed package itself will build, then run a build-test with the change package with all the other packages there as well 12:11:06 so it will rebuild any packages that depend on the package 12:11:39 when that's done, it'll report back to gerrit 12:12:14 an example can be http://review.merproject.org/314 , click "expand all" 12:12:42 when we speak of QA process, is there older models that were in use in meego we can use as reference to understand it better? 12:13:11 to gain terminology, etc 12:13:57 there are some terminology and process description in the wiki.meego.com 12:14:15 sadly the meego process never really took flight. the pieces were there but not really put together so I'd say we can agree on the terminology as we go 12:14:25 yep 12:15:35 sorry I'm late 12:15:57 http://wiki.merproject.org/wiki/Quality/Terminology helps at least 12:16:01 lbt: welcome 12:16:04 to be clear on the test plan thing... 12:16:09 http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.log.txt to catch up, for those late joining , mdfe 12:17:17 the meaning is "what logic should BOSS use to decide what tests to run" 12:17:33 good question? 12:17:43 tests depending on the package? 12:17:57 so that's something that should be designed and expressed by the group and (much) later, implemented in BOSS code 12:18:16 yes - it relates to the package naming conventions mentioned on the ml 12:18:36 personally I thing we should not use package naming - it's too simplistic 12:18:41 i think we need to possibly look at higher level view first, ie, change and 'release' testing 12:18:51 exactly 12:18:57 agree 12:18:57 package groups? 12:19:09 or meta 12:19:15 we do have a mer architecture too, you know 12:19:15 lets just say mapping 12:19:15 :P 12:19:25 :D 12:19:28 http://wiki.merproject.org/wiki/Architecture 12:20:05 *nod* ... and that's another area of testing 12:20:39 test-definition lets you define component and other information to it 12:20:40 May I ask how the BOSS is connected to a device, and how make sure the Device OS is in sync with the latest release? 12:21:05 mdfe_: lets leave BOSS out of it for now 12:21:10 k 12:21:29 we should say what steps we would ideally like to take... then we choose tools to execute steps 12:21:30 mdfe_: the general idea is that something would be creating an image, which then gets installed on your device, a test pc will then connect to device and run tests 12:21:37 BOSS will orchestrate those steps 12:21:39 like Stskeeps said, let's first define the higher level view 12:21:53 what kind of testing we want to have, and when they are executed 12:21:56 at the moment my biggest weakness is that i can't verify that a change will cause regressions 12:21:57 +1 12:22:08 E-P: yep 12:22:19 so mdfe_'s point is that there should be a process step that says "put the right OS on a device" 12:22:21 and it gives very late breaking problems 12:22:55 from the Mer core point of view, we would have: a change and release testing? 12:23:32 right, change to justify it's OK to merge something, release testing to justify the combination of changes is working? 12:23:51 E-P: the main reason for differentiation is the resource demands of testing 12:24:10 eg we can't run a week-long soak test before each commit :D 12:24:44 yep. that should be clear 12:24:50 so realistically change tests are lighter than release tests but IMHO only for that reason 12:25:10 do we want to test in the release testing something else than only changed packages? 12:25:28 ideally yes 12:25:39 the reason is that changes only represent build dependencies 12:25:44 lbt: yes, change tests should be light and fast to execute 12:25:56 API dependencies are not easily seen in this way 12:26:12 there is a need for both 12:26:13 eg if a service changes its dbus API 12:26:56 in meego we had a static tests for every release 12:27:06 and then "feature pack" set for changes 12:27:11 note: we must be careful to define some larger scope but deliver small steps 12:27:29 yep - that makes sense E-P 12:29:05 the static tests, eg. core tests, will test the overall status of the release 12:29:39 http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-12-12.01.log.txt to catch up, notmart 12:29:41 I'd like tests (test packages?) to be able to define test dependencies 12:29:53 in which sense? 12:30:04 'don't bother running these if the other ones don't pass'? 12:30:09 no 12:30:20 if any package that the test TDs on changes, run the tests 12:31:29 the test package should contain related unit tests for device deployment? 12:31:30 (yes, it's just a build-dep in one sense) 12:31:46 in maemo we had XB-Maemo-CI-Packages and XB-Maemo-CI-Stage fields in control file of the test package 12:32:14 which defined what packages the tests are for 12:32:18 and on what stage they should be run 12:32:19 Stskeeps: thx, sorry i was away ;) 12:32:26 Kaadlajk: I remember that, but that required a big database to work :) 12:32:30 Kaadlajk: yep, sounds reasonable 12:32:46 E-P: that also sounds right :/ 12:33:08 that's however a bit of optimization though, ie, optimization on top of 'what we ought to run if it was a full QA run' 12:34:09 so I can see a need for a service which does something like : Input=changed package list Output=ks of image with tests to run 12:34:30 from mer process POV, we need to strike at change (earliest we can check problems), and the various 'builds'/releases of full mer 12:34:44 yep 12:34:57 in mer, we have prereleases and full releases 12:35:18 each of these being a 'build', perhaps with some smaller releases leading up to it 12:36:02 so that also gives input to how we can test/make tests 12:36:06 +when 12:36:26 OK - but there is no reason to treat each commit any differently from a final build other than to filter which tests to run 12:36:34 :nod: 12:37:42 Maybe more detail : Input=changed package list & position in overall CI process (DBC, snap, pre, release) ; Output=ks of image with tests to run 12:37:51 E-P: btw, just a random note: if you see anything that for example me, or others's areas haven't done yet / not doing yet / should be doing to advance QA ability, you're more than welcome to tell us to do that :) 12:38:10 i know that our current process definition / release methods may be blocking things 12:39:30 Stskeeps: I definitely will 12:40:21 so since we've spent 40 minutes on this discussion, what can we conclude for this topic for now? 12:40:27 just to get moving on other agenda items 12:40:41 #info Mer Core would have change and release testing 12:41:24 #info Change testing is lighter than release testing 12:41:52 #info Release testing contains test for changed packages + Core test set 12:42:10 :nod: 12:42:16 is there agreement that the test:package mapping needs more detail than -tests: 12:43:07 i think it needs to be done on architectual domains instead, tbh, as sometimes one test will cover many packages 12:43:18 exactly 12:43:32 it's not just arch though 12:43:37 sometimes it will be 1:1 12:43:58 :nod: 12:44:13 we definitely need some kind of mapping 12:44:14 #info The test:package mapping needs more sophistication than -tests: 12:44:20 thanks 12:44:23 have you peeked at mwts-mcts-blts setup yet, lbt? 12:44:30 just to understand how that was done back in meego 12:44:57 setup as in, how they were laid out 12:44:59 does this mean the test package name is to contain the sense of it? 12:45:40 mdfe_: we're talking a bit on how we'd know what needs testing / what test suites we need to run, from a core perspective 12:46:23 (if i misunderstood, please rephrase question) 12:46:26 the mcts test packages has some faults, but we can discuss about them some other day 12:46:29 :nod: 12:46:29 ah, I got it now 12:46:34 mapping 12:46:39 ok 12:46:41 next topic? 12:46:49 anything to add to this topic? 12:47:27 not here 12:47:34 lets move on then 12:47:40 #topic Tools and tests 12:48:03 Like mentioned in the last meeting, Meego project provides many good tools 12:48:50 one issue is that where do we get tests? 12:49:08 yepp 12:49:24 we can use mcts (Meego Core test suite) tests, but they don't cover all the core packages 12:49:58 I think first using packages own tests if they have 12:50:17 for example qt has good and wide unittests 12:50:39 but also very big 12:50:56 part of the problem is that most tests from packages are meant to be run inside the source compilation tree 12:51:01 * lbt raises a question for AOB or next time : how do we handle version control of tests when they're not part of upstream 12:51:01 for cmake based packages I tried to seperate and package ctest based unit tests for target devices 12:51:21 I'm not in favour of running tests at build time 12:51:23 and for ARM targets, well, we don't run on arm hardware in practice 12:51:29 exactly 12:51:45 i'd be interested in innovative ideas on how we can extract those during build process, and run later on actual hardware 12:52:05 hack "make" and intercept make test? 12:52:22 or innovative *and* sane? 12:52:34 very good questions 12:52:41 i think one way we can approach all this is a 'week' goal, where we make it possible to add testability enabled images we can actually run simple tests against 12:52:50 that gives a better basis for all this work 12:53:15 and sets for what tools we need to package, work with 12:54:05 that is good idea 12:54:07 as another aside... it seems reasonable to be able to say "hmm. let me easily install run the tests for this package on this random image" 12:54:16 lbt: yes 12:54:27 +1 12:54:34 so after goal is accomplished, we can add test enablers to existing images such as Nemo, virtual machine images, Plasma Active, etc 12:54:46 and run tests from PC onto the devices 12:55:20 yup ... we could detect memory leaks and things like that 12:55:22 yep 12:55:25 I would like that the tests are own packages and you can just install them with zypper and execute them 12:55:31 agreed 12:55:46 i'm working a bit on 'eat' today to cover that side 12:55:48 +1 12:56:54 +1 for E-P 12:56:59 should these tests live in 'Core' ? I tend to say not (cf tools and -cross ... some comes from core but much doesn't and it's moved to a discrete area) 12:57:20 (I have to go soon, but you can continue the discussion without me) 12:57:28 :nod: i think we can wrap up soon 12:57:44 we have some goals and requirements/ideas, can we #info them? 12:58:00 lbt: i think external, as to make tests flexible to include new tools 12:58:10 they're obviously built against mer and released alongside 12:58:15 *nod* 12:58:20 +1 for that 12:58:24 lbt: In case they are in Core they are always available 12:58:46 that's another angle, yes 12:58:54 mdfe_: I see that as a slight negative 12:59:39 Stskeeps: can we use Mer OBS and git for storing some of the tools and tests? 12:59:39 In case someone has a bug, he is able to run the test if he like 12:59:52 mdfe_: +1 12:59:57 E-P: yes, mer-tests/* git is all yours if you need it 13:00:01 OBS we can make Mer:QA perhaps 13:00:13 we can put it on community obs for now 13:00:19 and then more officially release it later 13:00:25 great 13:00:28 #info tests should ideally be run at a discrete point in the process, not just/always inside the build environment 13:00:57 I'd like to package any QA tools into Mer:Tools 13:01:01 Tests into QA ? 13:01:12 Mer:Tools would be available in SDK? 13:01:14 #info tests should be easy to execute, eg using zypper to install them and then just 'executing' 13:01:18 yes 13:01:21 and Tests not necessarily ? 13:01:28 correct 13:01:37 though that might be a practicality, if we want to use it as a host 13:01:42 the tests come in .xml format, so 13:01:50 zypper ar 13:01:55 ok 13:01:59 or just enabling a repo 13:02:07 yep, that is good 13:02:27 #info QA tools should go into Mer:Tools 13:02:39 #info QA tests should go into Mer:QA 13:03:06 :nod: 13:03:37 anything to add? 13:03:40 anybody want to state their plans within QA area for the next week or so? 13:03:51 checking for more info points 13:04:08 #info work on bringing 'eat' up to mer/systemd ability 13:04:44 I can move some tools and tests to mer git and obs 13:04:50 #info innovative ideas welcome on how to extract tests made during the build process, to be run later on actual hardware 13:05:28 #action Create git and OBS projects for QA tools and tests 13:05:58 so ... can anyone (Kaadlajk?) take actions to write up either proposals or "how it's done elsewhere" notes on the test:package mapping 13:06:12 mdfe_: or requirements 13:06:40 actually I think we should refrain from proposals for a while yet 13:06:53 #info goal for nextg week: we make it possible to add testability enabled images we can actually run simple tests against 13:07:32 you have no idea of the stupid rabbitholes that involves 8D 13:07:43 :) 13:07:48 (but yeah) 13:08:02 are we good? 13:08:14 *cough* .... volunteers? 13:08:30 think so, else we can continue conversation in #mer - if you'd like to do anything, or doing it, just say :) 13:08:44 I can write something down how the CI testing was done in Maemo 13:08:58 E-P: thanks 13:09:00 that can help, gives a vendor perspective 13:09:21 hopefully timoph, iekku and Kaadlajk can fill that up then 13:09:21 #action E-P to document a little about how the CI testing was done in Maemo 13:09:52 don't count on me, i don't have any idea :( 13:10:12 #info OBS Publish process should distribute test packages to the right repos (cf -cross) 13:10:59 I have to go now 13:11:00 * lbt is all done ... ty 13:11:14 thank you 13:11:20 (all for coming) 13:11:21 let's continue the discussion in #mer and mailing list 13:11:34 #endmeeting