12:07:39 #startmeeting Release management meeting 20/3/2012 12:07:39 Meeting started Tue Mar 20 12:07:39 2012 UTC. The chair is Stskeeps. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings. 12:07:39 Useful Commands: #action #agreed #help #info #idea #link #topic. 12:07:45 * Sage_ stops the calendar reminder 12:07:51 so, long story short last release was a nightmare :) 12:08:02 and has shown why we need properly automated processes 12:08:04 .. and more disk space 12:08:33 how much does one release take disk space btw? 12:08:39 ~7Gb 12:08:56 and transient processing takes more 12:09:07 ok 12:09:43 Should we make policy that after a 1 month the snaphots/prereleases are removed and only releases are kept? 12:10:11 https://bugs.merproject.org/show_bug.cgi?id=175 12:10:39 * Sage_ cc's 12:10:50 so yes, I agree completely 12:11:02 makes sense 12:11:13 but either way, with CI coming in to play, we need to get more space for the release builder 12:11:28 we actually have a lot of disk space 12:11:35 it's just not well used 12:11:39 right 12:11:54 don't some of the hots have like 2x3TB disks? 12:11:58 hosts* 12:12:02 some of that is down to me not cleaning up infra 12:12:14 so, to the point -- how do we best get started implementing a saner release process? 12:12:25 oh, before that 12:12:40 i managed to screw up Core:armv7hl on sunday and i successfully resurrected it with latest mer release 12:12:52 just by building 12:13:03 which was a good experience as we know Mer is resurrect-able in case we get shut down for whatever reason 12:13:07 REL & CI counts ok? 12:14:04 yes, to my surprise, however, i think merds delivers wrong information 12:14:15 http://releases.merproject.org/~prjfetcher/ver.txt , CI_CNT should definately not be 1 at all times 12:15:00 #info We made a mistake and lost a part of the Mer release and rather than just restore we were able to verify that a rebuild worked as expected. This is very useful from a DR perspective. 12:15:53 ish, yeah 12:16:29 lbt: do you know how far phaeron got with putting his copy-whole-project patches on github? 12:16:47 I know it was close 12:17:17 couldn't say if it was finished though 12:19:21 I would like to tidy up the infra as a high prio task 12:19:38 we shouldn't really be running on the phosts 12:19:47 running? 12:19:57 ah, point 12:20:15 so, task, move prjfetcher 'stuff' to another vm? 12:20:23 yes 12:20:33 makes sense 12:20:50 and releases.* hosting too 12:20:58 so probably a release staging VM 12:21:08 and a release distribution VM 12:21:30 ok 12:21:42 #info move prjfetcher releasing scripts to release staging VM 12:21:53 #info move releases.* www+rsync to release distribution VM 12:22:10 just keep in mind releases.* has gerrit pushing to it 12:22:36 *nod* 12:23:14 I need to look at where source/git lives too 12:23:39 git lives on gerrit and mirrors to releases.* automatically 12:24:18 ok, so, process 12:24:22 I'm also keeping http://airy:5001/ as the primary docs place 12:24:47 i'm going to be a difficult person and ask if we can get that behind a htpasswd'ed url 12:24:58 i don't always have my ssh forwarding on 12:25:01 Sage_: I'm not sure what access you have to the infra - if you ever want to see anything then just yell 12:25:29 so it ends up with me not utilizing that wiki properly 12:25:34 Stskeeps: sure, I just never needed it 12:25:39 lbt: atm. I don't think I have access but I'm not really wanting one eithe atm. 12:26:10 #info task: airy wiki accessible through password protected url 12:26:35 I'm a bit busy as is anyway so wouldn't benefit anyone. If I don't need the access I don't want one. I'll inform if situation changes. 12:26:36 (ideally ssl) 12:26:37 Stskeeps: my main concern is that we need to be careful about recording credentials on there 12:26:43 lbt: naturally 12:26:45 ssl mandatory 12:29:17 lbt: i don't even mind ssl client certificates 12:29:18 :P 12:29:36 either way 12:30:46 process.. http://wiki.merproject.org/wiki/Process#New_process_2 12:31:22 the idea is that for CI checks we copy from 'master' to a change-specific project, then be able to release that as a full repo 12:31:30 so we can run automated tests on it, have vendors test it, etc 12:32:14 So in general we are moving all non-OBS-worker stuff off the phosts 12:32:14 I may have to take up vgrade's offer of a new phost 12:32:14 I'll wait until this afternoon's OBS meeting though 12:32:35 ok 12:35:50 i guess i should work more on MDS2 12:36:10 hmm 12:36:43 right now I'm more interested in a release process with more QA 12:37:08 I'd say to leave process v2 for a while - it depends on unwritten/deployed code anyway 12:37:24 and focus on automation on the existing CI and release process 12:37:36 installed BOSS is old apparently 12:37:39 i can only say that you need v2 in order to do this 12:37:49 i wouldn't be talking about it else 12:38:11 fair enough 12:38:29 we can't do QA image builds properly against partial repos 12:39:09 ah 12:40:06 I got this working just fine in Nokia... 12:40:21 yeah, well, we're not nokia 12:40:22 :P 12:40:47 so I'm trying to figure out what's blocking us now that didn't then 12:41:10 ks handling ? 12:42:01 no, MDS is one part of why 12:42:26 and the fact we're exporting things to vendors 12:43:11 I see 3 areas 12:43:28 CI img/rootfs tests as part of commit review 12:43:55 img/rootfs tests as part of release verification 12:44:11 possibly img/rootfs tests as part of vendor early-sharing 12:45:40 i think img/rootfs is too narrow a focus right now, we need to look ahead to running tests as well 12:46:05 on what though? 12:47:11 because I said "img/rootfs _tests_" ... ie create rootfs/img and run a (possibly null) set of tests on it 12:47:21 okay, if you don't mind i'll just run through what i see we need, get some terminology on same page, etc, without discussing specific technology 12:47:45 sure 12:47:52 creating an image is very nice test already so we avoid fiasco that was done with previuos release :) 12:48:09 exactly 12:48:36 tests is next step after that but at least we should do images (no need to boot either) just do image and check that they build ok. 12:48:38 a release = project-core consisting of packages, which are pointers to specific sha1's in packages git, + binaries matching these sources 12:49:00 you can run image builds against this, or build packages 12:49:08 any objections to that definition? 12:49:26 it's broad 12:49:27 ps. create image after each submit might be a bit too heavy in process but we should do it before (pre)release at least. 12:49:54 it encompasses both the MDS and the http://../releases 12:50:00 right 12:50:41 so if we split it to rpm release and mds release 12:50:56 then we can 'easily' test the rpm release wrt images 12:51:04 that's useful and incremental 12:51:15 a change = one or more modifications to packages and/or project core description 12:51:30 then we can move on to MDS testing (bearing in mind MDS2 isn't even written yet) 12:51:51 yep - that's OK 12:51:53 oi, MDS2 is written fine for the basic parts 12:51:55 :P 12:52:08 OK 12:52:29 so, old release + change = new release, right 12:52:34 Sage_: it's quicker than you imagine to make an image on a fast machine 12:53:16 Stskeeps: OK 12:53:40 lbt: under 5mins? I mean that there is need to do at least one image per arch. 12:53:44 (but old release + change doesn't mean you have to make a new release) 12:53:52 Sage_: worker farm 12:54:38 ok, I know it can be done, but as we don't have too much resources atm. AFAIK I meant that we should put those resources do images if we have better things for them. 12:54:48 if we already have the capacity why not 12:54:56 when a change comes in through gerrit, for example, we take an old release, add the change, and we make a 'release' based on that 12:55:20 Sage_: we have quite some spare capacity for IMG on the SLX machine 12:55:21 to test if it would be valid still 12:55:51 Stskeeps: OK - now you're using release in a way that I think will confuse 12:55:56 I know what you mean though 12:56:18 old release + change = release candidate (rc) 12:56:31 right 12:56:41 ie "we could release anytime if we wanted too but we'll only do it every other thursday" 12:56:58 that release candidate is then what is sent for testing around the place 12:57:12 ABI tests, image creation tests, automated test cases, etc 12:57:33 vendor checks.. 12:58:43 yes 12:59:03 vendor-checks is ... interesting 12:59:21 and here is where rpm vs mds gives us 2 deliverales 12:59:25 what doe vendor-checks mean? 12:59:42 Sage_: we signal to vendors there's a change that they want to QA 12:59:54 ok 13:00:01 for example, to the x86 adaptation vendor, please verify that this change doesn't break your systems 13:00:05 I'd like to focus initially on getting those rpms available in a state that permits img 13:00:16 now, just to tie this in with what i meant with process v2, in order for us to do old release + change = release candidate, we need to use project-copy 13:00:29 well 13:00:43 we need to ensure access to all those rpms 13:01:28 which may be as horrid as running create-rc-release.sh against the transient OBS project 13:01:45 so project-copy is certainly a solution 13:01:55 and I agree it's a good solution 13:02:08 but it blocks us until it's deployed 13:03:10 it does, but we can't work with 'blob' obs project practically -- you saw how the sync script nuked our armv7hl stuff as OBS protocol doesn't relay information on added packages when doing linked projects 13:03:31 ie, right now, i manually have to linkpac and rdelete packages upon update 13:03:34 it's not sustainable 13:03:49 update=upon new package, or delete package i mean 13:04:56 lbt: ok, instead of release, read "build" 13:05:49 yeah - just being careful on meanings 13:07:10 and project-core should be managed by process instead of manual work 13:07:37 so ... I was just noticing the insertion of a solution into the "no specific tech" part 13:07:52 i just wanted to tie that in in order to explain why it's a problem 13:08:06 and why we need this in order to continue 13:08:28 yep - so that part of the process (ie create an image after a trial build) needs to handle new+deleted packages 13:08:48 that's an issue and one solution is project copy 13:09:06 on top of that, we need to support topic branches 13:09:09 (and I think we've discussed that it's the best solution) 13:09:23 ie, changes that require other changes 13:09:46 new terminology alert 13:11:28 also .... looking at the time and wondering if we should go back to the RM meeting and then continue this discussion in #mer straight after? 13:12:05 this is fairly RM :P but we should close RM meeting soon anyway 13:12:47 agreed - just worried it may overshadow other things - if there are any? 13:12:58 we have plans from last meeting at least 13:13:00 and it's a huge and really important topic :) 13:13:08 (releases and stuff I mean) 13:15:05 hmm, where was the MINT github? 13:15:14 So Qt5 and Python2.7 next release 13:15:27 right, if possible 13:15:33 eglibc 2.15 is getting tested through atm 13:15:47 https://github.com/Merproject ? 13:16:15 https://github.com/MeeGoIntegration was looking at 13:16:31 oh yes 13:17:35 let's wrap up now and continue discussion after you've done the VM moves 13:18:05 as that's prerequistes for anything anyway 13:18:47 OK - that'll take a while too 13:18:55 I would like to see qt5 packaging already in review so that could be commented 13:18:58 both volume of data and planning 13:19:01 otherwise it goes quite close 13:19:02 Sage_: yes 13:19:13 at least the qt5 core or what ever 13:19:16 at least qtbase as a start 13:19:20 yes 13:19:29 IMO should be qt5-base or something 13:19:46 I will get an SB2 SDK out before starting the VM moves too 13:19:58 Sage_: qt5base 13:20:15 I also have to prep for this afternoons OBS meeting 13:20:22 yep 13:20:26 thank you all for coming 13:20:53 tt 13:21:02 ty even :) 13:21:13 #endmeeting