09:00:37 <chriadam|windows> #startmeeting CalDAV/CardDAV Test Server Planning
09:00:37 <merbot> Meeting started Mon Sep 19 09:00:37 2016 UTC.  The chair is chriadam|windows. Information about MeetBot at http://wiki.merproject.org/wiki/Meetings.
09:00:37 <merbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
09:00:46 <chriadam|windows> Hi dr_gogeta86, occirol, lbt
09:00:53 <chriadam|windows> I will stay out of the way as much as possible in this meeting
09:01:02 <chriadam|windows> since I don't know anything about system administration or the Mer infra ;-)
09:01:18 <chriadam|windows> I guess from my POV I would like to see:
09:01:33 <chriadam|windows> 1) concrete plan for setting up the first test server instance (preferably owncloud or nextcloud) within Mer Infra
09:01:46 <chriadam|windows> 2) a timeline for us to aim for (doesn't have to be concrete timeline, but best-effort ETA)
09:02:01 <chriadam|windows> lbt: I guess I'll hand over to you?
09:02:39 <chriadam|windows> I suppose the first step is to find out what you can provide on the Mer infra, and by when.  And then we can ask dr_gogeta86 and occirol whether they can use those resources / VMs to set up what we need to do the caldav/carddav testing?
09:03:42 <lbt> in terms of infra we use kvm VMs
09:04:21 <lbt> typically I restrict access pretty heavily for obvious reasons
09:04:41 <occirol> seems fair :)
09:04:47 <lbt> so shell access and suchlike to most infra is just not available - never mind root :)
09:05:19 <lbt> however I think we can set up a more isolated system for this kind of thing
09:05:55 <lbt> I'd want to use a different VPN to connect these machines together and then setup a gateway/firewall for monitoring and potentially remote control
09:06:41 <lbt> ie so the community machines wouldn't be able to reach into the mer infra but we could deploy eg a web api to reach out to the community machines to control them
09:07:06 <chriadam|windows> sounds sensible.  are there turnkey OSS solutions to do that sort of thing?
09:07:11 <chriadam|windows> or is it going to be painful to set up?
09:07:48 <lbt> we already have a vpn for all the infra so I 'just' need to setup a new one and then do some iptables on a router/firewall
09:08:18 <larstiq> the point is to get a publically reachable server?
09:08:26 <lbt> larstiq: yes
09:08:30 <larstiq> k
09:08:39 <lbt> and one which can have some kind of reset done on it
09:08:56 <lbt> and, to some degree, share the admin of the service
09:09:07 <larstiq> right
09:09:54 <lbt> so a community guy would manage the owncloud system and probably have root access (ideally just a user but ...) to do updates, DB setup etc
09:10:14 <lbt> they'd certainly have admin access but I'd expect shell to be needed too
09:10:29 <lbt> this model would replicate for every sync service
09:10:55 <lbt> as I say, we use kvm at the moment to spawn VMs
09:11:02 <larstiq> it's a bit hard to say at this point what kind of access is needed after setting it up
09:11:03 <lbt> other solutions include docker
09:11:07 <r0kk3rz> you can mod some vagrant scripts to use kvm for reproducable/updatable vm images
09:11:09 <larstiq> as an example of that: https://github.com/tomav/docker-mailserver
09:11:10 <lbt> larstiq: agreed
09:11:29 <r0kk3rz> which is probably a better idea than using docket
09:11:32 <r0kk3rz> *docker
09:11:55 <lbt> I'm thinking that we should probably aim to have resetable services rather than resetting VMs
09:11:59 <lbt> it feels more sane ?
09:12:01 <chriadam|windows> occirol: from your experience with apache+owncloud services, do you have any comments on what sort of access is required in these vms?
09:12:02 <larstiq> lbt: yeah
09:12:45 <larstiq> r0kk3rz: I'd like to discuss your viewpoints on vagrant, later today?
09:12:49 <lbt> larstiq: do you think this is a good solution for docker-isation?
09:12:55 * larstiq knows of it but not much experience
09:12:56 <lbt> candidate even
09:13:11 <larstiq> lbt: in my personal opinion, yes
09:13:31 <lbt> the main thing I can see is pre-population with a known set of data
09:13:44 <lbt> and I'd *love* to see this as a non-privileged service
09:13:50 * larstiq nods
09:14:16 <lbt> there are IP issues
09:14:18 <larstiq> lbt: both easily doable I think
09:14:29 <larstiq> network, or intellectual property?
09:14:32 <lbt> haha
09:14:32 <r0kk3rz> larstiq: yeah if you want, ive not really used it for anything but it seems like a good thing
09:14:33 <lbt> nw
09:14:47 <occirol> chriadam|windows: well, setting up apache, owncloud, etc would require a certain level of access to the VMs: installing packages, configuring apache files, and a db access
09:15:07 <r0kk3rz> the whole point of it is generating reproducable environments without causing a maintenance nightmare
09:15:08 <lbt> r0kk3rz: occirol: dr_gogeta86.... experience with docker ?
09:15:19 <larstiq> owncloud provides https://hub.docker.com/_/owncloud/
09:15:33 <lbt> occirol: yeah - that's what I'd like to see inside the docker image
09:15:47 <occirol> lbt: nop, no experience yet
09:15:52 <r0kk3rz> lbt: i did read a really good docker rant, i should find it
09:16:06 <larstiq> r0kk3rz: ooh, reminds me I have nice one for that too
09:16:33 <larstiq> r0kk3rz: https://circleci.com/blog/its-the-future/
09:17:17 <r0kk3rz> but if you're just going to use the provided docker things verbatim its probably not so bad
09:17:22 <lbt> docker is a well known name - I'm happy to look at other solutions but we need to be reasonable - whatever tech we use we must have the skills and they must commit to working on this
09:18:05 <larstiq> sure
09:18:13 * lbt asks chriadam|windows -- can we use IPv6 here?
09:18:34 <dr_gogeta86> è1
09:18:37 <dr_gogeta86> back on track
09:18:50 <chriadam|windows> shouldn't matter so long as it has an externally resolvable host name
09:18:51 <chriadam|windows> I think
09:18:56 <dr_gogeta86> i wanna focus on what we need
09:19:01 <lbt> or would you need v4 ? ... I'm thinking that we'd need to share a very limited number of v4 addresses to all these services (like 1-4)
09:19:01 <dr_gogeta86> like a matrix
09:19:19 <dr_gogeta86> lbt we can expose those services behind a reverse proxy
09:19:22 <lbt> chriadam|windows: yeah - if we can use an aliasing reverse-proxy - but that does bring it's own issues :D
09:19:22 <dr_gogeta86> without problems
09:19:33 <dr_gogeta86> lbt like ?
09:19:42 <lbt> dr_gogeta86: well - try doing streaming of large files :)
09:19:51 <lbt> eg git
09:19:54 <dr_gogeta86> of course
09:19:59 <lbt> or obs logs
09:20:04 <dr_gogeta86> but for caldav purpose
09:20:05 <lbt> or http push
09:20:19 <lbt> yeah - I'm just mentioning that some stuff could hurt
09:20:22 <chriadam|windows> caldav/carddav "should" only need to support simple GET/POST/PUT
09:20:35 <chriadam|windows> etc
09:20:54 <dr_gogeta86> is pretty simple imho
09:20:57 <lbt> *nod* ... but owncloud gui and ajax and clever bugger stuff
09:21:08 <occirol> adding a reverse proxy *could* bring some problems
09:21:37 <lbt> occirol: yeah - just mentioning it - I use that approach for *all* mer services so I know it hurts occasionally
09:21:44 <lbt> otoh I like it :)
09:21:54 <occirol> we have not to forget that it will used for testing purposes, so we need to avoid as much as possible "interferences", imho
09:22:09 <occirol> lbt: I like it too :)
09:22:19 <lbt> occirol: yes.
09:22:35 <lbt> even weird stuff like chunking or content negotiation
09:23:28 <lbt> so .. back to the virtualisation ?
09:23:47 <occirol> so, we are talking about how many services right now ? 3 (owncloud, nextcloud, radicale) ?
09:24:04 <lbt> occirol: lets aim for 10+ eventually
09:24:11 <lbt> also different versions
09:24:32 <chriadam|windows> eventually
09:24:38 <lbt> also we need to consider that a test needs a dedicated instance doesn't it ?
09:24:48 <chriadam|windows> well, not necessarily.
09:24:51 <lbt> what happens if 2 users run a transactional test at once?
09:25:00 <lbt> ie add/modify/delete a contact
09:25:05 <lbt> same test contact
09:25:05 <chriadam|windows> they'll fail.  but I'm not seeing these as CI / automated at this stage
09:25:20 <chriadam|windows> it's more like "developer can run a suite of tests, and get results"
09:25:44 <lbt> yeah - I'd like to just consider this for the future
09:25:48 <chriadam|windows> sure
09:26:17 <lbt> ie it would be nice to have a 'give me a hostname to a "personal" caldav server'
09:26:58 <lbt> it's not too hard as long as we realise it's a requirement
09:28:03 <chriadam|windows> right, it depends on the effort required, I guess.  I mean, it would be a 10x improvement to have just a single caldav server which I have full control over (e.g. resetable, which I can build some system tests against)
09:28:29 <lbt> chriadam|windows: yeah - that should fall out as stage 1
09:28:44 <lbt> and I'd avoid doing anything fancy so as not to delay it
09:29:10 <lbt> just some personal thoughts on naming and auto-adding vhosts to the rev proxy
09:29:36 <dr_gogeta86> there was also clever mapping scheme
09:29:40 <lbt> hmm and if we use letsencrypt then throwaway certs...
09:29:43 <dr_gogeta86> like map files
09:29:48 <lbt> dr_gogeta86: ?
09:29:59 <lbt> ah rev proxy
09:30:01 <lbt> *nod*
09:30:24 <lbt> larstiq: r0kk3rz: so ... docker ... any more discussion on that?
09:30:37 <lbt> any serious alternatives?
09:30:39 <dr_gogeta86> i don't think so
09:30:44 <larstiq> lbt: as far as I'm concerned I can start on that as soon as you have a vm and a vpn going
09:31:03 <dr_gogeta86> the advantage of Docker is the reproducible env
09:31:06 <lbt> could I run up a single VM and put docker on it and have many user accounts and have them all manage docker images ?
09:31:09 <dr_gogeta86> I' can work with it locally
09:31:29 <larstiq> lbt: yes
09:31:50 <lbt> so that really sounds like a jfdi for docker then :)
09:31:51 <larstiq> lbt: need to take some care perhaps with clashing ports usage, but with a proxy that becomes trivial
09:31:57 <lbt> *nod*
09:32:32 * lbt is thinking of an automated on-demand image deployment at some point
09:32:33 <dr_gogeta86> lbt first we need the image tailored
09:32:39 <lbt> yep
09:33:05 <dr_gogeta86> We need at least container for different owncloud versions out there
09:33:13 <dr_gogeta86> from 7 to 9 ( nextcloud )
09:33:18 <lbt> what OS on the VM ... Debian OK ? (please)
09:33:29 <dr_gogeta86> better without systemd
09:33:33 <dr_gogeta86> :-D
09:33:37 <lbt> really?
09:33:46 <r0kk3rz> systemd is fine :P
09:33:50 <dr_gogeta86> systemd inside container can be a pain in the but
09:33:51 <lbt> I can believe that from a cgroups PoV
09:33:56 <larstiq> dr_gogeta86: we can start with https://github.com/docker-library/owncloud as a base
09:34:16 <larstiq> dr_gogeta86: but outside systemd is fine
09:34:21 <dr_gogeta86> sure
09:34:35 <dr_gogeta86> but i think to another problem for people like chriadam|windows
09:34:36 <larstiq> lbt: Debian on the VM please :)
09:34:43 <dr_gogeta86> a simple way to check logs
09:34:45 <lbt> good
09:34:51 <dr_gogeta86> and enter into the machne
09:34:55 <larstiq> dr_gogeta86: right
09:35:13 <dr_gogeta86> we are going to build an infra for developers
09:35:19 <dr_gogeta86> not for sysadmins
09:35:20 <larstiq> dr_gogeta86: do you have a solution or do we need to think about that a bit?
09:35:25 <dr_gogeta86> keep in minnd
09:35:27 <dr_gogeta86> *mind
09:35:27 * larstiq nods
09:35:36 <dr_gogeta86> i need to check those docker files
09:35:51 <r0kk3rz> there are docker organisers like kubernetes
09:35:54 <dr_gogeta86> usually i do it myself
09:36:31 <dr_gogeta86> r0kk3rz, temptation is hard but KISS
09:36:55 <lbt> r0kk3rz: it feels to me like we first need our team to develop docker images
09:37:16 <larstiq> worst comes to worst there is `tail -F /var/lib/docker/containers/**/*.log` ;)
09:37:18 <lbt> then we look at image deployment solutions
09:37:20 <dr_gogeta86> docker comes with theis logs
09:37:25 <dr_gogeta86> chriadam|windows, got a testbed plans ?
09:37:25 <lbt> does kubernetes fall into the deployment side ?
09:37:25 <larstiq> r0kk3rz: that's overkill here
09:37:27 <dr_gogeta86> abranson_, too
09:37:37 <dr_gogeta86> larstiq, not at certain point
09:37:37 <lbt> 'cos I'm OK with manual-ish deployment in phase 1
09:37:37 <larstiq> lbt: yes, but not going to need that at the start
09:37:55 <lbt> good
09:37:56 <lbt> chriadam|windows: will be happy
09:37:56 <chriadam|windows> dr_gogeta86: to begin with, the tests will be simple (e.g., trigger a sync, inspect db, ensure content is expected)
09:38:01 <lbt> chriadam|windows: and get logs if it's not
09:38:16 <lbt> but that should be part of the image really
09:38:19 <dr_gogeta86> from bugzilla pov ... got reports from which caldav servers ?
09:38:35 <larstiq> lbt: let's jfdi
09:38:41 <lbt> ROFL
09:38:47 <lbt> was about to say that
09:39:34 <chriadam|windows> ok, so, concretely, what's the plan?
09:39:37 <lbt> so setup a VM with Debian and give it ports 80/443 on an external IP ... more on demand
09:39:47 <r0kk3rz> larstiq: yes, i said 'like' :P turns out theres a few simpler ones that just have a webui of whats running
09:40:13 <lbt> larstiq: install docker from debian packages or ?
09:40:13 <dr_gogeta86> lbt, lgtm
09:40:17 <larstiq> lbt: depends on version
09:40:19 <larstiq> lbt: we can figure that out offline
09:40:29 <lbt> r0kk3rz: can you setup a docker image as a user with such a thing ?
09:40:29 <lbt> larstiq: +1
09:40:29 <dr_gogeta86> we don't need fancy things for now
09:40:42 <dr_gogeta86> lbt go head for debian/devuan
09:41:17 <larstiq> r0kk3rz: first let's just get 1 owncloud instance up. Then it's just "is it running yes/no" ;)
09:41:20 <lbt> r0kk3rz: because I'm happy if you experiment and demo docker managers within your docker images
09:41:36 <lbt> but yeah - as a team we focus on what larstiq just said :)
09:41:39 <larstiq> lbt: docker/docker-compose are really plenty at the start
09:42:07 * lbt is looking forward to playing with this tech too
09:42:26 <lbt> OK .. I'm good I can set this up over the next couple of days
09:42:49 <lbt> I want to put up a new VPN and a firewall/router
09:43:03 <lbt> I'll play with LDAP accounts
09:43:21 <lbt> I'll post requirements for shell ssh keys etc too
09:43:22 <dr_gogeta86> i hope not openvpn
09:43:36 <lbt> we won't be exposing it
09:43:44 <lbt> this is a management VPN
09:43:58 <lbt> (and it's tinc for now - OVS soon probably)
09:44:20 <lbt> ssh shell access via a port (probably 2222)
09:44:47 <lbt> some minor ssh key management requirement stuff too
09:45:25 <lbt> so ... are we done ?
09:45:31 <chriadam|windows> just so sum up:
09:45:48 <chriadam|windows> 1) lbt will create the VM and install docker in it, get LDAP accounts set up, and give details to dr_gogeta86 and occirol
09:46:01 <chriadam|windows> 2) dr_gogeta86 and occirol will then ... get a docker image up and running with OwnCloud on it?
09:46:09 <chriadam|windows> 3) someone will let me know when that's done and how to use it
09:46:11 <chriadam|windows> ?
09:46:12 * LarstiQ inserts his name in there somewhere
09:46:22 <chriadam|windows> LarstiQ: does stuff I don't understand to make it work ;-)
09:46:31 * lbt will share setup with LarstiQ
09:46:59 <chriadam|windows> dr_gogeta86: occirol: ok, I will ping you guys this time next week to see what our status is
09:47:03 <chriadam|windows> if that makes sense to everyone?
09:47:08 <dr_gogeta86> ok for me
09:47:51 <lbt> chriadam|windows: another meeting this time next week ?
09:47:57 <chriadam|windows> it might take a couple of weeks for lbt + LarstiQ to get the VM etc setup done - depends on what crises occur in Jolla ;-)
09:48:09 <chriadam|windows> but if we weekly track progress, that might be helpful / keep everyone on same page
09:48:19 <lbt> *nod* ... I'm happy to do weekly updates.... yeah
09:48:19 <chriadam|windows> lbt: I think so, would be good, if everyone is ok with it
09:48:26 <chriadam|windows> great
09:48:31 <chriadam|windows> ok, thanks everuone - much appreciated!
09:48:36 <chriadam|windows> ending meeting in 5...
09:48:37 <occirol> ok, I think :)
09:48:47 <chriadam|windows> 3
09:48:50 <chriadam|windows> 2
09:48:51 <chriadam|windows> 1
09:48:53 <chriadam|windows> #endmeeting